Adv Spark Programming

33 / 52

Adv Spark Programming - Broadcast Variables Example

Code is available here: Broadcast Example

	var commonWords = Array("a", "an", "the", "of", "at", "is", "am","are","this","that","at", "in", "or", "and", "or", "not", "be", "for", "to", "it")
	val commonWordsMap = collection.mutable.Map[String, Int]()
	for(word <- commonWords){
	commonWordsMap(word) = 1
	}
	var commonWordsBC = sc.broadcast(commonWordsMap)

	var file = sc.textFile("/data/mr/wordcount/input/big.txt")
	def toWords(line:String):Array[String] = {
	var words = line.split(" ")
	var output = Array[String]();
	for(word <- words){
	if(! (commonWordsBC.value contains word.toLowerCase.trim.replaceAll("[^a-z]",""))) output = output :+ word;
	}
	return output;
	}
	var uncommonWords = file.flatMap(toWords)
	uncommonWords.take(100)

view raw broadcast-example.scala hosted with ❤ by GitHub

Slides - Adv Spark Programming (2)

Previous Index Next

Please login to comment

24 Comments

This comment has been removed.

Vagdevi K

4 years ago

Hi,

Please go through previous slides and chapters and let us know which topic you didn't understand.

Thanks.

Upvote Share

Vagdevi K

4 years ago

Also, we have provided the live session recordings, so feel free to go through them for better understanding.

Upvote Share

Mayank Chaubey

4 years ago

It is throwing an error. I am just copying and pasting the codes. Actually didn't understood even a single line.

Upvote Share

This comment has been removed.

Vagdevi K

4 years ago

Hi,

Please go through previous slides and chapters and let us know which topic you didn't understand.

Thanks.

Upvote Share

Sandeep Giri

4 years ago

Please go thru the previous sessions and also the recordings of the live session.

Upvote Share

Dhyey Kotecha

5 years ago

Hey! Cloudxlab Team,

It looks like the code needs some update. Upon running, it returned a NullPointerException error.

var commonWords = Array("a", "an", "the", "of", "at", "is", "am","are","this","that","at","in", "or", "and", "or", "not", "be", "for", "to", "it")

val commonWordsMap = collection.mutable.Map[String, Int]()

for (word <- commonWords) {
    commonWordsMap(word) = 1
}

var commonWordsBC = sc.broadcast(commonWordsMap)

var file = sc.textFile("/data/mr/wordcount/input/big.txt")

def toWords(line:String):Array[String] = {
    var words = line.split(" ")
    var output = Array[String]()
    
    for(word <- words) {
        if(!(commonWordsBC.value contains word.toLowerCase.trim.replaceAll("[^a-z]",""))) output = output :+ word
    }
    return output
}

var uncommonWords = file.flatMap(toWords)
uncommonWords.take(10)

Upvote Share

This comment has been removed.

Amit Padhi

5 years ago

Code is available here: there is no link

Upvote Share

Satyajit Das

5 years ago

Hi, Amit.

Here is the link you can find all the codes here.

https://gist.github.com/girisandeep/f12ab4bf2536dc5f0a8ca673efbac1db#file-broadcast-example-scala

All

Upvote Share

dilip kumar

5 years ago

There is no link or code shared as shown on the attached screen shot. Please share the link.

Upvote Share

Sachin Giri

5 years ago

Hi Dilip,

Code is there, can you please try reloading?.

It may take longer as it is rendering from github.

If it is still not visible then please go to this link: https://gist.github.com/girisandeep/f12ab4bf2536dc5f0a8ca673efbac1db#file-broadcast-example-scala

Upvote Share

Anant Kumar Vinocha

5 years ago

No code is visible as shared in snippet

Upvote Share

Sachin Giri

5 years ago

Hi Anant,

Code is there, can you please try reloading?.

It may take longer as it is rendering from github.

If it is still not visible then please go to this link: https://gist.github.com/girisandeep/f12ab4bf2536dc5f0a8ca673efbac1db#file-broadcast-example-scala

Upvote Share

Vaibhav Garg

5 years ago

Sir ,Why all these type of new heavy codes are not explained everytime ??

Upvote Share

Anubhav Gupta

5 years ago

please also explain all the terms and function you are using as we are doing scala for the very first time in every vedio you guys are coming up with a new operation and only keeps on dictating it instead of making us understand.

REALLY REALLY DISAPPOINTED!!!

1 Upvote Share

Hemin Modi

5 years ago

not able to find link

Upvote Share

Sarthak Srivastava

5 years ago

When I run the above code I get the following error---

Name: org.apache.spark.SparkException
Message: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): java.lang.NullPointerException
at $line13.$read$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$toWords$1.apply(<console>:39)
at $line13.$re

1 Upvote Share

Dhirendra Bamrara

5 years ago

Hi Sandeep,
I couldn't see any link below of "Code is available here:"

Upvote Share

Satyajit Das

5 years ago

Hi, Dhirendra.

Can you please check again!.
The code is just below the video!. It is completely visible!
Please let me know if still you could not find it.

All the best!

Upvote Share

Thiago M. Pinto

7 years ago

Code is available here:

But there is no link or code.

Upvote Share

Mohammad Shahrukh

7 years ago

Hi Thiago,
The code is there in the section just below the text.

Upvote Share