Hello Team, spark-submit --class "WordCount" --master "local[2]" target/scala-2.10/word-count_2.10-1.0.jar in above command. "Local[2]" what does it mean? Please explain.
We are here submitting the spark job with Local means the spark is running locally in your machine with 2 worker cores or cpu. if you provide local[*] then it will take all the cores available for processing.
Please login to comment
5 Comments
Please also provide the pdf link of each and every module of the slide so that we can download the pdf slide for revision of a particular topic.
Upvote ShareSpark streaming lessons seems to be on tougher side than Python.
Upvote ShareHello Team,
Upvote Sharespark-submit --class "WordCount" --master "local[2]"
target/scala-2.10/word-count_2.10-1.0.jar
in above command. "Local[2]" what does it mean?
Please explain.
As per my understanding, it will run on system with 2 cores.
Upvote ShareYes, you are right.
We are here submitting the spark job with Local means the spark is running locally in your machine with 2 worker cores or cpu.
if you provide local[*] then it will take all the cores available for processing.
All the best!
-- Satyajit Das
Upvote Share