Apache Spark Basics

31 / 89
Previous Index Next  

When we launch an spark-shell or pyspark, it also launches:


Note - Having trouble with the assessment engine? Follow the steps listed here


No hints are availble for this assesment

Answer is not availble for this assesment

Previous Index Next  
Please login to comment

5 Comments

Chilukuri Sai Sumanth
 10 months ago 

Hello 

Can you please explain the above question indetail?

  Upvote    Share
Shubh Tripathi
 10 months ago 

When you start spark-shell or pyspark, the shell environment initializes and starts a Spark Driver locally.

The Driver process creates a SparkContext (or SparkSession in the case of Spark 2.x and later).

The Driver then interacts with the cluster manager (e.g., YARN, Mesos, or the built-in standalone cluster manager) to allocate resources for the executors.

The Driver translates the user commands (such as transformations and actions on RDDs or DataFrames) into a DAG (Directed Acyclic Graph) of tasks.

It then schedules these tasks to be executed on the cluster's worker nodes (executors).

Therefore, when you start spark-shell or pyspark, you are launching a Spark Driver process.

  Upvote    Share
Munazir Ahsan
 5 years ago 

Please elaborate?
When we launch an spark-shell or pyspark, How does it start Spark Driver?

  Upvote    Share
Dinesh Ughade
 5 years ago 

Hello,
How does it start spark-application and who triggers it?

  Upvote    Share
Deshbandhumishra
 5 years ago 

spark-Application: It is installed on each and every data nodes. where as spark driver is on name node only which controls all the data nodes. So, It is the spark-drivers which triggers.

Lets join a group to discuss more.
https://chat.whatsapp.com/G...

  Upvote    Share