Spark architecture (driver, executors, cluster manager) in Apache Spark - Step-by-Step Execution

Concept Flow - Spark architecture (driver, executors, cluster manager)

User submits Spark job

↓

Driver program starts

↓

Driver requests resources from Cluster Manager

↓

Cluster Manager allocates Executors

↓

Executors run tasks

↓

Executors send results back to Driver

↓

Driver collects results and completes job

This flow shows how a Spark job moves from user submission through the driver, cluster manager, executors, and back to the driver for results.

Execution Sample

Apache Spark

# Pseudocode for Spark job execution
spark = SparkSession.builder.appName('Example').getOrCreate()
data = spark.read.csv('data.csv')
result = data.filter("cast(_c1 as int) > 10").count()
print(result)

This code reads data, filters rows where second column > 10, counts them, and prints the result.

Execution Table

Step	Component	Action	Details	Output
1	User	Submits job	Job with filter and count	Job sent to Driver
2	Driver	Starts	Initializes SparkContext and DAG	Ready to schedule tasks
3	Driver	Requests resources	Asks Cluster Manager for executors	Resource request sent
4	Cluster Manager	Allocates executors	Assigns executors on worker nodes	Executors launched
5	Executors	Run tasks	Filter and count tasks executed on data partitions	Partial counts computed
6	Executors	Send results	Partial counts sent back to Driver	Partial results received
7	Driver	Aggregates results	Sums partial counts	Final count computed
8	Driver	Job complete	Prints final count	Output displayed to user
9	-	Exit	Job finished successfully	-

💡 Job completes after driver aggregates results and prints output

Variable Tracker

Variable	Start	After Step 5	After Step 7	Final
spark	None	SparkSession active	SparkSession active	SparkSession active
data	None	Data loaded as DataFrame	DataFrame unchanged	DataFrame unchanged
partial_counts	None	List of counts from executors	Aggregated sum	Aggregated sum
final_count	None	None	Sum of partial counts	Printed output

Key Moments - 3 Insights

Why does the driver request resources from the cluster manager before running tasks?

The driver needs executors to run tasks. It asks the cluster manager to allocate these executors on worker nodes, as shown in steps 3 and 4 of the execution table.

What is the role of executors in Spark architecture?

Executors run the actual tasks on data partitions and send results back to the driver. This is shown in steps 5 and 6 where executors process data and return partial counts.

Why does the driver aggregate results after executors finish tasks?

Executors compute partial results on data partitions. The driver collects these partial results and combines them to get the final output, as seen in step 7.

Visual Quiz - 3 Questions

Look at the execution table, at which step does the cluster manager allocate executors?

A Step 2

B Step 4

C Step 6

D Step 8

💡 Hint

Check the 'Component' and 'Action' columns in the execution table for cluster manager activities.

According to the variable tracker, what is the state of 'partial_counts' after step 5?

A List of counts from executors

B None

C Aggregated sum

D Printed output

💡 Hint

Look at the 'partial_counts' row and the 'After Step 5' column in the variable tracker.

If the driver did not aggregate results, what would be missing in the execution table?

A Executors running tasks

B Cluster manager allocating executors

C Final count computed

D User submitting job

💡 Hint

Refer to step 7 in the execution table where the driver aggregates results.

Concept Snapshot

Spark architecture overview:
- Driver: coordinates job, creates tasks
- Cluster Manager: allocates executors
- Executors: run tasks on data partitions
- Data flows from user to driver, then executors, back to driver
- Driver aggregates results and completes job

Full Transcript

In Spark architecture, the user submits a job that starts the driver program. The driver initializes and requests resources from the cluster manager. The cluster manager allocates executors on worker nodes. Executors run tasks on data partitions and send partial results back to the driver. The driver aggregates these results to produce the final output and completes the job. Variables like the SparkSession, data, partial counts, and final count change state through these steps. Key points include the driver's role in resource requests and result aggregation, and executors' role in task execution.

Practice

1. Which component in Spark is responsible for planning the tasks and controlling the job execution?

easy

A. Task Scheduler

B. Executor

C. Cluster Manager

D. Driver

Solution

1. Step 1: Understand the role of the Driver

   The Driver is the main program that creates the SparkContext and plans the tasks to run.

2. Step 2: Compare with other components

   Executors run the tasks, and the Cluster Manager manages resources, but only the Driver plans and controls the job.

3. Final Answer:

   Driver -> Option D

4. Quick Check:

   Driver = plans tasks [OK]

Hint: Driver plans tasks; executors run them [OK]

Common Mistakes:

• Confusing executors with driver
• Thinking cluster manager plans tasks
• Mixing driver with resource manager

2. Which of the following is the correct way to describe the role of executors in Spark?

easy

A. They manage cluster resources and allocate executors

B. They plan the job and schedule tasks

C. They run tasks and store data for the application

D. They submit jobs to the cluster manager

Solution

1. Step 1: Identify executor responsibilities

   Executors run the tasks assigned by the driver and store data in memory or disk during processing.

2. Step 2: Eliminate other options

   Cluster manager manages resources, driver plans jobs, and job submission is done by the driver, not executors.

3. Final Answer:

   They run tasks and store data for the application -> Option C

4. Quick Check:

   Executors = run tasks + store data [OK]

Hint: Executors run tasks and keep data [OK]

Common Mistakes:

• Confusing executors with cluster manager
• Thinking executors plan jobs
• Mixing driver and executor roles

3. Consider a Spark application running on a cluster. Which component is responsible for requesting resources from the cluster manager?

medium

A. Driver

B. Executor

C. Cluster Manager

D. Task Scheduler

Solution

1. Step 1: Understand resource request flow

   The Driver requests resources (executors) from the Cluster Manager to run tasks.

2. Step 2: Clarify roles of other components

   Executors run tasks, Cluster Manager allocates resources, but Driver initiates the request.

3. Final Answer:

   Driver -> Option A

4. Quick Check:

   Driver requests resources [OK]

Hint: Driver asks cluster manager for executors [OK]

Common Mistakes:

• Thinking executors request resources
• Confusing cluster manager with driver
• Assuming task scheduler requests resources

4. You see a Spark application failing because no executors are allocated. Which component should you check first to fix this issue?

medium

A. Cluster Manager configuration

B. Executor logs

C. Driver program code

D. Task Scheduler settings

Solution

1. Step 1: Identify the cause of no executors

   No executors means the cluster manager did not allocate resources.

2. Step 2: Check cluster manager setup

   Cluster Manager manages resource allocation; misconfiguration here can block executors.

3. Final Answer:

   Cluster Manager configuration -> Option A

4. Quick Check:

   Cluster Manager controls executors allocation [OK]

Hint: Check cluster manager if no executors start [OK]

Common Mistakes:

• Checking driver code first instead of cluster manager
• Looking at executor logs when none exist
• Confusing task scheduler with resource manager

5. In a Spark cluster, if the driver fails but executors continue running, what is the expected behavior and why?

hard

A. Executors keep running tasks independently without driver

B. Executors stop working because driver controls task scheduling

C. Cluster manager automatically restarts executors without driver

D. Driver failure does not affect executors as they are independent

Solution

1. Step 1: Understand driver role in task scheduling

   The driver plans and schedules tasks; executors depend on it for instructions.

2. Step 2: Analyze executor behavior on driver failure

   Without the driver, executors cannot receive new tasks and will stop processing.

3. Final Answer:

   Executors stop working because driver controls task scheduling -> Option B

4. Quick Check:

   Driver failure stops executors [OK]

Hint: Executors need driver to schedule tasks [OK]

Common Mistakes:

• Assuming executors run independently
• Thinking cluster manager restarts executors automatically
• Believing driver failure has no impact