Spark architecture (driver, executors, cluster manager) in Apache Spark - Step-by-Step Execution

Concept Flow - Spark architecture (driver, executors, cluster manager)

User submits Spark job

↓

Driver program starts

↓

Driver requests resources from Cluster Manager

↓

Cluster Manager allocates Executors

↓

Executors run tasks

↓

Executors send results back to Driver

↓

Driver collects results and completes job

This flow shows how a Spark job moves from user submission through the driver, cluster manager, executors, and back to the driver for results.

Execution Sample

Apache Spark

# Pseudocode for Spark job execution
spark = SparkSession.builder.appName('Example').getOrCreate()
data = spark.read.csv('data.csv')
result = data.filter("cast(_c1 as int) > 10").count()
print(result)

This code reads data, filters rows where second column > 10, counts them, and prints the result.

Execution Table

Step	Component	Action	Details	Output
1	User	Submits job	Job with filter and count	Job sent to Driver
2	Driver	Starts	Initializes SparkContext and DAG	Ready to schedule tasks
3	Driver	Requests resources	Asks Cluster Manager for executors	Resource request sent
4	Cluster Manager	Allocates executors	Assigns executors on worker nodes	Executors launched
5	Executors	Run tasks	Filter and count tasks executed on data partitions	Partial counts computed
6	Executors	Send results	Partial counts sent back to Driver	Partial results received
7	Driver	Aggregates results	Sums partial counts	Final count computed
8	Driver	Job complete	Prints final count	Output displayed to user
9	-	Exit	Job finished successfully	-

💡 Job completes after driver aggregates results and prints output

Variable Tracker

Variable	Start	After Step 5	After Step 7	Final
spark	None	SparkSession active	SparkSession active	SparkSession active
data	None	Data loaded as DataFrame	DataFrame unchanged	DataFrame unchanged
partial_counts	None	List of counts from executors	Aggregated sum	Aggregated sum
final_count	None	None	Sum of partial counts	Printed output

Key Moments - 3 Insights

Why does the driver request resources from the cluster manager before running tasks?

The driver needs executors to run tasks. It asks the cluster manager to allocate these executors on worker nodes, as shown in steps 3 and 4 of the execution table.

What is the role of executors in Spark architecture?

Executors run the actual tasks on data partitions and send results back to the driver. This is shown in steps 5 and 6 where executors process data and return partial counts.

Why does the driver aggregate results after executors finish tasks?

Executors compute partial results on data partitions. The driver collects these partial results and combines them to get the final output, as seen in step 7.

Visual Quiz - 3 Questions

Look at the execution table, at which step does the cluster manager allocate executors?

A Step 2

B Step 4

C Step 6

D Step 8

💡 Hint

Check the 'Component' and 'Action' columns in the execution table for cluster manager activities.

According to the variable tracker, what is the state of 'partial_counts' after step 5?

A List of counts from executors

B None

C Aggregated sum

D Printed output

💡 Hint

Look at the 'partial_counts' row and the 'After Step 5' column in the variable tracker.

If the driver did not aggregate results, what would be missing in the execution table?

A Executors running tasks

B Cluster manager allocating executors

C Final count computed

D User submitting job

💡 Hint

Refer to step 7 in the execution table where the driver aggregates results.

Concept Snapshot

Spark architecture overview:
- Driver: coordinates job, creates tasks
- Cluster Manager: allocates executors
- Executors: run tasks on data partitions
- Data flows from user to driver, then executors, back to driver
- Driver aggregates results and completes job

Full Transcript

In Spark architecture, the user submits a job that starts the driver program. The driver initializes and requests resources from the cluster manager. The cluster manager allocates executors on worker nodes. Executors run tasks on data partitions and send partial results back to the driver. The driver aggregates these results to produce the final output and completes the job. Variables like the SparkSession, data, partial counts, and final count change state through these steps. Key points include the driver's role in resource requests and result aggregation, and executors' role in task execution.