Spark architecture (driver, executors, cluster manager) in Apache Spark - Mini Project: Build & Apply

Understanding Spark Architecture: Driver, Executors, and Cluster Manager

📖 Scenario: You are working with Apache Spark to process large data sets. To understand how Spark runs your code, you need to learn about its architecture components: the driver, executors, and cluster manager.Think of Spark like a team working on a big project. The driver is the team leader who plans the work. The executors are the team members who do the tasks. The cluster manager is like the office manager who assigns resources to the team.

🎯 Goal: Build a simple Spark program that shows how to create a Spark session (driver), configure executors, and connect to a cluster manager. This will help you see how Spark components work together.

📋 What You'll Learn

Create a SparkSession named spark to act as the driver.

Set a configuration for the number of executors using spark.conf.set.

Specify the cluster manager URL in the SparkSession builder.

Print the Spark configuration to show the settings.

💡 Why This Matters

🌍 Real World

Understanding Spark architecture helps you manage big data processing efficiently by controlling how tasks are distributed and executed across a cluster.

💼 Career

Knowledge of Spark's driver, executors, and cluster manager is essential for data engineers and data scientists working with distributed data processing frameworks.

1

Create SparkSession as Driver

Create a SparkSession called spark using SparkSession.builder() with the app name 'SparkArchitectureApp'. This SparkSession acts as the driver in Spark architecture.

Apache Spark

# Create a SparkSession called spark with app name 'SparkArchitectureApp'
# Your code here

Hint

Use SparkSession.builder.appName('SparkArchitectureApp').getOrCreate() to create the driver session.

2

Configure Number of Executors

Use spark.conf.set to set the configuration 'spark.executor.instances' to '3'. This simulates setting up 3 executors in the Spark cluster.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('SparkArchitectureApp').getOrCreate()
# Set spark.executor.instances to '3'
# Your code here

Hint

Use spark.conf.set('spark.executor.instances', '3') to configure executors.

3

Specify Cluster Manager URL

Modify the SparkSession builder to include .master('spark://localhost:7077') to specify the cluster manager URL. This tells Spark where the cluster manager is running.

Apache Spark

from pyspark.sql import SparkSession

# Create SparkSession with app name and master URL
spark = SparkSession.builder.appName('SparkArchitectureApp').master('spark://localhost:7077').getOrCreate()

spark.conf.set('spark.executor.instances', '3')

Hint

Add .master('spark://localhost:7077') to the SparkSession builder.

4

Print Spark Configuration

Print the value of spark.conf.get('spark.executor.instances') to show the number of executors configured.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('SparkArchitectureApp').master('spark://localhost:7077').getOrCreate()
spark.conf.set('spark.executor.instances', '3')

# Print the number of executors
# Your code here

Hint

Use print(spark.conf.get('spark.executor.instances')) to display the executor count.

Practice

1. Which component in Spark is responsible for planning the tasks and controlling the job execution?

easy

A. Task Scheduler

B. Executor

C. Cluster Manager

D. Driver

Solution

1. Step 1: Understand the role of the Driver

   The Driver is the main program that creates the SparkContext and plans the tasks to run.

2. Step 2: Compare with other components

   Executors run the tasks, and the Cluster Manager manages resources, but only the Driver plans and controls the job.

3. Final Answer:

   Driver -> Option D

4. Quick Check:

   Driver = plans tasks [OK]

Hint: Driver plans tasks; executors run them [OK]

Common Mistakes:

• Confusing executors with driver
• Thinking cluster manager plans tasks
• Mixing driver with resource manager

2. Which of the following is the correct way to describe the role of executors in Spark?

easy

A. They manage cluster resources and allocate executors

B. They plan the job and schedule tasks

C. They run tasks and store data for the application

D. They submit jobs to the cluster manager

Solution

1. Step 1: Identify executor responsibilities

   Executors run the tasks assigned by the driver and store data in memory or disk during processing.

2. Step 2: Eliminate other options

   Cluster manager manages resources, driver plans jobs, and job submission is done by the driver, not executors.

3. Final Answer:

   They run tasks and store data for the application -> Option C

4. Quick Check:

   Executors = run tasks + store data [OK]

Hint: Executors run tasks and keep data [OK]

Common Mistakes:

• Confusing executors with cluster manager
• Thinking executors plan jobs
• Mixing driver and executor roles

3. Consider a Spark application running on a cluster. Which component is responsible for requesting resources from the cluster manager?

medium

A. Driver

B. Executor

C. Cluster Manager

D. Task Scheduler

Solution

1. Step 1: Understand resource request flow

   The Driver requests resources (executors) from the Cluster Manager to run tasks.

2. Step 2: Clarify roles of other components

   Executors run tasks, Cluster Manager allocates resources, but Driver initiates the request.

3. Final Answer:

   Driver -> Option A

4. Quick Check:

   Driver requests resources [OK]

Hint: Driver asks cluster manager for executors [OK]

Common Mistakes:

• Thinking executors request resources
• Confusing cluster manager with driver
• Assuming task scheduler requests resources

4. You see a Spark application failing because no executors are allocated. Which component should you check first to fix this issue?

medium

A. Cluster Manager configuration

B. Executor logs

C. Driver program code

D. Task Scheduler settings

Solution

1. Step 1: Identify the cause of no executors

   No executors means the cluster manager did not allocate resources.

2. Step 2: Check cluster manager setup

   Cluster Manager manages resource allocation; misconfiguration here can block executors.

3. Final Answer:

   Cluster Manager configuration -> Option A

4. Quick Check:

   Cluster Manager controls executors allocation [OK]

Hint: Check cluster manager if no executors start [OK]

Common Mistakes:

• Checking driver code first instead of cluster manager
• Looking at executor logs when none exist
• Confusing task scheduler with resource manager

5. In a Spark cluster, if the driver fails but executors continue running, what is the expected behavior and why?

hard

A. Executors keep running tasks independently without driver

B. Executors stop working because driver controls task scheduling

C. Cluster manager automatically restarts executors without driver

D. Driver failure does not affect executors as they are independent

Solution

1. Step 1: Understand driver role in task scheduling

   The driver plans and schedules tasks; executors depend on it for instructions.

2. Step 2: Analyze executor behavior on driver failure

   Without the driver, executors cannot receive new tasks and will stop processing.

3. Final Answer:

   Executors stop working because driver controls task scheduling -> Option B

4. Quick Check:

   Driver failure stops executors [OK]

Hint: Executors need driver to schedule tasks [OK]

Common Mistakes:

• Assuming executors run independently
• Thinking cluster manager restarts executors automatically
• Believing driver failure has no impact