Spark architecture (driver, executors, cluster manager) in Apache Spark - Mini Project: Build & Apply

Understanding Spark Architecture: Driver, Executors, and Cluster Manager

📖 Scenario: You are working with Apache Spark to process large data sets. To understand how Spark runs your code, you need to learn about its architecture components: the driver, executors, and cluster manager.Think of Spark like a team working on a big project. The driver is the team leader who plans the work. The executors are the team members who do the tasks. The cluster manager is like the office manager who assigns resources to the team.

🎯 Goal: Build a simple Spark program that shows how to create a Spark session (driver), configure executors, and connect to a cluster manager. This will help you see how Spark components work together.

📋 What You'll Learn

Create a SparkSession named spark to act as the driver.

Set a configuration for the number of executors using spark.conf.set.

Specify the cluster manager URL in the SparkSession builder.

Print the Spark configuration to show the settings.

💡 Why This Matters

🌍 Real World

Understanding Spark architecture helps you manage big data processing efficiently by controlling how tasks are distributed and executed across a cluster.

💼 Career

Knowledge of Spark's driver, executors, and cluster manager is essential for data engineers and data scientists working with distributed data processing frameworks.

1

Create SparkSession as Driver

Create a SparkSession called spark using SparkSession.builder() with the app name 'SparkArchitectureApp'. This SparkSession acts as the driver in Spark architecture.

Apache Spark

# Create a SparkSession called spark with app name 'SparkArchitectureApp'
# Your code here

Need a hint?

Use SparkSession.builder.appName('SparkArchitectureApp').getOrCreate() to create the driver session.

2

Configure Number of Executors

Use spark.conf.set to set the configuration 'spark.executor.instances' to '3'. This simulates setting up 3 executors in the Spark cluster.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('SparkArchitectureApp').getOrCreate()
# Set spark.executor.instances to '3'
# Your code here

Need a hint?

Use spark.conf.set('spark.executor.instances', '3') to configure executors.

3

Specify Cluster Manager URL

Modify the SparkSession builder to include .master('spark://localhost:7077') to specify the cluster manager URL. This tells Spark where the cluster manager is running.

Apache Spark

from pyspark.sql import SparkSession

# Create SparkSession with app name and master URL
spark = SparkSession.builder.appName('SparkArchitectureApp').master('spark://localhost:7077').getOrCreate()

spark.conf.set('spark.executor.instances', '3')

Need a hint?

Add .master('spark://localhost:7077') to the SparkSession builder.

4

Print Spark Configuration

Print the value of spark.conf.get('spark.executor.instances') to show the number of executors configured.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('SparkArchitectureApp').master('spark://localhost:7077').getOrCreate()
spark.conf.set('spark.executor.instances', '3')

# Print the number of executors
# Your code here

Need a hint?

Use print(spark.conf.get('spark.executor.instances')) to display the executor count.