Local mode vs cluster mode in Apache Spark - Hands-On Comparison

Understanding Local Mode vs Cluster Mode in Apache Spark

📖 Scenario: You are working with Apache Spark to process data. Spark can run in two main ways: local mode and cluster mode. Local mode runs Spark on your own computer using all cores, while cluster mode runs Spark on many computers together.Understanding these modes helps you choose the right setup for your data tasks.

🎯 Goal: You will create a simple Spark program that runs in local mode and then configure it to run in cluster mode. You will see how to set up Spark contexts for both modes and print the mode used.

📋 What You'll Learn

Create a SparkSession configured for local mode

Create a variable to hold the cluster mode setting

Use the cluster mode setting to configure SparkSession for cluster mode

Print the Spark master URL to show which mode is running

💡 Why This Matters

🌍 Real World

Data engineers and data scientists use Spark in local mode for development and testing on their computers. For big data processing, they run Spark in cluster mode on many machines to handle large datasets efficiently.

💼 Career

Understanding local vs cluster mode is essential for roles involving big data processing, as it helps optimize resource use and job performance in real-world data pipelines.

1

Create SparkSession in Local Mode

Create a SparkSession called spark with the master set to local[*] to run Spark locally using all cores.

Apache Spark

# Create SparkSession with master 'local[*]'
# Your code here

Need a hint?

Use SparkSession.builder.master('local[*]') to set local mode.

2

Add Cluster Mode Configuration Variable

Create a variable called cluster_mode and set it to 'local[*]' initially to represent local mode. This variable will help switch between local and cluster modes.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.master('local[*]').appName('LocalModeApp').getOrCreate()

# Create variable cluster_mode and set to 'local[*]'
# Your code here

Need a hint?

Set cluster_mode = 'local[*]' and use it in master().

3

Switch to Cluster Mode Using Configuration Variable

Change the value of cluster_mode to 'spark://master:7077' to simulate running Spark in cluster mode. Then create a new SparkSession called spark using this cluster_mode variable as the master URL.

Apache Spark

from pyspark.sql import SparkSession

# Change cluster_mode to 'spark://master:7077'
cluster_mode = 'spark://master:7077'

# Create SparkSession with master set to cluster_mode
# Your code here

Need a hint?

Set cluster_mode = 'spark://master:7077' and use it in SparkSession.builder.master().

4

Print the Spark Master URL to Show Mode

Print the Spark master URL by accessing spark.sparkContext.master to display whether Spark is running in local or cluster mode.

Apache Spark

from pyspark.sql import SparkSession

cluster_mode = 'spark://master:7077'
spark = SparkSession.builder.master(cluster_mode).appName('ClusterModeApp').getOrCreate()

# Print the Spark master URL
# Your code here

Need a hint?

Use print(spark.sparkContext.master) to show the mode.