Local mode vs cluster mode in Apache Spark - Visual Side-by-Side Comparison

Concept Flow - Local mode vs cluster mode

Start Spark Application

↓

Choose Mode

↓

Local Mode

↓

Run on Local

↓

Single JVM

↓

Collect Results

↓

End

Spark starts and chooses either local mode or cluster mode. Local mode runs everything on one machine. Cluster mode splits work across many machines, then collects results.

Execution Sample

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local[*]').appName('LocalApp').getOrCreate()
data = spark.range(5).collect()
spark.stop()

This code runs a Spark job in local mode, creating a small dataset and collecting it back.

Execution Table

Step	Action	Mode	Details	Result
1	Start SparkSession	Local	master='local[*]'	Spark runs locally on all CPU cores
2	Create Dataset	Local	spark.range(5)	Dataset with numbers 0 to 4 created
3	Collect Data	Local	collect()	Data collected to driver as list [0, 1, 2, 3, 4]
4	Stop Spark	Local	spark.stop()	Spark session ends
5	Start SparkSession	Cluster	master='spark://cluster-master:7077'	Spark connects to cluster manager
6	Create Dataset	Cluster	spark.range(5)	Dataset created and split across cluster nodes
7	Distribute Tasks	Cluster	Tasks sent to worker nodes	Parallel processing on multiple machines
8	Collect Data	Cluster	collect()	Results gathered back to driver node
9	Stop Spark	Cluster	spark.stop()	Spark session ends
10	Exit	-	-	All jobs finished

💡 Execution stops after Spark session is stopped in both modes.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 6	After Step 8	Final
spark	None	SparkSession(local)	SparkSession(local)	SparkSession(cluster)	SparkSession(cluster)	None
data	None	Dataset(0-4)	[0, 1, 2, 3, 4]	Dataset(0-4)	[0, 1, 2, 3, 4]	None

Key Moments - 2 Insights

Why does local mode use 'local[*]' as master and cluster mode use a URL?

Local mode uses 'local[*]' to run Spark on all CPU cores of one machine (see execution_table step 1). Cluster mode uses a URL like 'spark://...' to connect to a cluster manager that controls many machines (see step 5).

How does data processing differ between local and cluster modes?

In local mode, all processing happens inside one JVM on one machine (step 3). In cluster mode, tasks are split and run on multiple worker nodes in parallel (step 7), then results are collected back (step 8).

Visual Quiz - 3 Questions

Look at the execution table, what is the result after step 3 in local mode?

A Data collected as list [0, 1, 2, 3, 4]

B Dataset created but not collected

C Spark session stopped

D Tasks distributed to cluster nodes

💡 Hint

Check the 'Result' column for step 3 in the execution_table.

At which step does Spark distribute tasks to multiple machines in cluster mode?

A Step 8

B Step 6

C Step 7

D Step 9

💡 Hint

Look for 'Distribute Tasks' action in cluster mode in the execution_table.

If you change master from 'local[*]' to 'local[1]', how does the execution change?

A Spark runs on multiple cluster nodes

B Spark runs on only one CPU core instead of all cores

C Spark cannot start

D Spark runs on GPU

💡 Hint

Refer to the meaning of 'local[*]' vs 'local[1]' in the concept_snapshot.

Concept Snapshot

Spark Modes Quick Reference:
- Local mode: master='local[*]' runs Spark on all CPU cores of one machine.
- Cluster mode: master='spark://...' connects to a cluster manager for distributed computing.
- Local mode runs in a single JVM; cluster mode runs tasks on many machines.
- Use local mode for testing and small jobs; cluster mode for big data processing.
- collect() gathers distributed data back to the driver program.

Full Transcript

This visual execution compares Spark's local mode and cluster mode. Spark starts by choosing a mode. In local mode, Spark runs everything on one machine using all CPU cores. The example code creates a small dataset of numbers 0 to 4 and collects it back to the driver. The execution table shows steps like starting SparkSession with 'local[*]', creating data, collecting it, and stopping Spark. In cluster mode, Spark connects to a cluster manager URL, splits the dataset across many machines, distributes tasks to worker nodes, collects results, and stops. Variables like 'spark' and 'data' change accordingly. Key moments clarify why local mode uses 'local[*]' and how cluster mode distributes work. The quiz tests understanding of execution steps and mode differences. The snapshot summarizes the main points for quick recall.

Practice

1. What is the main difference between local mode and cluster mode in Apache Spark?

easy

A. Local mode requires a cluster manager; cluster mode does not.

B. Local mode is used only for streaming data; cluster mode is for batch data.

C. Local mode runs Spark on one computer; cluster mode runs Spark on many computers.

D. Local mode uses more memory than cluster mode.

Solution

1. Step 1: Understand local vs cluster mode

   Local mode runs Spark on a single machine, suitable for small data and testing. Cluster mode runs Spark across multiple machines to handle big data.

2. Final Answer:

   Local mode runs Spark on one computer; cluster mode runs Spark on many computers. -> Option C

3. Quick Check:

   Local = single machine, Cluster = multiple machines [OK]

Hint: Local = one computer, cluster = many computers [OK]

Common Mistakes:

• Confusing local mode with cluster mode resource needs
• Thinking local mode requires cluster manager
• Assuming local mode is only for streaming

2. Which of the following is the correct way to set Spark to run in local mode on all CPU cores using SparkSession in Python?

easy

A. spark = SparkSession.builder.master('remote').getOrCreate()

B. spark = SparkSession.builder.master('cluster').getOrCreate()

C. spark = SparkSession.builder.master('local').getOrCreate()

D. spark = SparkSession.builder.master('local[*]').getOrCreate()

Solution

1. Step 1: Recall local mode syntax for all cores

   Local mode can be set with 'local[*]' to use all CPU cores. The correct syntax is spark = SparkSession.builder.master('local[*]').getOrCreate().

2. Final Answer:

   spark = SparkSession.builder.master('local[*]').getOrCreate() -> Option D

3. Quick Check:

   Use 'local[*]' for local mode with all cores [OK]

Hint: Use 'local[*]' to run Spark locally on all CPU cores [OK]

Common Mistakes:

• Using 'cluster' or 'remote' as master for local mode
• Forgetting to call getOrCreate()
• Using 'local' without brackets when multiple cores needed

3. Given this code snippet, what will be the output if Spark is run in local mode?

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local').appName('TestApp').getOrCreate()
data = [('Alice', 30), ('Bob', 25)]
df = spark.createDataFrame(data, ['Name', 'Age'])
df.show()

medium

A. A table showing two rows: Alice 30 and Bob 25

B. An error because local mode cannot create DataFrame

C. Empty output with no rows

D. A table showing only one row: Alice 30

Solution

1. Step 1: Analyze code in local mode

   Local mode runs Spark on one machine and supports DataFrame creation and display. The code creates a DataFrame with two rows: ('Alice', 30) and ('Bob', 25), and df.show() prints both rows as a table.

2. Final Answer:

   A table showing two rows: Alice 30 and Bob 25 -> Option A

3. Quick Check:

   Local mode supports DataFrame and df.show() prints all rows [OK]

Hint: df.show() prints all DataFrame rows in local mode [OK]

Common Mistakes:

• Thinking local mode cannot create DataFrames
• Expecting empty output without data
• Assuming only one row prints by default

4. You wrote this code to run Spark in cluster mode but get an error:

spark = SparkSession.builder.master('local').appName('ClusterApp').getOrCreate()

What is the likely mistake?

medium

A. Not calling getOrCreate() at the end

B. Using 'local' instead of a cluster manager URL in master setting

C. Missing import statement for SparkSession

D. Using appName instead of clusterName

Solution

1. Step 1: Identify mistake in cluster mode setup

   Cluster mode requires a cluster manager URL like 'spark://...' or 'yarn', not 'local'. Using 'local' runs Spark locally, causing the cluster setup to fail.

2. Final Answer:

   Using 'local' instead of a cluster manager URL in master setting -> Option B

3. Quick Check:

   Cluster mode needs cluster manager URL, not 'local' [OK]

Hint: Cluster mode needs cluster manager URL, not 'local' [OK]

Common Mistakes:

• Confusing local and cluster master settings
• Forgetting to import SparkSession
• Missing getOrCreate() call

5. You want to test your Spark code on your laptop before running it on a big cluster. Which setting should you use for the master parameter to simulate cluster behavior locally using all CPU cores?

hard

A. 'local[*]'

B. 'yarn-client'

C. 'spark://master:7077'

D. 'local[1]'

Solution

1. Step 1: Select local setting to simulate cluster

   'local[1]' uses one core; 'local[*]' uses all CPU cores on your laptop to simulate parallelism like a cluster locally.

2. Final Answer:

   'local[*]' -> Option A

3. Quick Check:

   'local[*]' uses all cores to mimic cluster parallelism [OK]

Hint: Use 'local[*]' to simulate cluster with all cores locally [OK]

Common Mistakes:

• Choosing 'local[1]' which uses only one core
• Using actual cluster URLs when testing locally
• Confusing 'yarn-client' with local mode