Local mode vs cluster mode in Apache Spark - Practice Questions

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Spark Execution Modes

Which statement correctly describes the difference between Spark's local mode and cluster mode?

A Local mode is used only for production, while cluster mode is only for testing and development.

B Local mode requires a cluster manager, but cluster mode runs without any cluster manager.

C Local mode runs Spark on a single machine using one JVM, while cluster mode runs Spark across multiple machines with distributed resources.

D Local mode supports distributed data processing, but cluster mode processes data only on a single node.

💡 Hint

Think about how many machines and JVMs are involved in each mode.

Local mode runs Spark on one machine using a single JVM, suitable for testing or small data. Cluster mode distributes tasks across multiple machines, enabling large-scale data processing.

❓ Predict Output

intermediate

Spark Master URL in Local Mode

What will be the output of the following Spark configuration code snippet?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local[4]').appName('TestApp').getOrCreate()
print(spark.sparkContext.master)

A local[4]

B yarn

C spark://master:7077

D local[*]

💡 Hint

Check the master URL string passed to the builder.

The master URL 'local[4]' means Spark runs locally with 4 worker threads. The print statement outputs this exact string.

❓ data_output

advanced

Data Partitioning in Local vs Cluster Mode

Consider this Spark code that creates an RDD and counts partitions:

rdd = spark.sparkContext.parallelize(range(10), 3)
print(rdd.getNumPartitions())

What will be the output when running in local mode and cluster mode respectively?

A 1 in local mode, 3 in cluster mode

B 3 in both local and cluster mode

C 3 in local mode, 1 in cluster mode

D 1 in both local and cluster mode

💡 Hint

The number of partitions is set explicitly in the parallelize call.

The number of partitions is controlled by the second argument to parallelize. It is 3 regardless of execution mode.

🔧 Debug

advanced

Error in Cluster Mode Spark Job Submission

A user submits a Spark job with master URL 'local' but expects it to run on a cluster. The job fails with connection errors. What is the most likely cause?

A The Spark version is incompatible with the cluster mode.

B The cluster manager is down, causing connection errors.

C The application name is missing, causing the job to fail.

D Using 'local' master URL runs Spark only on the local machine, so it cannot connect to the cluster resources.

💡 Hint

Check what the 'local' master URL means for Spark execution.

The 'local' master URL tells Spark to run locally on one machine. It does not connect to any cluster, so connection errors occur if cluster resources are expected.

🚀 Application

expert

Choosing Execution Mode for a Large Dataset

You have a dataset of 5 TB stored on HDFS and want to run a Spark job to analyze it. Which execution mode should you choose and why?

A Cluster mode, because it distributes the workload across multiple machines to handle large data efficiently.

B Local mode, because it is simpler and faster for large datasets.

C Local mode, because it uses multiple JVMs to speed up processing.

D Cluster mode, because it runs Spark on a single machine with more memory.

💡 Hint

Think about the size of data and how Spark handles distributed processing.

Cluster mode is designed to process large datasets by distributing tasks across many machines. Local mode runs on one machine and is not suitable for very large data.

Practice

1. What is the main difference between local mode and cluster mode in Apache Spark?

easy

A. Local mode requires a cluster manager; cluster mode does not.

B. Local mode is used only for streaming data; cluster mode is for batch data.

C. Local mode runs Spark on one computer; cluster mode runs Spark on many computers.

D. Local mode uses more memory than cluster mode.

Solution

1. Step 1: Understand local vs cluster mode

   Local mode runs Spark on a single machine, suitable for small data and testing. Cluster mode runs Spark across multiple machines to handle big data.

2. Final Answer:

   Local mode runs Spark on one computer; cluster mode runs Spark on many computers. -> Option C

3. Quick Check:

   Local = single machine, Cluster = multiple machines [OK]

Hint: Local = one computer, cluster = many computers [OK]

Common Mistakes:

• Confusing local mode with cluster mode resource needs
• Thinking local mode requires cluster manager
• Assuming local mode is only for streaming

2. Which of the following is the correct way to set Spark to run in local mode on all CPU cores using SparkSession in Python?

easy

A. spark = SparkSession.builder.master('remote').getOrCreate()

B. spark = SparkSession.builder.master('cluster').getOrCreate()

C. spark = SparkSession.builder.master('local').getOrCreate()

D. spark = SparkSession.builder.master('local[*]').getOrCreate()

Solution

1. Step 1: Recall local mode syntax for all cores

   Local mode can be set with 'local[*]' to use all CPU cores. The correct syntax is spark = SparkSession.builder.master('local[*]').getOrCreate().

2. Final Answer:

   spark = SparkSession.builder.master('local[*]').getOrCreate() -> Option D

3. Quick Check:

   Use 'local[*]' for local mode with all cores [OK]

Hint: Use 'local[*]' to run Spark locally on all CPU cores [OK]

Common Mistakes:

• Using 'cluster' or 'remote' as master for local mode
• Forgetting to call getOrCreate()
• Using 'local' without brackets when multiple cores needed

3. Given this code snippet, what will be the output if Spark is run in local mode?

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local').appName('TestApp').getOrCreate()
data = [('Alice', 30), ('Bob', 25)]
df = spark.createDataFrame(data, ['Name', 'Age'])
df.show()

medium

A. A table showing two rows: Alice 30 and Bob 25

B. An error because local mode cannot create DataFrame

C. Empty output with no rows

D. A table showing only one row: Alice 30

Solution

1. Step 1: Analyze code in local mode

   Local mode runs Spark on one machine and supports DataFrame creation and display. The code creates a DataFrame with two rows: ('Alice', 30) and ('Bob', 25), and df.show() prints both rows as a table.

2. Final Answer:

   A table showing two rows: Alice 30 and Bob 25 -> Option A

3. Quick Check:

   Local mode supports DataFrame and df.show() prints all rows [OK]

Hint: df.show() prints all DataFrame rows in local mode [OK]

Common Mistakes:

• Thinking local mode cannot create DataFrames
• Expecting empty output without data
• Assuming only one row prints by default

4. You wrote this code to run Spark in cluster mode but get an error:

spark = SparkSession.builder.master('local').appName('ClusterApp').getOrCreate()

What is the likely mistake?

medium

A. Not calling getOrCreate() at the end

B. Using 'local' instead of a cluster manager URL in master setting

C. Missing import statement for SparkSession

D. Using appName instead of clusterName

Solution

1. Step 1: Identify mistake in cluster mode setup

   Cluster mode requires a cluster manager URL like 'spark://...' or 'yarn', not 'local'. Using 'local' runs Spark locally, causing the cluster setup to fail.

2. Final Answer:

   Using 'local' instead of a cluster manager URL in master setting -> Option B

3. Quick Check:

   Cluster mode needs cluster manager URL, not 'local' [OK]

Hint: Cluster mode needs cluster manager URL, not 'local' [OK]

Common Mistakes:

• Confusing local and cluster master settings
• Forgetting to import SparkSession
• Missing getOrCreate() call

5. You want to test your Spark code on your laptop before running it on a big cluster. Which setting should you use for the master parameter to simulate cluster behavior locally using all CPU cores?

hard

A. 'local[*]'

B. 'yarn-client'

C. 'spark://master:7077'

D. 'local[1]'

Solution

1. Step 1: Select local setting to simulate cluster

   'local[1]' uses one core; 'local[*]' uses all CPU cores on your laptop to simulate parallelism like a cluster locally.

2. Final Answer:

   'local[*]' -> Option A

3. Quick Check:

   'local[*]' uses all cores to mimic cluster parallelism [OK]

Hint: Use 'local[*]' to simulate cluster with all cores locally [OK]

Common Mistakes:

• Choosing 'local[1]' which uses only one core
• Using actual cluster URLs when testing locally
• Confusing 'yarn-client' with local mode