Local mode vs cluster mode in Apache Spark - Practice Questions

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Spark Execution Modes

Which statement correctly describes the difference between Spark's local mode and cluster mode?

A Local mode is used only for production, while cluster mode is only for testing and development.

B Local mode requires a cluster manager, but cluster mode runs without any cluster manager.

C Local mode runs Spark on a single machine using one JVM, while cluster mode runs Spark across multiple machines with distributed resources.

D Local mode supports distributed data processing, but cluster mode processes data only on a single node.

💡 Hint

Think about how many machines and JVMs are involved in each mode.

Local mode runs Spark on one machine using a single JVM, suitable for testing or small data. Cluster mode distributes tasks across multiple machines, enabling large-scale data processing.

❓ Predict Output

intermediate

Spark Master URL in Local Mode

What will be the output of the following Spark configuration code snippet?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local[4]').appName('TestApp').getOrCreate()
print(spark.sparkContext.master)

A local[4]

B yarn

C spark://master:7077

D local[*]

💡 Hint

Check the master URL string passed to the builder.

The master URL 'local[4]' means Spark runs locally with 4 worker threads. The print statement outputs this exact string.

❓ data_output

advanced

Data Partitioning in Local vs Cluster Mode

Consider this Spark code that creates an RDD and counts partitions:

rdd = spark.sparkContext.parallelize(range(10), 3)
print(rdd.getNumPartitions())

What will be the output when running in local mode and cluster mode respectively?

A 1 in local mode, 3 in cluster mode

B 3 in both local and cluster mode

C 3 in local mode, 1 in cluster mode

D 1 in both local and cluster mode

💡 Hint

The number of partitions is set explicitly in the parallelize call.

The number of partitions is controlled by the second argument to parallelize. It is 3 regardless of execution mode.

🔧 Debug

advanced

Error in Cluster Mode Spark Job Submission

A user submits a Spark job with master URL 'local' but expects it to run on a cluster. The job fails with connection errors. What is the most likely cause?

A The Spark version is incompatible with the cluster mode.

B The cluster manager is down, causing connection errors.

C The application name is missing, causing the job to fail.

D Using 'local' master URL runs Spark only on the local machine, so it cannot connect to the cluster resources.

💡 Hint

Check what the 'local' master URL means for Spark execution.

The 'local' master URL tells Spark to run locally on one machine. It does not connect to any cluster, so connection errors occur if cluster resources are expected.

🚀 Application

expert

Choosing Execution Mode for a Large Dataset

You have a dataset of 5 TB stored on HDFS and want to run a Spark job to analyze it. Which execution mode should you choose and why?

A Cluster mode, because it distributes the workload across multiple machines to handle large data efficiently.

B Local mode, because it is simpler and faster for large datasets.

C Local mode, because it uses multiple JVMs to speed up processing.

D Cluster mode, because it runs Spark on a single machine with more memory.

💡 Hint

Think about the size of data and how Spark handles distributed processing.

Cluster mode is designed to process large datasets by distributing tasks across many machines. Local mode runs on one machine and is not suitable for very large data.