Spark architecture (driver, executors, cluster manager) in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Role of the Driver in Spark Architecture

What is the main responsibility of the Driver in a Spark application?

A It manages the cluster resources and allocates executors to applications.

B It coordinates the execution by maintaining the SparkContext and scheduling tasks.

C It executes the tasks on worker nodes and processes data partitions.

D It stores the data permanently in the distributed file system.

💡 Hint

Think about which component controls the flow and task scheduling in Spark.

The Driver is the central coordinator that creates the SparkContext, converts user code into tasks, and schedules them on executors.

🧠 Conceptual

intermediate

Function of Executors in Spark

Which of the following best describes the role of Executors in Spark?

A They run tasks and store data for the application during execution.

B They monitor the health of the Spark driver.

C They submit Spark jobs to the cluster manager.

D They allocate resources and manage the cluster nodes.

💡 Hint

Executors are like workers that do the actual data processing.

Executors run the tasks assigned by the driver and hold data in memory or disk during the job execution.

🧠 Conceptual

advanced

Cluster Manager's Role in Spark

What is the primary role of the Cluster Manager in Spark?

A It manages the allocation of resources across multiple Spark applications.

B It executes the Spark driver program on the client machine.

C It schedules tasks within executors and manages data shuffling.

D It stores the Spark application logs and metrics.

💡 Hint

Think about who controls resource distribution for all applications running on the cluster.

The Cluster Manager allocates resources like CPU and memory to Spark applications and manages multiple applications running on the cluster.

❓ data_output

advanced

Output of Spark Executor Task Count

Consider a Spark application with 3 executors, each running 4 cores. If the application runs a job with 12 tasks, how many tasks will each executor run assuming tasks are evenly distributed?

Apache Spark

executors = 3
cores_per_executor = 4
tasks = 12

# Calculate tasks per executor assuming even distribution
tasks_per_executor = tasks // executors
print(tasks_per_executor)

A 7

B 3

C 12

D 4

💡 Hint

Divide total tasks by number of executors for even distribution.

With 12 tasks and 3 executors, each executor runs 12 / 3 = 4 tasks.

🧠 Conceptual

expert

Impact of Driver Failure in Spark Cluster

What happens if the Driver fails during a Spark job execution in a cluster mode?

A The cluster manager automatically restarts the driver and resumes the job from the last checkpoint.

B The executors continue running and complete the job without interruption.

C The entire Spark application fails and all executors stop processing tasks.

D The driver failure only affects logging but the job continues running.

💡 Hint

Consider the driver's role in task scheduling and coordination.

The driver coordinates the job execution. If it fails, executors lose instructions and the job stops.