Spark UI for debugging performance in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Spark UI Components

Which Spark UI tab helps you identify the stages where tasks are taking the longest time to complete?

A Jobs tab

B Executors tab

C Stages tab

D SQL tab

💡 Hint

Think about where you can see detailed timing for each part of the job.

The Stages tab breaks down the job into stages and shows task durations, helping identify slow stages.

❓ data_output

intermediate

Interpreting Executor Metrics

Given the Executors tab in Spark UI, which metric directly indicates how much memory is used by cached RDDs on an executor?

A Disk Used

B Storage Memory

C Input Metrics

D Shuffle Read

💡 Hint

Look for memory related to caching in the Executors tab.

Storage Memory shows the amount of memory used to cache RDDs or DataFrames on each executor.

❓ Predict Output

advanced

Analyzing Task Duration from Spark UI Logs

Consider this simplified Spark UI task duration data for a stage:

Task 1: 5 seconds
Task 2: 7 seconds
Task 3: 3 seconds
Task 4: 7 seconds
Task 5: 4 seconds

What is the median task duration shown in the Spark UI for this stage?

A 5 seconds

B 7 seconds

C 4 seconds

D 6 seconds

💡 Hint

Order the task durations and find the middle value.

Sorted durations: 3,4,5,7,7. The middle (median) is 5 seconds.

❓ visualization

advanced

Identifying Skewed Tasks in Spark UI

You see a bar chart in the Spark UI showing task durations for a stage. Most bars are around 2 seconds, but a few bars are much taller, around 20 seconds. What does this indicate?

A The stage has skewed tasks causing performance bottlenecks

B The cluster is running out of memory

C The shuffle read is very high for all tasks

D The job has completed successfully without issues

💡 Hint

Think about what uneven task durations mean for performance.

Some tasks taking much longer than others means data skew, causing slowdowns.

🚀 Application

expert

Using Spark UI to Optimize Shuffle Operations

You notice in the Spark UI that shuffle read size is very large and shuffle write time is high for a stage. Which optimization is most likely to reduce this shuffle overhead?

A Cache the input data before shuffle to speed up reads

B Reduce the number of executors to limit shuffle traffic

C Disable speculative execution to avoid duplicate tasks

D Increase the number of shuffle partitions to reduce data per partition

💡 Hint

Think about how partitioning affects shuffle data size and task parallelism.

Increasing shuffle partitions splits data into smaller chunks, reducing shuffle size per task and improving parallelism.