Spark UI for debugging performance in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the Spark UI used for?

The Spark UI is a web interface that helps you see how your Spark jobs run. It shows details about tasks, stages, and executors to help find performance problems.

beginner

What does the 'Stages' tab in Spark UI show?

The 'Stages' tab shows each step of your Spark job. It displays how long each stage took, how many tasks succeeded or failed, and helps find slow parts.

intermediate

How can the 'Executors' tab help debug performance?

The 'Executors' tab shows details about each worker node running tasks. You can see memory use, CPU time, and if any executors are slow or failing.

intermediate

What information does the 'SQL' tab provide in Spark UI?

The 'SQL' tab shows details about SQL queries run by Spark. It helps find slow queries and shows query plans to optimize performance.

intermediate

Why is it important to check 'Task Time' and 'Shuffle Read/Write' in Spark UI?

Task Time shows how long each task takes. Shuffle Read/Write shows data moved between nodes. High shuffle or long tasks can slow jobs, so checking these helps improve speed.

Which Spark UI tab shows the progress of each job step?

A Environment

B Executors

C SQL

D Stages

The 'Stages' tab shows each step or stage of a Spark job and its progress.

What does high shuffle read/write in Spark UI usually indicate?

A Lots of data moving between nodes

B Low memory usage

C Fast task completion

D No data movement

High shuffle read/write means a lot of data is moving between nodes, which can slow down the job.

Where can you find memory and CPU usage of worker nodes in Spark UI?

A SQL tab

B Stages tab

C Executors tab

D Jobs tab

The 'Executors' tab shows memory and CPU usage for each worker node.

What is a common sign of a slow Spark job in the UI?

A No tasks shown

B Long task times and many failed tasks

C All tasks finish instantly

D No shuffle data

Long task times and many failed tasks usually mean the job is slow or has problems.

Which tab helps optimize SQL queries in Spark?

A SQL

B Executors

C Storage

D Environment

The 'SQL' tab shows query plans and performance details to help optimize SQL queries.

Explain how you would use the Spark UI to find why a job is running slowly.

Think about what parts of the UI show task details and resource use.

Describe the role of the Executors tab in debugging Spark performance.

Focus on what information about workers the tab provides.