Spark UI for debugging performance in Apache Spark - Mini Project: Build & Apply

Spark UI for debugging performance

📖 Scenario: You are working with Apache Spark to process a large dataset. Sometimes your Spark jobs run slower than expected. To understand why, you want to use the Spark UI, a tool that helps you see what happens inside your Spark job.The Spark UI shows stages, tasks, and resource usage. By learning to read it, you can find slow parts and improve your job's speed.

🎯 Goal: Learn how to start a Spark session, run a simple job, and open the Spark UI to check the job's performance details.

📋 What You'll Learn

Create a Spark session in Python

Load a small dataset into a DataFrame

Run a simple transformation and action

Access the Spark UI URL to view job details

💡 Why This Matters

🌍 Real World

Data engineers and data scientists use Spark UI to monitor and debug large data processing jobs to ensure they run efficiently.

💼 Career

Knowing how to use Spark UI is essential for optimizing Spark applications, a key skill for big data roles.

1

Create a Spark session

Write code to create a Spark session called spark with the app name 'SparkUIDebug'.

Apache Spark

# Your code here to create a Spark session named spark

Need a hint?

Use SparkSession.builder.appName('SparkUIDebug').getOrCreate() to create the session.

2

Load a small dataset

Create a DataFrame called df by loading the JSON file 'examples/src/main/resources/people.json' using spark.read.json().

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('SparkUIDebug').getOrCreate()
# Load the JSON file into a DataFrame called df

Need a hint?

Use spark.read.json('examples/src/main/resources/people.json') to load the data.

3

Run a transformation and action

Create a new DataFrame called adults by filtering df for people with age >= 21. Then use adults.show() to display the results.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('SparkUIDebug').getOrCreate()
df = spark.read.json('examples/src/main/resources/people.json')
# Filter df for age >= 21 and assign to adults
# Show the adults DataFrame

Need a hint?

Use df.filter(df.age >= 21) to filter and adults.show() to display.

4

Access the Spark UI

Print the Spark UI web URL by accessing spark.sparkContext.uiWebUrl.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('SparkUIDebug').getOrCreate()
df = spark.read.json('examples/src/main/resources/people.json')
adults = df.filter(df.age >= 21)
adults.show()
# Print the Spark UI URL

Need a hint?

Use print(spark.sparkContext.uiWebUrl) to see the Spark UI URL.