Monitoring with Spark History Server in Apache Spark - Mini Project: Build & Apply

Monitoring with Spark History Server

📖 Scenario: You are a data engineer working with Apache Spark. You want to monitor your Spark jobs after they finish by using the Spark History Server. This helps you see job details like stages, tasks, and performance metrics.

🎯 Goal: Set up a simple Spark application log directory, configure Spark to save event logs there, run a Spark job to generate logs, and then verify the logs are available for the Spark History Server to display.

📋 What You'll Learn

Create a directory path variable for Spark event logs

Configure Spark to enable event logging and set the log directory

Run a simple Spark job that generates event logs

Print the path where Spark event logs are saved

💡 Why This Matters

🌍 Real World

Spark History Server is used in real data engineering to monitor and debug Spark jobs after they finish, helping teams improve performance and troubleshoot issues.

💼 Career

Knowing how to configure and use Spark History Server is important for roles like Data Engineer, Big Data Developer, and DevOps Engineer working with Spark clusters.

1

Create Spark Event Log Directory Variable

Create a variable called event_log_dir and set it to the string "/tmp/spark-events" which will be used as the directory to store Spark event logs.

Apache Spark

# Create a variable called event_log_dir and set it to "/tmp/spark-events"
# Your code here

Hint

Use a simple string assignment to create the variable.

2

Configure Spark to Enable Event Logging

Create a dictionary called spark_conf with two entries: "spark.eventLog.enabled" set to "true" and "spark.eventLog.dir" set to the variable event_log_dir.

Apache Spark

event_log_dir = "/tmp/spark-events"
# Create a dictionary called spark_conf with keys "spark.eventLog.enabled" and "spark.eventLog.dir"
# Your code here

Hint

Use a dictionary with exact keys and values as strings.

3

Run a Simple Spark Job to Generate Logs

Using PySpark, create a SparkSession with the configuration spark_conf. Then create an RDD from the list [1, 2, 3, 4, 5], map each number to its square, and collect the results into a variable called squares.

Apache Spark

event_log_dir = "/tmp/spark-events"
spark_conf = {
    "spark.eventLog.enabled": "true",
    "spark.eventLog.dir": event_log_dir
}
# Create SparkSession with spark_conf
# Create RDD from [1, 2, 3, 4, 5]
# Map each number to its square
# Collect results into squares
# Your code here

Hint

Use SparkSession builder with config, then create RDD and map.

4

Print the Spark Event Log Directory

Write a print statement to display the text "Spark event logs are saved in: " followed by the value of the variable event_log_dir.

Apache Spark

from pyspark.sql import SparkSession

event_log_dir = "/tmp/spark-events"
spark_conf = {
    "spark.eventLog.enabled": "true",
    "spark.eventLog.dir": event_log_dir
}

spark = SparkSession.builder.appName("MonitorExample")
for k, v in spark_conf.items():
    spark = spark.config(k, v)
spark = spark.getOrCreate()

rdd = spark.sparkContext.parallelize([1, 2, 3, 4, 5])
squares = rdd.map(lambda x: x * x).collect()
# Print the event_log_dir with a message
# Your code here

Hint

Use print with string concatenation or f-string.

Practice

1. What is the main purpose of the Spark History Server in monitoring Spark applications?

easy

A. To monitor real-time streaming data only

B. To view details of completed Spark jobs using saved event logs

C. To configure Spark cluster resources dynamically

D. To submit new Spark jobs to the cluster

Solution

1. Step 1: Understand Spark History Server role

   The Spark History Server reads event logs from completed jobs to provide detailed information about them.

2. Step 2: Differentiate from other Spark components

   Unlike job submission or real-time monitoring, it focuses on past job performance using saved logs.

3. Final Answer:

   To view details of completed Spark jobs using saved event logs -> Option B

4. Quick Check:

   Spark History Server = View completed job details [OK]

Hint: History Server shows past jobs, not live or submission tasks [OK]

Common Mistakes:

• Confusing History Server with job submission tools
• Thinking it monitors only live streaming jobs
• Assuming it manages cluster resources

2. Which Spark configuration property must be set to enable event logging for the History Server?

easy

A. spark.sql.shuffle.partitions

B. spark.history.fs.logDirectory

C. spark.executor.memory

D. spark.eventLog.enabled

Solution

1. Step 1: Identify event logging property

   The property spark.eventLog.enabled controls whether Spark saves event logs for completed jobs.

2. Step 2: Differentiate from other configs

   spark.history.fs.logDirectory sets log location, not enabling logging; others relate to memory or shuffle partitions.

3. Final Answer:

   spark.eventLog.enabled -> Option D

4. Quick Check:

   Enable event logging = spark.eventLog.enabled [OK]

Hint: Enable event logging with spark.eventLog.enabled true [OK]

Common Mistakes:

• Confusing log directory with enabling logging
• Choosing unrelated Spark properties
• Assuming memory or shuffle configs enable logging

3. Given the configuration:

spark.eventLog.enabled=true
spark.history.fs.logDirectory=/tmp/spark-events

What will happen after a Spark job finishes?

medium

A. Event logs will be saved to /tmp/spark-events for History Server access

B. No event logs will be saved because logging is disabled

C. Logs will be saved but History Server cannot read them

D. Spark job will fail due to invalid log directory

Solution

1. Step 1: Check event logging enabled

   Since spark.eventLog.enabled=true, Spark saves event logs after job completion.

2. Step 2: Confirm log directory setting

   The logs are saved to /tmp/spark-events, which the History Server reads to show job details.

3. Final Answer:

   Event logs will be saved to /tmp/spark-events for History Server access -> Option A

4. Quick Check:

   Logging enabled + valid directory = logs saved [OK]

Hint: True eventLog.enabled + valid directory = logs saved [OK]

Common Mistakes:

• Assuming logs are not saved if directory is custom
• Thinking logging is off despite true setting
• Believing job fails due to log directory path

4. You started the Spark History Server but see no jobs listed in the UI. Which is the most likely cause?

medium

A. The Spark History Server UI port is blocked by firewall

B. Spark jobs are still running and not completed

C. spark.eventLog.enabled is set to false or event logs are missing

D. The Spark History Server service is running on the wrong machine

Solution

1. Step 1: Check event logging status

   If spark.eventLog.enabled is false or event logs are missing, History Server has no data to show.

2. Step 2: Understand UI port and job status

   Firewall blocking UI port prevents access but not empty job list; running jobs don't appear until finished.

3. Final Answer:

   spark.eventLog.enabled is set to false or event logs are missing -> Option C

4. Quick Check:

   No jobs in UI = logging off or missing logs [OK]

Hint: No jobs shown? Check if event logging is enabled [OK]

Common Mistakes:

• Assuming firewall blocks job list display
• Expecting running jobs to appear in History Server
• Thinking server machine location affects job visibility

5. You want to monitor completed Spark jobs on a cluster. Which sequence of steps correctly sets up the Spark History Server?

hard

A. Enable event logging, set log directory, start History Server, open UI

B. Start History Server, submit jobs, enable event logging, open UI

C. Set log directory, open UI, enable event logging, start History Server

D. Submit jobs, start History Server, set log directory, enable event logging

Solution

1. Step 1: Enable event logging and set log directory

   First, enable spark.eventLog.enabled=true and set spark.history.fs.logDirectory so logs are saved properly.

2. Step 2: Start History Server and open UI

   Then start the History Server to read logs and open its web UI to monitor completed jobs.

3. Final Answer:

   Enable event logging, set log directory, start History Server, open UI -> Option A

4. Quick Check:

   Enable logging + set dir + start server + open UI = monitor jobs [OK]

Hint: Enable logging first, then start History Server and open UI [OK]

Common Mistakes:

• Starting History Server before enabling logging
• Opening UI before logs exist
• Setting log directory after starting server