Monitoring with Spark History Server in Apache Spark - Step-by-Step Execution

Concept Flow - Monitoring with Spark History Server

Spark Job Runs

↓

Event Logs Generated

↓

Logs Stored in Configured Directory

↓

Spark History Server Reads Logs

↓

User Accesses History Server UI

↓

View Job Details, Stages, Executors, Metrics

Spark jobs produce event logs stored in a directory. The Spark History Server reads these logs and shows job details in its web UI for monitoring.

Execution Sample

Apache Spark

spark.eventLog.enabled=true
spark.eventLog.dir=hdfs:///spark-logs
start-history-server.sh
Access UI at http://localhost:18080

Enable event logging, set log directory, start Spark History Server, then open its web UI to monitor past jobs.

Execution Table

Step	Action	System State	Output/Result
1	Spark job runs	Job executes, event logs start recording	Event logs created in memory
2	Job finishes	Event logs saved to configured directory	Logs stored at hdfs:///spark-logs/application_12345
3	Start Spark History Server	Server reads logs from hdfs:///spark-logs	History Server running, ready to serve UI
4	User opens History Server UI	UI loads job list from logs	List of completed jobs displayed
5	User selects a job	UI loads job details from logs	Stages, tasks, executors info shown
6	User views metrics	UI shows job duration, shuffle read/write, errors	Detailed monitoring data visible
7	No new logs	No updates in UI until new jobs run	UI shows last available job data

💡 Execution stops when no new event logs are generated or user closes the UI

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	Final
Event Logs	None	Saved to hdfs:///spark-logs	Read by History Server	Loaded in UI job list	Loaded job details	Displayed in UI
History Server	Stopped	Stopped	Running	Running	Running	Running
UI State	Not accessed	Not accessed	Not accessed	Job list shown	Job details shown	Metrics shown

Key Moments - 3 Insights

Why don't I see my running job in the History Server UI?

The History Server only shows completed jobs with saved event logs (see execution_table step 2 and 4). Running jobs appear in Spark's live UI, not the History Server.

What happens if event logging is not enabled?

No event logs are created (execution_table step 1), so the History Server has no data to read and will show no jobs (step 3 and 4).

How does the History Server get updated with new jobs?

It reads new event logs saved in the configured directory (step 2 and 3). It does not update in real-time but refreshes when new logs appear.

Visual Quiz - 3 Questions

Look at the execution table, at which step does the History Server start reading event logs?

A Step 2

B Step 3

C Step 4

D Step 1

💡 Hint

Check the 'Action' and 'System State' columns in step 3 where the server reads logs.

According to the variable tracker, what is the state of 'Event Logs' after step 4?

A None

B Saved to hdfs:///spark-logs

C Loaded in UI job list

D Displayed in UI

💡 Hint

Look at the 'Event Logs' row under 'After Step 4' column.

If event logging is disabled, what will the History Server UI show?

A No jobs available

B Running jobs only

C List of completed jobs

D Error message

💡 Hint

Refer to key moment about event logging and execution_table steps 1 and 4.

Concept Snapshot

Enable spark.eventLog.enabled=true to record job events.
Set spark.eventLog.dir to store logs.
Start Spark History Server to read logs.
Access UI at http://localhost:18080 to monitor completed jobs.
History Server shows job details, stages, and metrics from saved logs.

Full Transcript

When you run a Spark job, it can create event logs if logging is enabled. These logs are saved in a directory you set, like hdfs:///spark-logs. The Spark History Server reads these logs to show you past job details. You start the History Server, then open its web UI to see a list of completed jobs. Selecting a job shows stages, tasks, and metrics. The History Server only shows jobs that finished and saved logs. If logging is off, no jobs appear. The UI updates when new logs are saved after jobs finish.

Practice

1. What is the main purpose of the Spark History Server in monitoring Spark applications?

easy

A. To monitor real-time streaming data only

B. To view details of completed Spark jobs using saved event logs

C. To configure Spark cluster resources dynamically

D. To submit new Spark jobs to the cluster

Solution

1. Step 1: Understand Spark History Server role

   The Spark History Server reads event logs from completed jobs to provide detailed information about them.

2. Step 2: Differentiate from other Spark components

   Unlike job submission or real-time monitoring, it focuses on past job performance using saved logs.

3. Final Answer:

   To view details of completed Spark jobs using saved event logs -> Option B

4. Quick Check:

   Spark History Server = View completed job details [OK]

Hint: History Server shows past jobs, not live or submission tasks [OK]

Common Mistakes:

• Confusing History Server with job submission tools
• Thinking it monitors only live streaming jobs
• Assuming it manages cluster resources

2. Which Spark configuration property must be set to enable event logging for the History Server?

easy

A. spark.sql.shuffle.partitions

B. spark.history.fs.logDirectory

C. spark.executor.memory

D. spark.eventLog.enabled

Solution

1. Step 1: Identify event logging property

   The property spark.eventLog.enabled controls whether Spark saves event logs for completed jobs.

2. Step 2: Differentiate from other configs

   spark.history.fs.logDirectory sets log location, not enabling logging; others relate to memory or shuffle partitions.

3. Final Answer:

   spark.eventLog.enabled -> Option D

4. Quick Check:

   Enable event logging = spark.eventLog.enabled [OK]

Hint: Enable event logging with spark.eventLog.enabled true [OK]

Common Mistakes:

• Confusing log directory with enabling logging
• Choosing unrelated Spark properties
• Assuming memory or shuffle configs enable logging

3. Given the configuration:

spark.eventLog.enabled=true
spark.history.fs.logDirectory=/tmp/spark-events

What will happen after a Spark job finishes?

medium

A. Event logs will be saved to /tmp/spark-events for History Server access

B. No event logs will be saved because logging is disabled

C. Logs will be saved but History Server cannot read them

D. Spark job will fail due to invalid log directory

Solution

1. Step 1: Check event logging enabled

   Since spark.eventLog.enabled=true, Spark saves event logs after job completion.

2. Step 2: Confirm log directory setting

   The logs are saved to /tmp/spark-events, which the History Server reads to show job details.

3. Final Answer:

   Event logs will be saved to /tmp/spark-events for History Server access -> Option A

4. Quick Check:

   Logging enabled + valid directory = logs saved [OK]

Hint: True eventLog.enabled + valid directory = logs saved [OK]

Common Mistakes:

• Assuming logs are not saved if directory is custom
• Thinking logging is off despite true setting
• Believing job fails due to log directory path

4. You started the Spark History Server but see no jobs listed in the UI. Which is the most likely cause?

medium

A. The Spark History Server UI port is blocked by firewall

B. Spark jobs are still running and not completed

C. spark.eventLog.enabled is set to false or event logs are missing

D. The Spark History Server service is running on the wrong machine

Solution

1. Step 1: Check event logging status

   If spark.eventLog.enabled is false or event logs are missing, History Server has no data to show.

2. Step 2: Understand UI port and job status

   Firewall blocking UI port prevents access but not empty job list; running jobs don't appear until finished.

3. Final Answer:

   spark.eventLog.enabled is set to false or event logs are missing -> Option C

4. Quick Check:

   No jobs in UI = logging off or missing logs [OK]

Hint: No jobs shown? Check if event logging is enabled [OK]

Common Mistakes:

• Assuming firewall blocks job list display
• Expecting running jobs to appear in History Server
• Thinking server machine location affects job visibility

5. You want to monitor completed Spark jobs on a cluster. Which sequence of steps correctly sets up the Spark History Server?

hard

A. Enable event logging, set log directory, start History Server, open UI

B. Start History Server, submit jobs, enable event logging, open UI

C. Set log directory, open UI, enable event logging, start History Server

D. Submit jobs, start History Server, set log directory, enable event logging

Solution

1. Step 1: Enable event logging and set log directory

   First, enable spark.eventLog.enabled=true and set spark.history.fs.logDirectory so logs are saved properly.

2. Step 2: Start History Server and open UI

   Then start the History Server to read logs and open its web UI to monitor completed jobs.

3. Final Answer:

   Enable event logging, set log directory, start History Server, open UI -> Option A

4. Quick Check:

   Enable logging + set dir + start server + open UI = monitor jobs [OK]

Hint: Enable logging first, then start History Server and open UI [OK]

Common Mistakes:

• Starting History Server before enabling logging
• Opening UI before logs exist
• Setting log directory after starting server