Monitoring with Spark History Server in Apache Spark - Deep Dive

Overview - Monitoring with Spark History Server

What is it?

The Spark History Server is a tool that helps you look at past runs of your Apache Spark jobs. It collects and shows detailed information about how your jobs performed, like how long they took and what resources they used. This helps you understand and improve your Spark applications by reviewing their history. It works by reading event logs that Spark saves during job execution.

Why it matters

Without the Spark History Server, you would have no easy way to see what happened in your past Spark jobs after they finish. This makes it hard to find problems or improve performance. By using the History Server, you can spot slow tasks, resource bottlenecks, or errors, which helps save time and computing costs. It makes Spark job monitoring clear and accessible, even after the job is done.

Where it fits

Before learning about the Spark History Server, you should understand basic Spark concepts like jobs, stages, and tasks, and how Spark runs on clusters. After mastering the History Server, you can explore advanced Spark monitoring tools, real-time metrics with Spark UI, and integrating Spark logs with external monitoring systems like Prometheus or Grafana.

Mental Model

Core Idea

The Spark History Server is like a replay tool that lets you watch and analyze the detailed story of your Spark jobs after they have finished running.

Think of it like...

Imagine you recorded a sports game and later watch the replay to see what went well or wrong. The History Server is that replay for your Spark jobs, showing every play and player performance.

┌───────────────────────────────┐
│       Spark Job Runs          │
│  (executed on cluster nodes) │
└─────────────┬─────────────────┘
              │ Event Logs (JSON files)
              ▼
┌───────────────────────────────┐
│     Spark History Server       │
│  Reads event logs from storage │
│  Displays UI with job details  │
└─────────────┬─────────────────┘
              │ User accesses via web browser
              ▼
┌───────────────────────────────┐
│       Job Performance UI       │
│  Stages, tasks, timelines,     │
│  resource usage, errors        │
└───────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

What is Spark History Server

Concept: Introduction to the Spark History Server and its purpose.

The Spark History Server is a web application that shows the details of completed Spark jobs. It reads event logs that Spark writes during job execution and presents them in an easy-to-understand interface. This helps users review job performance after the job finishes.

Result

You understand that the History Server is a tool to view past Spark job details through saved logs.

Knowing that Spark saves event logs and that the History Server reads these logs helps you see how Spark keeps track of job history for later analysis.

2

Foundation

How Spark Event Logs Work

Concept: Understanding the event logs that Spark generates and stores.

When Spark runs a job, it creates event logs that record important events like job start, stage completion, and task failures. These logs are saved as files in a configured storage location, such as HDFS or local disk. The History Server reads these logs to reconstruct the job's execution details.

Result

You know where Spark stores job event logs and what kind of information they contain.

Understanding event logs is key because the History Server depends entirely on these logs to show job history.

3

Intermediate

Setting Up Spark History Server

🤔Before reading on: Do you think the History Server needs to run on every Spark node or just one central place? Commit to your answer.

Concept: How to configure and start the Spark History Server to read event logs.

To use the History Server, you configure Spark to save event logs by setting 'spark.eventLog.enabled' to true and specifying a directory with 'spark.eventLog.dir'. Then, you start the History Server process pointing it to this directory. The server runs as a web app accessible via a browser, showing all completed jobs found in the logs.

Result

You can start the History Server and access a web UI listing past Spark jobs.

Knowing that the History Server is a separate service that reads logs from a shared location helps you design your Spark cluster monitoring setup.

4

Intermediate

Navigating the History Server UI

🤔Before reading on: Do you think the UI shows only job summaries or detailed task-level info? Commit to your answer.

Concept: Exploring the different views and metrics available in the History Server interface.

The History Server UI shows a list of completed jobs with start time and duration. Clicking a job reveals stages and tasks with timelines, shuffle read/write, and executor usage. You can see failed tasks, job DAGs, and environment details. This helps diagnose performance issues or errors.

Result

You can interpret the UI to find slow stages, resource bottlenecks, or task failures.

Understanding the UI layout and metrics lets you quickly pinpoint where Spark jobs spend time or fail.

5

Intermediate

Event Log Storage and Retention

Concept: Managing where logs are stored and how long they are kept.

Event logs can be stored on HDFS, S3, or local disk. You must ensure enough storage space and decide how long to keep logs, as they grow over time. Cleaning old logs prevents storage overload. You can configure retention policies or automate cleanup.

Result

You know how to manage event log storage to keep the History Server efficient.

Knowing log storage limits helps maintain a responsive History Server and avoids losing important job history.

6

Advanced

Integrating History Server with Cluster Managers

🤔Before reading on: Does the History Server depend on the cluster manager to run? Commit to your answer.

Concept: How the History Server works alongside cluster managers like YARN or Kubernetes.

The History Server is independent of cluster managers but complements them. It reads logs from shared storage accessible by all nodes. On YARN, logs are often stored in HDFS; on Kubernetes, you might use persistent volumes or object storage. Proper integration ensures logs are available for the History Server after job completion.

Result

You understand how to configure the History Server in different cluster environments.

Knowing the decoupling from cluster managers clarifies deployment options and troubleshooting.

7

Expert

Performance and Scalability of History Server

🤔Before reading on: Do you think the History Server loads all logs at once or streams them on demand? Commit to your answer.

Concept: How the History Server handles many large event logs and scales for big clusters.

The History Server loads event logs on demand, caching parsed data to improve speed. For very large clusters with many jobs, it can become slow or use much memory. Techniques like log compression, partitioning logs by date, or using multiple History Servers can help. Understanding internal caching and parsing helps optimize performance.

Result

You can plan for scaling the History Server and avoid performance bottlenecks.

Knowing internal data handling prevents surprises in large production environments and guides tuning.

Under the Hood

Spark jobs emit events during execution, such as job start, stage completion, and task metrics. These events are serialized into JSON and written to event log files in a configured directory. The Spark History Server reads these files, parses the JSON events, and reconstructs the job execution timeline and metrics. It stores parsed data in memory or disk cache to serve the web UI efficiently. The UI then presents this data as interactive tables, timelines, and graphs.

Why designed this way?

The History Server was designed to separate live job monitoring from post-job analysis. Writing event logs asynchronously during job execution avoids slowing down the job. Storing logs externally allows the History Server to be restarted or run independently. This design balances performance, fault tolerance, and usability. Alternatives like real-time monitoring exist but do not provide the same detailed historical view.

┌───────────────┐       ┌─────────────────────┐       ┌───────────────────────┐
│ Spark Job Run │──────▶│ Event Log (JSON file)│──────▶│ Spark History Server   │
│ (executors)   │       │ stored in shared     │       │ Reads and parses logs │
└───────────────┘       │ storage (HDFS, S3)   │       │ Caches data           │
                        └─────────────────────┘       └─────────────┬─────────┘
                                                                  │
                                                                  ▼
                                                      ┌───────────────────────┐
                                                      │ Web UI for Job History │
                                                      │ Shows stages, tasks,   │
                                                      │ timelines, metrics     │
                                                      └───────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does the History Server show live running jobs or only completed ones? Commit to yes or no.

Common Belief: The History Server shows the status of live running Spark jobs in real time.

Reality: The History Server only shows completed jobs by reading saved event logs; live jobs are monitored via the Spark UI on the driver node.

Why it matters: Confusing live monitoring with History Server leads to missing real-time issues and delays in troubleshooting.

Quick: Do you think the History Server stores logs itself or just reads existing files? Commit to your answer.

Common Belief: The History Server stores and manages event logs internally in a database.

Reality: The History Server only reads event log files stored externally; it does not store logs itself.

Why it matters: Assuming the History Server manages logs can cause misconfiguration and data loss if external storage is not properly maintained.

Quick: Do you think enabling event logging slows down Spark jobs significantly? Commit to yes or no.

Common Belief: Turning on event logging causes large slowdowns in Spark job execution.

Reality: Event logging is designed to be asynchronous and lightweight, causing minimal overhead in most cases.

Why it matters: Avoiding event logging due to fear of slowdown prevents access to valuable job history and diagnostics.

Quick: Does the History Server automatically clean up old logs? Commit to yes or no.

Common Belief: The History Server automatically deletes old event logs to save space.

Reality: The History Server does not manage log retention; users must manually or externally clean old logs.

Why it matters: Ignoring log cleanup can cause storage to fill up and degrade History Server performance.

Expert Zone

1

The History Server caches parsed event logs in memory, so frequent UI refreshes do not re-parse logs, improving responsiveness.

2

Event logs can be compressed to save storage, but the History Server must be configured to decompress them on read.

3

In multi-tenant clusters, securing event log storage is critical to prevent unauthorized access to job details.

When NOT to use

The History Server is not suitable for real-time monitoring or alerting during job execution. For live metrics, use Spark's live UI or integrate with monitoring tools like Prometheus. Also, if event logs are not enabled or accessible, the History Server cannot function.

Production Patterns

In production, teams often run a dedicated History Server connected to a shared HDFS or S3 bucket where all Spark event logs are stored. Logs are rotated and archived regularly. The History Server is integrated with authentication systems for secure access. Some setups use multiple History Servers partitioned by date or project to scale for large clusters.

Connections

Distributed System Logging

The Spark History Server builds on the concept of collecting logs from distributed components to reconstruct system behavior.

Understanding distributed logging helps grasp why event logs are essential for analyzing complex Spark jobs running on many nodes.

Web Application Monitoring

Both Spark History Server and web app monitoring tools visualize performance metrics and errors after execution.

Knowing how web monitoring dashboards work clarifies how the History Server presents Spark job data for troubleshooting.

Replay Debugging in Software Engineering

The History Server acts like a replay debugger by reconstructing past job executions from logs.

Recognizing this connection shows how replay debugging principles apply beyond code to big data jobs.

Common Pitfalls

#1 Not enabling event logging before running Spark jobs.

Wrong approach:

spark-submit --class MyApp myapp.jar
# No eventLog.enabled set

Correct approach:

spark-submit --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs:///spark-logs --class MyApp myapp.jar

Root cause:

Forgetting to enable event logging means no logs are saved, so the History Server has no data to show.

#2 Pointing the History Server to a wrong or inaccessible event log directory.

Wrong approach:

spark.history.fs.logDirectory=file:///wrong/path/to/logs

Correct approach:

spark.history.fs.logDirectory=hdfs:///spark-logs

Root cause:

Misconfiguring the log directory causes the History Server to show no jobs or errors.

#3 Expecting the History Server to show live job progress.

Wrong approach:

Opening History Server UI during job run expecting real-time updates.

Correct approach:

Use Spark UI on the driver node for live monitoring; use History Server only after job completion.

Root cause:

Misunderstanding the History Server's purpose leads to confusion and missed real-time issues.

Key Takeaways

The Spark History Server lets you review detailed information about completed Spark jobs by reading event logs saved during execution.

Event logging must be enabled and properly configured for the History Server to work, including setting the log directory.

The History Server is a separate service that reads logs from shared storage and provides a web UI for post-job analysis, not live monitoring.

Managing event log storage and retention is important to keep the History Server responsive and avoid storage issues.

Understanding the History Server's design and limitations helps you integrate it effectively into Spark cluster monitoring and troubleshooting workflows.

Practice

1. What is the main purpose of the Spark History Server in monitoring Spark applications?

easy

A. To monitor real-time streaming data only

B. To view details of completed Spark jobs using saved event logs

C. To configure Spark cluster resources dynamically

D. To submit new Spark jobs to the cluster

Solution

1. Step 1: Understand Spark History Server role

   The Spark History Server reads event logs from completed jobs to provide detailed information about them.

2. Step 2: Differentiate from other Spark components

   Unlike job submission or real-time monitoring, it focuses on past job performance using saved logs.

3. Final Answer:

   To view details of completed Spark jobs using saved event logs -> Option B

4. Quick Check:

   Spark History Server = View completed job details [OK]

Hint: History Server shows past jobs, not live or submission tasks [OK]

Common Mistakes:

• Confusing History Server with job submission tools
• Thinking it monitors only live streaming jobs
• Assuming it manages cluster resources

2. Which Spark configuration property must be set to enable event logging for the History Server?

easy

A. spark.sql.shuffle.partitions

B. spark.history.fs.logDirectory

C. spark.executor.memory

D. spark.eventLog.enabled

Solution

1. Step 1: Identify event logging property

   The property spark.eventLog.enabled controls whether Spark saves event logs for completed jobs.

2. Step 2: Differentiate from other configs

   spark.history.fs.logDirectory sets log location, not enabling logging; others relate to memory or shuffle partitions.

3. Final Answer:

   spark.eventLog.enabled -> Option D

4. Quick Check:

   Enable event logging = spark.eventLog.enabled [OK]

Hint: Enable event logging with spark.eventLog.enabled true [OK]

Common Mistakes:

• Confusing log directory with enabling logging
• Choosing unrelated Spark properties
• Assuming memory or shuffle configs enable logging

3. Given the configuration:

spark.eventLog.enabled=true
spark.history.fs.logDirectory=/tmp/spark-events

What will happen after a Spark job finishes?

medium

A. Event logs will be saved to /tmp/spark-events for History Server access

B. No event logs will be saved because logging is disabled

C. Logs will be saved but History Server cannot read them

D. Spark job will fail due to invalid log directory

Solution

1. Step 1: Check event logging enabled

   Since spark.eventLog.enabled=true, Spark saves event logs after job completion.

2. Step 2: Confirm log directory setting

   The logs are saved to /tmp/spark-events, which the History Server reads to show job details.

3. Final Answer:

   Event logs will be saved to /tmp/spark-events for History Server access -> Option A

4. Quick Check:

   Logging enabled + valid directory = logs saved [OK]

Hint: True eventLog.enabled + valid directory = logs saved [OK]

Common Mistakes:

• Assuming logs are not saved if directory is custom
• Thinking logging is off despite true setting
• Believing job fails due to log directory path

4. You started the Spark History Server but see no jobs listed in the UI. Which is the most likely cause?

medium

A. The Spark History Server UI port is blocked by firewall

B. Spark jobs are still running and not completed

C. spark.eventLog.enabled is set to false or event logs are missing

D. The Spark History Server service is running on the wrong machine

Solution

1. Step 1: Check event logging status

   If spark.eventLog.enabled is false or event logs are missing, History Server has no data to show.

2. Step 2: Understand UI port and job status

   Firewall blocking UI port prevents access but not empty job list; running jobs don't appear until finished.

3. Final Answer:

   spark.eventLog.enabled is set to false or event logs are missing -> Option C

4. Quick Check:

   No jobs in UI = logging off or missing logs [OK]

Hint: No jobs shown? Check if event logging is enabled [OK]

Common Mistakes:

• Assuming firewall blocks job list display
• Expecting running jobs to appear in History Server
• Thinking server machine location affects job visibility

5. You want to monitor completed Spark jobs on a cluster. Which sequence of steps correctly sets up the Spark History Server?

hard

A. Enable event logging, set log directory, start History Server, open UI

B. Start History Server, submit jobs, enable event logging, open UI

C. Set log directory, open UI, enable event logging, start History Server

D. Submit jobs, start History Server, set log directory, enable event logging

Solution

1. Step 1: Enable event logging and set log directory

   First, enable spark.eventLog.enabled=true and set spark.history.fs.logDirectory so logs are saved properly.

2. Step 2: Start History Server and open UI

   Then start the History Server to read logs and open its web UI to monitor completed jobs.

3. Final Answer:

   Enable event logging, set log directory, start History Server, open UI -> Option A

4. Quick Check:

   Enable logging + set dir + start server + open UI = monitor jobs [OK]

Hint: Enable logging first, then start History Server and open UI [OK]

Common Mistakes:

• Starting History Server before enabling logging
• Opening UI before logs exist
• Setting log directory after starting server