Monitoring with Spark History Server in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Spark History Server Log Directory Listing

You have configured Spark History Server with the log directory set to /var/log/spark-events. What is the output of the command ls /var/log/spark-events if the directory contains two completed application event logs named app-20240601-123456-0001 and app-20240601-123457-0002?

Apache Spark

ls /var/log/spark-events

A app-20240601-123456-0001, app-20240601-123457-0002

B app-20240601-123456-0001 app-20240601-123457-0002

C No such file or directory

D

app-20240601-123456-0001
app-20240601-123457-0002

💡 Hint

Think about how the ls command outputs multiple files in a directory by default.

The ls command lists files line by line by default when output is to a terminal. So each file name appears on its own line.

🧠 Conceptual

intermediate

Purpose of Spark History Server

What is the main purpose of the Spark History Server in a Spark cluster?

A To provide a web UI to view completed Spark application event logs and their metrics

B To schedule new Spark jobs on the cluster

C To monitor real-time resource usage of Spark executors

D To manage user authentication and authorization for Spark

💡 Hint

Think about what happens after a Spark job finishes and you want to review its details.

The Spark History Server reads event logs from completed applications and provides a web UI to analyze job stages, tasks, and metrics after the job finishes.

❓ Configuration

advanced

Configuring Spark History Server Log Directory

Which configuration property must be set in spark-defaults.conf to specify the directory where Spark History Server reads event logs?

A spark.history.fs.logDirectory

B spark.eventLog.dir

C spark.history.log.dir

D spark.eventLog.enabled

💡 Hint

Look for the property that starts with spark.history and relates to the log directory.

The property spark.history.fs.logDirectory defines the directory path where the Spark History Server looks for event logs.

❓ Troubleshoot

advanced

Spark History Server Not Showing Completed Applications

You started the Spark History Server and pointed it to the correct event log directory, but no completed applications appear in the UI. What is the most likely cause?

A The Spark cluster is not running any active jobs

B The Spark History Server service is running on the wrong port

C Event logging was not enabled during Spark application runs

D The Spark History Server UI requires a browser restart

💡 Hint

Think about what must happen during Spark job execution for the History Server to have logs to read.

If event logging is not enabled in Spark applications, no event logs are created, so the History Server has nothing to display.

✅ Best Practice

expert

Best Practice for Spark Event Log Storage

For a production Spark cluster with many completed jobs, what is the best practice for storing Spark event logs to ensure efficient monitoring with Spark History Server?

A Store event logs in a relational database for faster querying

B Store event logs on a shared distributed filesystem like HDFS or S3 accessible by all Spark nodes and the History Server

C Disable event logging to reduce storage usage and rely on real-time monitoring only

D Store event logs locally on each Spark driver node's disk

💡 Hint

Consider accessibility and scalability for multiple Spark nodes and the History Server.

Using a shared distributed filesystem like HDFS or S3 ensures all nodes and the History Server can access event logs reliably and scalably.

Practice

1. What is the main purpose of the Spark History Server in monitoring Spark applications?

easy

A. To monitor real-time streaming data only

B. To view details of completed Spark jobs using saved event logs

C. To configure Spark cluster resources dynamically

D. To submit new Spark jobs to the cluster

Solution

1. Step 1: Understand Spark History Server role

   The Spark History Server reads event logs from completed jobs to provide detailed information about them.

2. Step 2: Differentiate from other Spark components

   Unlike job submission or real-time monitoring, it focuses on past job performance using saved logs.

3. Final Answer:

   To view details of completed Spark jobs using saved event logs -> Option B

4. Quick Check:

   Spark History Server = View completed job details [OK]

Hint: History Server shows past jobs, not live or submission tasks [OK]

Common Mistakes:

• Confusing History Server with job submission tools
• Thinking it monitors only live streaming jobs
• Assuming it manages cluster resources

2. Which Spark configuration property must be set to enable event logging for the History Server?

easy

A. spark.sql.shuffle.partitions

B. spark.history.fs.logDirectory

C. spark.executor.memory

D. spark.eventLog.enabled

Solution

1. Step 1: Identify event logging property

   The property spark.eventLog.enabled controls whether Spark saves event logs for completed jobs.

2. Step 2: Differentiate from other configs

   spark.history.fs.logDirectory sets log location, not enabling logging; others relate to memory or shuffle partitions.

3. Final Answer:

   spark.eventLog.enabled -> Option D

4. Quick Check:

   Enable event logging = spark.eventLog.enabled [OK]

Hint: Enable event logging with spark.eventLog.enabled true [OK]

Common Mistakes:

• Confusing log directory with enabling logging
• Choosing unrelated Spark properties
• Assuming memory or shuffle configs enable logging

3. Given the configuration:

spark.eventLog.enabled=true
spark.history.fs.logDirectory=/tmp/spark-events

What will happen after a Spark job finishes?

medium

A. Event logs will be saved to /tmp/spark-events for History Server access

B. No event logs will be saved because logging is disabled

C. Logs will be saved but History Server cannot read them

D. Spark job will fail due to invalid log directory

Solution

1. Step 1: Check event logging enabled

   Since spark.eventLog.enabled=true, Spark saves event logs after job completion.

2. Step 2: Confirm log directory setting

   The logs are saved to /tmp/spark-events, which the History Server reads to show job details.

3. Final Answer:

   Event logs will be saved to /tmp/spark-events for History Server access -> Option A

4. Quick Check:

   Logging enabled + valid directory = logs saved [OK]

Hint: True eventLog.enabled + valid directory = logs saved [OK]

Common Mistakes:

• Assuming logs are not saved if directory is custom
• Thinking logging is off despite true setting
• Believing job fails due to log directory path

4. You started the Spark History Server but see no jobs listed in the UI. Which is the most likely cause?

medium

A. The Spark History Server UI port is blocked by firewall

B. Spark jobs are still running and not completed

C. spark.eventLog.enabled is set to false or event logs are missing

D. The Spark History Server service is running on the wrong machine

Solution

1. Step 1: Check event logging status

   If spark.eventLog.enabled is false or event logs are missing, History Server has no data to show.

2. Step 2: Understand UI port and job status

   Firewall blocking UI port prevents access but not empty job list; running jobs don't appear until finished.

3. Final Answer:

   spark.eventLog.enabled is set to false or event logs are missing -> Option C

4. Quick Check:

   No jobs in UI = logging off or missing logs [OK]

Hint: No jobs shown? Check if event logging is enabled [OK]

Common Mistakes:

• Assuming firewall blocks job list display
• Expecting running jobs to appear in History Server
• Thinking server machine location affects job visibility

5. You want to monitor completed Spark jobs on a cluster. Which sequence of steps correctly sets up the Spark History Server?

hard

A. Enable event logging, set log directory, start History Server, open UI

B. Start History Server, submit jobs, enable event logging, open UI

C. Set log directory, open UI, enable event logging, start History Server

D. Submit jobs, start History Server, set log directory, enable event logging

Solution

1. Step 1: Enable event logging and set log directory

   First, enable spark.eventLog.enabled=true and set spark.history.fs.logDirectory so logs are saved properly.

2. Step 2: Start History Server and open UI

   Then start the History Server to read logs and open its web UI to monitor completed jobs.

3. Final Answer:

   Enable event logging, set log directory, start History Server, open UI -> Option A

4. Quick Check:

   Enable logging + set dir + start server + open UI = monitor jobs [OK]

Hint: Enable logging first, then start History Server and open UI [OK]

Common Mistakes:

• Starting History Server before enabling logging
• Opening UI before logs exist
• Setting log directory after starting server