Error handling and dead letter queues in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of error handling in Apache Spark?

Error handling in Apache Spark helps to catch and manage errors during data processing to prevent the entire job from failing and to allow for graceful recovery or logging.

beginner

Define a dead letter queue (DLQ) in the context of data processing.

A dead letter queue is a storage location where messages or data records that cannot be processed successfully are sent for later inspection and handling.

intermediate

How can you implement error handling in Spark Structured Streaming?

You can implement error handling by using try-catch blocks in your processing logic and by writing failed records to a dead letter queue or error log for later analysis.

intermediate

Why is it important to use dead letter queues in streaming data pipelines?

Dead letter queues help isolate bad data without stopping the entire pipeline, allowing the system to continue processing good data while problematic records are reviewed and fixed later.

advanced

Show a simple example of writing failed records to a dead letter queue in Spark.

In Spark, you can filter out bad records and write them to a separate storage like this: val badRecords = inputData.filter(record => /* condition for failure */) badRecords.write.format("parquet").save("/path/to/dead_letter_queue")

What happens to records sent to a dead letter queue?

A They are stored for later inspection and correction.

B They are processed immediately again.

C They are discarded permanently.

D They are merged back into the main data stream.

Records in a dead letter queue are stored separately so they can be inspected and corrected later without affecting the main data flow.

Which Spark feature helps catch errors during streaming processing?

A Automatic data correction

B Try-catch blocks in processing logic

C Spark UI monitoring only

D Spark SQL optimizer

Try-catch blocks allow you to catch exceptions during processing and handle errors gracefully.

Why should you not stop the entire Spark job when a few records fail?

A To allow processing of good data to continue.

B Because errors are ignored automatically.

C Because failed records fix themselves.

D Because Spark cannot stop jobs.

Stopping the entire job wastes resources and delays processing of valid data; error handling and dead letter queues help avoid this.

What is a common format to store dead letter queue data in Spark?

A CSV only

B HTML

C Parquet

D Plain text logs only

Parquet is a common efficient columnar format used in Spark for storing data including dead letter queue records.

Which of these is NOT a benefit of using dead letter queues?

A Isolating bad data

B Enabling later data correction

C Allowing continuous processing

D Automatically fixing errors

Dead letter queues store bad data but do not automatically fix errors; manual review is needed.

Explain how error handling and dead letter queues work together in a Spark streaming pipeline.

Think about how you keep the pipeline running while handling bad data.

Describe a simple approach to implement a dead letter queue in Apache Spark.

Focus on how to separate and save bad data.

Practice

1. What is the main purpose of using a dead letter queue in Apache Spark data pipelines?

easy

A. To speed up the processing of good data by caching it

B. To store bad or invalid data separately for later analysis or correction

C. To automatically fix errors in the data during processing

D. To merge multiple data sources into one unified dataset

Solution

1. Step 1: Understand the role of error handling in data pipelines

   Error handling separates good data from bad data to prevent pipeline failures.

2. Step 2: Define the purpose of a dead letter queue

   A dead letter queue stores bad or invalid data separately so it can be reviewed or fixed later without stopping the main pipeline.

3. Final Answer:

   To store bad or invalid data separately for later analysis or correction -> Option B

4. Quick Check:

   Dead letter queue = store bad data separately [OK]

Hint: Dead letter queue holds bad data separately [OK]

Common Mistakes:

• Thinking dead letter queue fixes data automatically
• Confusing dead letter queue with caching good data
• Assuming it merges data sources

2. Which of the following Spark code snippets correctly filters out invalid records and writes them to a dead letter queue DataFrame named dlqDF?

easy

A. val dlqDF = inputDF.filter(col("age") < 0)

B. val dlqDF = inputDF.select("age")

C. val dlqDF = inputDF.filter(col("age") > 0)

D. val dlqDF = inputDF.drop("age")

Solution

1. Step 1: Identify the condition for invalid data

   Invalid records have age less than 0, so filter with col("age") < 0.

2. Step 2: Check which code filters invalid data correctly

   val dlqDF = inputDF.filter(col("age") < 0) filters rows where age is less than 0, capturing invalid data for the dead letter queue.

3. Final Answer:

   val dlqDF = inputDF.filter(col("age") < 0) -> Option A

4. Quick Check:

   Filter invalid age < 0 = val dlqDF = inputDF.filter(col("age") < 0) [OK]

Hint: Filter invalid rows with condition in filter() [OK]

Common Mistakes:

• Filtering valid data instead of invalid
• Using select instead of filter
• Dropping columns instead of filtering rows

3. Given the following Spark code, what will be the count of records in dlqDF?

val inputDF = spark.createDataFrame(Seq((1, 25), (2, -5), (3, 30), (4, -1))).toDF("id", "age")
val dlqDF = inputDF.filter(col("age") < 0)
dlqDF.count()

medium

A. 2

B. 4

C. 0

D. 3

Solution

1. Step 1: Identify records with age less than 0

   Records with age -5 and -1 meet the condition, so 2 records are invalid.

2. Step 2: Count filtered records in dlqDF

   dlqDF contains these 2 invalid records, so count is 2.

3. Final Answer:

   2 -> Option A

4. Quick Check:

   Count invalid age < 0 = 2 [OK]

Hint: Count rows where age < 0 to find dlq size [OK]

Common Mistakes:

• Counting all rows instead of filtered
• Miscounting invalid records
• Confusing filter condition direction

4. The following Spark code is intended to separate invalid records (age < 0) into a dead letter queue DataFrame dlqDF. However, it does not work as expected. What is the error?

val dlqDF = inputDF.filter("age > 0")

medium

A. The DataFrame name dlqDF is reserved and cannot be used

B. The filter syntax is incorrect; it should use col() function

C. The filter method cannot be used with string conditions

D. The filter condition is reversed; it selects valid data instead of invalid

Solution

1. Step 1: Analyze the filter condition

   The condition "age > 0" selects records with positive age, which are valid, not invalid.

2. Step 2: Identify the correct condition for invalid data

   Invalid data has age less than 0, so the filter should be "age < 0" to capture bad records.

3. Final Answer:

   The filter condition is reversed; it selects valid data instead of invalid -> Option D

4. Quick Check:

   Filter condition reversed = The filter condition is reversed; it selects valid data instead of invalid [OK]

Hint: Check filter logic matches invalid data condition [OK]

Common Mistakes:

• Using wrong comparison operator in filter
• Thinking string filter syntax is invalid
• Assuming DataFrame names are restricted

5. You have a Spark DataFrame inputDF with a column status that can be 'valid' or 'invalid'. You want to create two DataFrames: cleanDF with only valid records and dlqDF with invalid records. Which code correctly implements this error handling with dead letter queue?

hard

A. val cleanDF = inputDF.filter(col("status") = "valid")
val dlqDF = inputDF.filter(col("status") = "invalid")

B. val cleanDF = inputDF.select("status" == "valid")
val dlqDF = inputDF.select("status" == "invalid")

C. val cleanDF = inputDF.filter(col("status") === "valid")
val dlqDF = inputDF.filter(col("status") === "invalid")

D. val cleanDF = inputDF.drop("invalid")
val dlqDF = inputDF.drop("valid")

Solution

1. Step 1: Use correct equality operator in filter

   In Spark Scala, use triple equals (===) to compare column values to strings.

2. Step 2: Filter valid and invalid records separately

   Filter status === "valid" for cleanDF and status === "invalid" for dlqDF to separate data correctly.

3. Final Answer:

   val cleanDF = inputDF.filter(col("status") === "valid") val dlqDF = inputDF.filter(col("status") === "invalid") -> Option C

4. Quick Check:

   Use === for equality and filter separately [OK]

Hint: Use === for string equality in filter() [OK]

Common Mistakes:

• Using single = instead of === in filter
• Using select instead of filter for conditions
• Dropping columns instead of filtering rows