Error handling in streams in Kafka - Step-by-Step Execution

Process Flow - Error handling in streams

Start Stream Processing

↓

Read Message from Topic

↓

Process Message

↓

Error Occurs?

No→Send Output to Next Topic

Yes↓

Handle Error

↓

Log Error

↓

Continue Processing Next Message

This flow shows how a stream reads messages, processes them, checks for errors, handles errors by logging and sending to a special topic, then continues.

Execution Sample

Kafka

try {
  process(record);
  forward(record);
} catch (Exception e) {
  log(e);
  sendToDeadLetter(record);
}

This code tries to process a record, forwards it if successful, or logs and sends it to a dead letter topic if an error occurs.

Process Table

Step	Action	Message Content	Error Occurred?	Error Handling	Output
1	Read message	valid-data-1	No	None	Process and forward
2	Process message	valid-data-1	No	None	Forwarded to next topic
3	Read message	bad-data-2	Yes	Catch exception	Send to dead letter topic
4	Log error	bad-data-2	Yes	Logged error details	No forward
5	Read message	valid-data-3	No	None	Process and forward
6	Process message	valid-data-3	No	None	Forwarded to next topic
7	End of stream	-	-	-	Stop processing

💡 Reached end of stream, no more messages to process

Status Tracker

Variable	Start	After 1	After 2	After 3	After 4	After 5	After 6	Final
message	null	valid-data-1	valid-data-1	bad-data-2	bad-data-2	valid-data-3	valid-data-3	null
errorOccurred	false	false	false	true	true	false	false	false
outputTopic	empty	empty	valid-data-1	valid-data-1	valid-data-1	valid-data-1,valid-data-3	valid-data-1,valid-data-3	valid-data-1,valid-data-3
deadLetterTopic	empty	empty	empty	bad-data-2	bad-data-2	bad-data-2	bad-data-2	bad-data-2

Key Moments - 3 Insights

Why does the bad message not get forwarded to the output topic?

Because in the execution_table at step 3 and 4, the errorOccurred is true, triggering error handling that sends the message to the dead letter topic instead of forwarding.

What happens after an error is handled?

As shown in the concept_flow and execution_table, after logging and sending to dead letter, the stream continues processing the next message without stopping.

Is the stream stopped when an error occurs?

No, the stream continues as seen in the execution_table steps 5 and 6 where valid messages are processed after the error.

Visual Quiz - 3 Questions

Look at the execution_table at step 3, what is the value of 'Error Occurred?'

A No

B Unknown

C Yes

D False

💡 Hint

Check the 'Error Occurred?' column in execution_table row 3

At which step does the message get sent to the dead letter topic?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Look at the 'Error Handling' and 'Output' columns in execution_table rows 3 and 4

If the error handling was removed, what would happen to bad-data-2?

A It would cause the stream to stop

B It would be logged but not forwarded

C It would be forwarded to the next topic

D It would be ignored silently

💡 Hint

Refer to the concept_flow and key_moments about error handling importance

Concept Snapshot

Error handling in streams:
- Try processing each message
- If error occurs, catch exception
- Log error and send message to dead letter topic
- Continue processing next messages
- Prevents stream from stopping on errors

Full Transcript

This visual trace shows how Kafka stream processing handles errors. The stream reads messages one by one. For each message, it tries to process it. If no error occurs, the message is forwarded to the next topic. If an error happens, the stream catches the exception, logs the error, and sends the problematic message to a dead letter topic. Then the stream continues processing the next messages without stopping. This approach keeps the stream running smoothly even when some messages cause errors.

Practice

1. What is the main purpose of error handling in Kafka streams?

easy

A. To delete all bad data permanently

B. To keep the stream running smoothly even if some data is bad

C. To stop the stream immediately when an error occurs

D. To ignore errors and continue processing without any checks

Solution

1. Step 1: Understand error handling goal

   Error handling aims to manage problems without stopping the whole process.

2. Step 2: Identify correct purpose in Kafka streams

   Kafka streams use error handling to keep processing even if some data is bad.

3. Final Answer:

   To keep the stream running smoothly even if some data is bad -> Option B

4. Quick Check:

   Error handling = keep stream running [OK]

Hint: Error handling means keep running despite bad data [OK]

Common Mistakes:

• Thinking error handling stops the stream
• Assuming bad data is deleted automatically
• Ignoring the need to handle errors

2. Which syntax correctly shows how to catch exceptions in a Kafka Streams processor in Java?

easy

A. catch (Exception e) { processRecord(); } try { handleError(e); }

B. processRecord() catch Exception e handleError(e)

C. try { processRecord(); } catch (Exception e) { handleError(e); }

D. try processRecord() catch Exception handleError()

Solution

1. Step 1: Recall Java try-catch syntax

   Java uses try { } catch (Exception e) { } to handle exceptions.

2. Step 2: Match correct syntax for Kafka Streams processor

   try { processRecord(); } catch (Exception e) { handleError(e); } correctly wraps processRecord() in try and catches Exception e to handle errors.

3. Final Answer:

   try { processRecord(); } catch (Exception e) { handleError(e); } -> Option C

4. Quick Check:

   Java try-catch syntax = try { processRecord(); } catch (Exception e) { handleError(e); } [OK]

Hint: Remember Java try-catch blocks use braces and catch after try [OK]

Common Mistakes:

• Placing catch before try
• Missing braces {} around try or catch blocks
• Using incorrect keywords or order

3. Given this Kafka Streams code snippet, what happens when a bad record causes an exception?

streams.foreach((key, value) -> {
  try {
    process(value);
  } catch (Exception e) {
    deadLetterTopic.send(key, value);
  }
});

medium

A. The exception crashes the whole application

B. The stream stops processing immediately

C. The bad record is ignored and lost

D. The bad record is sent to a dead-letter topic for later review

Solution

1. Step 1: Analyze try-catch block in code

   The code tries to process each record; if an exception occurs, it catches it.

2. Step 2: Check catch block action

   In catch, the bad record is sent to deadLetterTopic for later review.

3. Final Answer:

   The bad record is sent to a dead-letter topic for later review -> Option D

4. Quick Check:

   Exception caught = send to dead-letter topic [OK]

Hint: Catch exceptions and send bad data to dead-letter topic [OK]

Common Mistakes:

• Assuming stream stops on exception
• Thinking bad data is lost without handling
• Believing exception crashes app without catch

4. Identify the error in this Kafka Streams error handling code:

streams.foreach((key, value) -> {
  try {
    process(value);
  } catch Exception e {
    log.error(e.getMessage());
  }
});

medium

A. Missing parentheses around Exception e in catch block

B. Using foreach instead of map for error handling

C. Logging error message is incorrect

D. Try block should be outside the foreach

Solution

1. Step 1: Check catch block syntax

   In Java, catch must have parentheses around the exception type and variable.

2. Step 2: Identify missing parentheses

   The code has 'catch Exception e' missing parentheses; correct is 'catch (Exception e)'.

3. Final Answer:

   Missing parentheses around Exception e in catch block -> Option A

4. Quick Check:

   Catch syntax requires parentheses [OK]

Hint: Catch blocks always need parentheses around exception [OK]

Common Mistakes:

• Omitting parentheses in catch
• Confusing foreach and map usage
• Thinking logging method is wrong

5. You want to build a Kafka Streams application that processes user events but sends any malformed events to a separate topic without stopping the main stream. Which approach best achieves this?

hard

A. Use a try-catch inside the stream processor to send bad events to a dead-letter topic

B. Stop the stream when a malformed event is detected to fix data immediately

C. Ignore malformed events and continue processing without logging

D. Filter out malformed events before processing without saving them

Solution

1. Step 1: Understand requirement to keep stream running

   The stream must continue processing even if some events are bad.

2. Step 2: Choose method to handle bad events

   Using try-catch to catch errors and send bad events to a dead-letter topic allows this.

3. Final Answer:

   Use a try-catch inside the stream processor to send bad events to a dead-letter topic -> Option A

4. Quick Check:

   Try-catch + dead-letter topic = best error handling [OK]

Hint: Try-catch plus dead-letter topic keeps stream healthy [OK]

Common Mistakes:

• Stopping stream on error instead of handling
• Ignoring bad data without saving for review
• Filtering bad data without storing it