Error handling in streams in Kafka - Commands & Configuration

Introduction

When processing data streams, errors can happen like bad data or connection issues. Handling these errors properly helps keep the stream running smoothly without losing important data or crashing.

When your stream processing application receives malformed or unexpected data.

When you want to retry processing a message that failed due to temporary issues.

When you want to send problematic messages to a separate topic for later review.

When you want to log errors for monitoring and debugging purposes.

When you want to avoid stopping the entire stream because of one bad message.

Config File - error-handling-streams.properties

error-handling-streams.properties

bootstrap.servers=localhost:9092
application.id=error-handling-app
processing.guarantee=exactly_once

# Configure dead letter queue topic
dead.letter.queue.topic=error-topic

# Configure retry settings
retry.backoff.ms=1000
max.poll.records=10

# Enable logging of errors
log4j.logger.org.apache.kafka.streams.processor.internals.StreamThread=INFO

bootstrap.servers: Connects to Kafka server.

application.id: Unique ID for this stream app.

processing.guarantee: Ensures messages are processed exactly once.

dead.letter.queue.topic: Topic where bad messages go.

retry.backoff.ms: Wait time before retrying a failed message.

max.poll.records: Number of messages processed at once.

log4j.logger.org.apache.kafka.streams.processor.internals.StreamThread: Enables logging for error tracking.

Commands

Create a dead letter queue topic named 'error-topic' to store messages that fail processing.

Terminal

kafka-topics --create --topic error-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Expected Output

Created topic error-topic.

→

--topic - Name of the Kafka topic to create

→

--partitions - Number of partitions for the topic

→

--replication-factor - Number of replicas for fault tolerance

Send test messages to the input topic to simulate stream data including some bad data.

Terminal

kafka-console-producer --topic input-topic --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Topic to send messages to

→

--bootstrap-server - Kafka server address

Read messages from the dead letter queue to see which messages failed processing.

Terminal

kafka-console-consumer --topic error-topic --bootstrap-server localhost:9092 --from-beginning

Expected Output

bad message 1

bad message 2

→

--topic - Topic to read messages from

→

--from-beginning - Read all messages from the start

Key Concept

If you remember nothing else from this pattern, remember: send bad or failed messages to a separate topic to keep your stream running smoothly and fix errors later.

Common Mistakes

Not creating a dead letter queue topic before starting the stream application.

The application will fail or lose bad messages because there is nowhere to send them.

Always create the dead letter queue topic in Kafka before running the stream app.

Ignoring error logs and not monitoring the dead letter queue.

You miss important information about data problems and cannot fix issues timely.

Regularly check error logs and consume messages from the dead letter queue topic.

Not configuring retries or backoff, causing the app to crash on first error.

Temporary issues cause the whole stream to stop instead of retrying.

Set retry and backoff settings to handle temporary failures gracefully.

Summary

Create a dead letter queue topic to hold messages that fail processing.

Send test messages including bad data to the input topic to simulate errors.

Consume messages from the dead letter queue to review and fix errors.

Practice

1. What is the main purpose of error handling in Kafka streams?

easy

A. To delete all bad data permanently

B. To keep the stream running smoothly even if some data is bad

C. To stop the stream immediately when an error occurs

D. To ignore errors and continue processing without any checks

Solution

1. Step 1: Understand error handling goal

   Error handling aims to manage problems without stopping the whole process.

2. Step 2: Identify correct purpose in Kafka streams

   Kafka streams use error handling to keep processing even if some data is bad.

3. Final Answer:

   To keep the stream running smoothly even if some data is bad -> Option B

4. Quick Check:

   Error handling = keep stream running [OK]

Hint: Error handling means keep running despite bad data [OK]

Common Mistakes:

• Thinking error handling stops the stream
• Assuming bad data is deleted automatically
• Ignoring the need to handle errors

2. Which syntax correctly shows how to catch exceptions in a Kafka Streams processor in Java?

easy

A. catch (Exception e) { processRecord(); } try { handleError(e); }

B. processRecord() catch Exception e handleError(e)

C. try { processRecord(); } catch (Exception e) { handleError(e); }

D. try processRecord() catch Exception handleError()

Solution

1. Step 1: Recall Java try-catch syntax

   Java uses try { } catch (Exception e) { } to handle exceptions.

2. Step 2: Match correct syntax for Kafka Streams processor

   try { processRecord(); } catch (Exception e) { handleError(e); } correctly wraps processRecord() in try and catches Exception e to handle errors.

3. Final Answer:

   try { processRecord(); } catch (Exception e) { handleError(e); } -> Option C

4. Quick Check:

   Java try-catch syntax = try { processRecord(); } catch (Exception e) { handleError(e); } [OK]

Hint: Remember Java try-catch blocks use braces and catch after try [OK]

Common Mistakes:

• Placing catch before try
• Missing braces {} around try or catch blocks
• Using incorrect keywords or order

3. Given this Kafka Streams code snippet, what happens when a bad record causes an exception?

streams.foreach((key, value) -> {
  try {
    process(value);
  } catch (Exception e) {
    deadLetterTopic.send(key, value);
  }
});

medium

A. The exception crashes the whole application

B. The stream stops processing immediately

C. The bad record is ignored and lost

D. The bad record is sent to a dead-letter topic for later review

Solution

1. Step 1: Analyze try-catch block in code

   The code tries to process each record; if an exception occurs, it catches it.

2. Step 2: Check catch block action

   In catch, the bad record is sent to deadLetterTopic for later review.

3. Final Answer:

   The bad record is sent to a dead-letter topic for later review -> Option D

4. Quick Check:

   Exception caught = send to dead-letter topic [OK]

Hint: Catch exceptions and send bad data to dead-letter topic [OK]

Common Mistakes:

• Assuming stream stops on exception
• Thinking bad data is lost without handling
• Believing exception crashes app without catch

4. Identify the error in this Kafka Streams error handling code:

streams.foreach((key, value) -> {
  try {
    process(value);
  } catch Exception e {
    log.error(e.getMessage());
  }
});

medium

A. Missing parentheses around Exception e in catch block

B. Using foreach instead of map for error handling

C. Logging error message is incorrect

D. Try block should be outside the foreach

Solution

1. Step 1: Check catch block syntax

   In Java, catch must have parentheses around the exception type and variable.

2. Step 2: Identify missing parentheses

   The code has 'catch Exception e' missing parentheses; correct is 'catch (Exception e)'.

3. Final Answer:

   Missing parentheses around Exception e in catch block -> Option A

4. Quick Check:

   Catch syntax requires parentheses [OK]

Hint: Catch blocks always need parentheses around exception [OK]

Common Mistakes:

• Omitting parentheses in catch
• Confusing foreach and map usage
• Thinking logging method is wrong

5. You want to build a Kafka Streams application that processes user events but sends any malformed events to a separate topic without stopping the main stream. Which approach best achieves this?

hard

A. Use a try-catch inside the stream processor to send bad events to a dead-letter topic

B. Stop the stream when a malformed event is detected to fix data immediately

C. Ignore malformed events and continue processing without logging

D. Filter out malformed events before processing without saving them

Solution

1. Step 1: Understand requirement to keep stream running

   The stream must continue processing even if some events are bad.

2. Step 2: Choose method to handle bad events

   Using try-catch to catch errors and send bad events to a dead-letter topic allows this.

3. Final Answer:

   Use a try-catch inside the stream processor to send bad events to a dead-letter topic -> Option A

4. Quick Check:

   Try-catch + dead-letter topic = best error handling [OK]

Hint: Try-catch plus dead-letter topic keeps stream healthy [OK]

Common Mistakes:

• Stopping stream on error instead of handling
• Ignoring bad data without saving for review
• Filtering bad data without storing it