Error handling in streams in Kafka - Commands & Configuration

Introduction

When processing data streams, errors can happen like bad data or connection issues. Handling these errors properly helps keep the stream running smoothly without losing important data or crashing.

When your stream processing application receives malformed or unexpected data.

When you want to retry processing a message that failed due to temporary issues.

When you want to send problematic messages to a separate topic for later review.

When you want to log errors for monitoring and debugging purposes.

When you want to avoid stopping the entire stream because of one bad message.

Config File - error-handling-streams.properties

error-handling-streams.properties

bootstrap.servers=localhost:9092
application.id=error-handling-app
processing.guarantee=exactly_once

# Configure dead letter queue topic
dead.letter.queue.topic=error-topic

# Configure retry settings
retry.backoff.ms=1000
max.poll.records=10

# Enable logging of errors
log4j.logger.org.apache.kafka.streams.processor.internals.StreamThread=INFO

bootstrap.servers: Connects to Kafka server.

application.id: Unique ID for this stream app.

processing.guarantee: Ensures messages are processed exactly once.

dead.letter.queue.topic: Topic where bad messages go.

retry.backoff.ms: Wait time before retrying a failed message.

max.poll.records: Number of messages processed at once.

log4j.logger.org.apache.kafka.streams.processor.internals.StreamThread: Enables logging for error tracking.

Commands

Create a dead letter queue topic named 'error-topic' to store messages that fail processing.

Terminal

kafka-topics --create --topic error-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Expected Output

Created topic error-topic.

→

--topic - Name of the Kafka topic to create

→

--partitions - Number of partitions for the topic

→

--replication-factor - Number of replicas for fault tolerance

Send test messages to the input topic to simulate stream data including some bad data.

Terminal

kafka-console-producer --topic input-topic --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Topic to send messages to

→

--bootstrap-server - Kafka server address

Read messages from the dead letter queue to see which messages failed processing.

Terminal

kafka-console-consumer --topic error-topic --bootstrap-server localhost:9092 --from-beginning

Expected Output

bad message 1

bad message 2

→

--topic - Topic to read messages from

→

--from-beginning - Read all messages from the start

Key Concept

If you remember nothing else from this pattern, remember: send bad or failed messages to a separate topic to keep your stream running smoothly and fix errors later.

Common Mistakes

Not creating a dead letter queue topic before starting the stream application.

The application will fail or lose bad messages because there is nowhere to send them.

Always create the dead letter queue topic in Kafka before running the stream app.

Ignoring error logs and not monitoring the dead letter queue.

You miss important information about data problems and cannot fix issues timely.

Regularly check error logs and consume messages from the dead letter queue topic.

Not configuring retries or backoff, causing the app to crash on first error.

Temporary issues cause the whole stream to stop instead of retrying.

Set retry and backoff settings to handle temporary failures gracefully.

Summary

Create a dead letter queue topic to hold messages that fail processing.

Send test messages including bad data to the input topic to simulate errors.

Consume messages from the dead letter queue to review and fix errors.