Dead letter queue pattern in Kafka - Commands & Configuration

Introduction

Sometimes messages sent to a Kafka topic cannot be processed due to errors. The dead letter queue pattern helps by moving these bad messages to a separate topic so they don't block the main processing flow.

When a message has invalid data and cannot be processed by the consumer.

When a consumer application crashes or fails repeatedly on certain messages.

When you want to isolate problematic messages for later inspection without stopping the whole system.

When you want to keep your main topic clean and only process valid messages.

When you want to track and analyze errors in message processing separately.

Config File - dead_letter_queue_config.properties

dead_letter_queue_config.properties

bootstrap.servers=localhost:9092
acks=all
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer

# Dead Letter Queue topic name
dead.letter.queue.topic=example-dead-letter-queue

# Main topic name
main.topic=example-main-topic

# Consumer group
group.id=example-consumer-group

# Enable auto commit
enable.auto.commit=false

# Max retries before sending to DLQ
max.retries=3

This configuration file sets up Kafka producer and consumer properties.

bootstrap.servers: Kafka server address.

dead.letter.queue.topic: Topic where bad messages go.

main.topic: Topic with normal messages.

max.retries: Number of times to retry processing before sending to DLQ.

Commands

Create the main Kafka topic where normal messages will be sent and consumed.

Terminal

kafka-topics --create --topic example-main-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Expected Output

Created topic example-main-topic.

→

--partitions - Number of partitions for the topic

→

--replication-factor - Number of copies of the topic data

Create the dead letter queue topic where failed messages will be sent.

Terminal

kafka-topics --create --topic example-dead-letter-queue --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Expected Output

Created topic example-dead-letter-queue.

→

--partitions - Number of partitions for the DLQ topic

→

--replication-factor - Number of copies of the DLQ topic data

Start a producer to send messages to the main topic. You can type messages here.

Terminal

kafka-console-producer --topic example-main-topic --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Topic to send messages to

→

--bootstrap-server - Kafka server address

Start a consumer to read messages from the main topic from the beginning. This consumer should be configured to send bad messages to the DLQ after retries.

Terminal

kafka-console-consumer --topic example-main-topic --bootstrap-server localhost:9092 --group example-consumer-group --from-beginning

Expected Output

This command will output messages from the topic as they arrive, for example:

Hello World

BadMessage

Another Good Message

→

--group - Consumer group id

→

--from-beginning - Read messages from the start of the topic

Consume messages from the dead letter queue topic to inspect failed messages.

Terminal

kafka-console-consumer --topic example-dead-letter-queue --bootstrap-server localhost:9092 --from-beginning

Expected Output

BadMessage

MalformedData

TimeoutError

→

--from-beginning - Read all messages from the DLQ topic

Key Concept

If you remember nothing else from this pattern, remember: move messages that fail processing repeatedly to a separate dead letter queue topic to keep your main flow running smoothly.

Common Mistakes

Not creating the dead letter queue topic before sending messages to it.

Kafka will reject messages sent to a non-existent topic, causing errors.

Always create the DLQ topic with kafka-topics before producing messages to it.

Not handling retries in the consumer before sending messages to the DLQ.

Messages may be sent to the DLQ too early or not at all, causing loss or blocking.

Implement retry logic in the consumer to attempt processing multiple times before sending to DLQ.

Consuming the DLQ topic without monitoring or alerting.

Failed messages may pile up unnoticed, causing hidden problems.

Regularly monitor the DLQ topic and set alerts to investigate and fix issues.

Summary

Create both main and dead letter queue Kafka topics before use.

Send normal messages to the main topic and configure consumers to retry processing.

After retries fail, send bad messages to the dead letter queue topic for later inspection.

Consume the dead letter queue topic separately to monitor and fix message issues.