Dead letter queue pattern in Kafka - Commands & Configuration

Introduction

Sometimes messages sent to a Kafka topic cannot be processed due to errors. The dead letter queue pattern helps by moving these bad messages to a separate topic so they don't block the main processing flow.

When a message has invalid data and cannot be processed by the consumer.

When a consumer application crashes or fails repeatedly on certain messages.

When you want to isolate problematic messages for later inspection without stopping the whole system.

When you want to keep your main topic clean and only process valid messages.

When you want to track and analyze errors in message processing separately.

Config File - dead_letter_queue_config.properties

dead_letter_queue_config.properties

bootstrap.servers=localhost:9092
acks=all
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer

# Dead Letter Queue topic name
dead.letter.queue.topic=example-dead-letter-queue

# Main topic name
main.topic=example-main-topic

# Consumer group
group.id=example-consumer-group

# Enable auto commit
enable.auto.commit=false

# Max retries before sending to DLQ
max.retries=3

This configuration file sets up Kafka producer and consumer properties.

bootstrap.servers: Kafka server address.

dead.letter.queue.topic: Topic where bad messages go.

main.topic: Topic with normal messages.

max.retries: Number of times to retry processing before sending to DLQ.

Commands

Create the main Kafka topic where normal messages will be sent and consumed.

Terminal

kafka-topics --create --topic example-main-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Expected Output

Created topic example-main-topic.

→

--partitions - Number of partitions for the topic

→

--replication-factor - Number of copies of the topic data

Create the dead letter queue topic where failed messages will be sent.

Terminal

kafka-topics --create --topic example-dead-letter-queue --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Expected Output

Created topic example-dead-letter-queue.

→

--partitions - Number of partitions for the DLQ topic

→

--replication-factor - Number of copies of the DLQ topic data

Start a producer to send messages to the main topic. You can type messages here.

Terminal

kafka-console-producer --topic example-main-topic --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Topic to send messages to

→

--bootstrap-server - Kafka server address

Start a consumer to read messages from the main topic from the beginning. This consumer should be configured to send bad messages to the DLQ after retries.

Terminal

kafka-console-consumer --topic example-main-topic --bootstrap-server localhost:9092 --group example-consumer-group --from-beginning

Expected Output

This command will output messages from the topic as they arrive, for example:

Hello World

BadMessage

Another Good Message

→

--group - Consumer group id

→

--from-beginning - Read messages from the start of the topic

Consume messages from the dead letter queue topic to inspect failed messages.

Terminal

kafka-console-consumer --topic example-dead-letter-queue --bootstrap-server localhost:9092 --from-beginning

Expected Output

BadMessage

MalformedData

TimeoutError

→

--from-beginning - Read all messages from the DLQ topic

Key Concept

If you remember nothing else from this pattern, remember: move messages that fail processing repeatedly to a separate dead letter queue topic to keep your main flow running smoothly.

Common Mistakes

Not creating the dead letter queue topic before sending messages to it.

Kafka will reject messages sent to a non-existent topic, causing errors.

Always create the DLQ topic with kafka-topics before producing messages to it.

Not handling retries in the consumer before sending messages to the DLQ.

Messages may be sent to the DLQ too early or not at all, causing loss or blocking.

Implement retry logic in the consumer to attempt processing multiple times before sending to DLQ.

Consuming the DLQ topic without monitoring or alerting.

Failed messages may pile up unnoticed, causing hidden problems.

Regularly monitor the DLQ topic and set alerts to investigate and fix issues.

Summary

Create both main and dead letter queue Kafka topics before use.

Send normal messages to the main topic and configure consumers to retry processing.

After retries fail, send bad messages to the dead letter queue topic for later inspection.

Consume the dead letter queue topic separately to monitor and fix message issues.

Practice

1. What is the main purpose of a Dead Letter Queue (DLQ) in Kafka?

easy

A. To store messages that failed processing for later analysis

B. To speed up message processing by duplicating messages

C. To permanently delete messages after processing

D. To encrypt messages before sending to consumers

Solution

1. Step 1: Understand the role of DLQ

   A Dead Letter Queue is used to hold messages that cannot be processed successfully by the main consumer.

2. Step 2: Identify the purpose of storing failed messages

   Storing failed messages allows developers to analyze and fix issues without blocking the main processing flow.

3. Final Answer:

   To store messages that failed processing for later analysis -> Option A

4. Quick Check:

   DLQ = store failed messages [OK]

Hint: DLQ holds failed messages to avoid blocking main flow [OK]

Common Mistakes:

• Thinking DLQ speeds up processing by duplicating messages
• Confusing DLQ with message deletion
• Assuming DLQ encrypts messages

2. Which Kafka configuration is commonly used to specify the Dead Letter Queue topic for a consumer?

easy

A. enable.auto.commit

B. auto.offset.reset

C. max.poll.records

D. dead.letter.queue.topic.name

Solution

1. Step 1: Identify DLQ related configuration

   The configuration dead.letter.queue.topic.name is used to specify the topic where failed messages are sent.

2. Step 2: Differentiate from other common configs

   Other options like enable.auto.commit and auto.offset.reset control offset behavior, not DLQ.

3. Final Answer:

   dead.letter.queue.topic.name -> Option D

4. Quick Check:

   DLQ topic config = dead.letter.queue.topic.name [OK]

Hint: DLQ topic config usually contains 'dead.letter.queue' phrase [OK]

Common Mistakes:

• Confusing offset configs with DLQ topic config
• Using unrelated consumer configs for DLQ
• Assuming DLQ is automatic without config

3. Given the following Kafka consumer code snippet, what happens when message processing fails?

try {
  processMessage(record);
} catch (Exception e) {
  producer.send(new ProducerRecord<>("dlq-topic", record.key(), record.value()));
}

medium

A. The message is silently dropped without notification

B. The message is retried immediately in the same consumer loop

C. The failed message is sent to the "dlq-topic" for later review

D. The consumer stops processing all messages

Solution

1. Step 1: Analyze the catch block behavior

   When an exception occurs, the code sends the failed message to the "dlq-topic" using a producer.

2. Step 2: Understand the effect on message processing

   The message is not retried or dropped silently; it is stored in the DLQ topic for later analysis.

3. Final Answer:

   The failed message is sent to the "dlq-topic" for later review -> Option C

4. Quick Check:

   Exception handling sends message to DLQ [OK]

Hint: Catch block sends failed messages to DLQ topic [OK]

Common Mistakes:

• Assuming immediate retry happens automatically
• Thinking consumer stops on failure
• Believing message is dropped silently

4. Identify the error in this Kafka consumer code snippet that tries to send failed messages to a DLQ:

try {
  process(record);
} catch (Exception e) {
  producer.send("dlq-topic", record.key(), record.value());
}

medium

A. The catch block should rethrow the exception

B. The producer.send method is missing a ProducerRecord object

C. The process method should not be inside try-catch

D. The record.key() cannot be used as a message key

Solution

1. Step 1: Check the producer.send method usage

   The producer.send method requires a ProducerRecord object, not separate parameters.

2. Step 2: Identify correct usage

   The correct call is producer.send(new ProducerRecord<>("dlq-topic", record.key(), record.value()));

3. Final Answer:

   The producer.send method is missing a ProducerRecord object -> Option B

4. Quick Check:

   producer.send needs ProducerRecord object [OK]

Hint: producer.send requires ProducerRecord, not separate args [OK]

Common Mistakes:

• Passing topic and data as separate arguments to send
• Ignoring exception handling best practices
• Misusing record key as invalid

5. You want to implement a Dead Letter Queue pattern in Kafka that retries failed messages up to 3 times before sending them to the DLQ. Which approach correctly achieves this?

hard

A. Use a retry counter in message headers; on failure increment and retry, else send to DLQ after 3 tries

B. Send all failed messages immediately to DLQ without retries

C. Configure Kafka broker to automatically retry messages 3 times before DLQ

D. Use a separate consumer that deletes messages after 3 retries

Solution

1. Step 1: Understand retry logic with DLQ

   Retries require tracking how many times a message was processed; using headers to count retries is common.

2. Step 2: Implement conditional retry or DLQ send

   If retry count is less than 3, retry processing; if 3 or more, send message to DLQ topic.

3. Step 3: Evaluate other options

   Kafka broker does not handle retries automatically; immediate DLQ send skips retries; deleting messages after retries is incorrect.

4. Final Answer:

   Use a retry counter in message headers; on failure increment and retry, else send to DLQ after 3 tries -> Option A

5. Quick Check:

   Retry count in headers controls DLQ after 3 tries [OK]

Hint: Track retries in headers; send to DLQ after max retries [OK]

Common Mistakes:

• Expecting Kafka broker to retry automatically
• Sending all failures immediately to DLQ
• Deleting messages instead of sending to DLQ