How to Implement Dead Letter Queue in Kafka: Simple Guide

To implement a dead letter queue (DLQ) in Kafka, create a separate Kafka topic to store messages that fail processing. Configure your consumer or Kafka Streams application to catch processing errors and send those failed messages to the DLQ topic for later inspection or reprocessing.

Syntax

Implementing a dead letter queue in Kafka involves these parts:

• DLQ Topic: A dedicated Kafka topic to hold failed messages.
• Producer/Consumer Logic: Code that sends messages to the DLQ topic when processing fails.
• Error Handling: Catch exceptions during message processing to redirect messages.

java

try {
    // Process message
} catch (Exception e) {
    // Send message to DLQ topic
    producer.send(new ProducerRecord<>("dlq-topic", messageKey, messageValue));
}

Example

This example shows a simple Kafka consumer in Java that processes messages and sends failed ones to a dead letter queue topic named my-dlq-topic.

java

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class KafkaDLQExample {
    public static void main(String[] args) {
        Properties consumerProps = new Properties();
        consumerProps.put("bootstrap.servers", "localhost:9092");
        consumerProps.put("group.id", "my-group");
        consumerProps.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        consumerProps.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        Properties producerProps = new Properties();
        producerProps.put("bootstrap.servers", "localhost:9092");
        producerProps.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        producerProps.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProps);
             KafkaProducer<String, String> producer = new KafkaProducer<>(producerProps)) {

            consumer.subscribe(Collections.singletonList("my-topic"));

            while (true) {
                for (ConsumerRecord<String, String> record : consumer.poll(Duration.ofMillis(100))) {
                    try {
                        // Simulate processing
                        if (record.value().contains("fail")) {
                            throw new RuntimeException("Processing failed");
                        }
                        System.out.println("Processed message: " + record.value());
                    } catch (Exception e) {
                        // Send to DLQ
                        producer.send(new ProducerRecord<>("my-dlq-topic", record.key(), record.value()));
                        System.out.println("Sent to DLQ: " + record.value());
                    }
                }
            }
        }
    }
}

Output

Processed message: hello
Sent to DLQ: fail message
Processed message: world

Common Pitfalls

Common mistakes when implementing a Kafka dead letter queue include:

• Not creating the DLQ topic before sending messages, causing errors.
• Failing to handle exceptions properly, which can crash the consumer.
• Ignoring message keys, which can affect message ordering in the DLQ.
• Not monitoring the DLQ topic, missing failed messages.

java

try {
    // Process message
} catch (Exception e) {
    // Wrong: ignoring exception and not sending to DLQ
    // Correct: send failed message to DLQ topic
    producer.send(new ProducerRecord<>("dlq-topic", messageKey, messageValue));
}

Quick Reference

• Create a dedicated DLQ topic in Kafka.
• Catch processing errors in your consumer or stream app.
• Send failed messages to the DLQ topic with the original key and value.
• Monitor and process DLQ messages separately.

Key Takeaways

Create a separate Kafka topic to act as the dead letter queue.

Catch exceptions during message processing to redirect failed messages to the DLQ.

Always preserve message keys when sending to the DLQ to maintain ordering.

Monitor the DLQ topic regularly to handle and reprocess failed messages.

Ensure the DLQ topic exists before sending messages to avoid errors.