How to Use Dead Letter Topic in Spring Kafka for Error Handling

In Spring Kafka, you use a DeadLetterPublishingRecoverer with a SeekToCurrentErrorHandler to send failed messages to a dead letter topic. Configure your KafkaListenerContainerFactory to use this error handler so that messages that fail processing after retries are forwarded to the dead letter topic automatically.

Syntax

The main components to use a dead letter topic in Spring Kafka are:

• DeadLetterPublishingRecoverer: Sends failed messages to the dead letter topic.
• SeekToCurrentErrorHandler: Handles retries and delegates to the recoverer after retries fail.
• ConcurrentKafkaListenerContainerFactory: Configured with the error handler.

Basic syntax example:

java

DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(kafkaTemplate);

SeekToCurrentErrorHandler errorHandler = new SeekToCurrentErrorHandler(recoverer, new FixedBackOff(1000L, 2));

ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory);
factory.setErrorHandler(errorHandler);

Example

This example shows how to configure Spring Kafka to send failed messages to a dead letter topic after 2 retries with 1 second delay.

java

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.SeekToCurrentErrorHandler;
import org.springframework.util.backoff.FixedBackOff;

@Configuration
public class KafkaConfig {

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory(
            ConsumerFactory<String, String> consumerFactory,
            KafkaTemplate<String, String> kafkaTemplate) {

        DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(kafkaTemplate);
        SeekToCurrentErrorHandler errorHandler = new SeekToCurrentErrorHandler(recoverer, new FixedBackOff(1000L, 2));

        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory);
        factory.setErrorHandler(errorHandler);
        return factory;
    }

    @KafkaListener(topics = "my-topic", groupId = "group_id")
    public void listen(ConsumerRecord<String, String> record) {
        if (record.value().contains("fail")) {
            throw new RuntimeException("Processing failed");
        }
        System.out.println("Processed message: " + record.value());
    }
}

Output

Processed message: hello
Processed message: world
// For messages containing "fail", after 2 retries, message is sent to dead letter topic "my-topic.DLT"

Common Pitfalls

• Not configuring the DeadLetterPublishingRecoverer causes failed messages to be lost or retried indefinitely.
• Forgetting to set the error handler on the listener container factory means dead letter topics won't be used.
• Not creating the dead letter topic in Kafka beforehand can cause errors; ensure the dead letter topic exists or enable auto-creation.
• Using infinite retries without a dead letter topic can block processing on bad messages.

java

/* Wrong: No error handler set, messages fail silently or retry forever */
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory);

/* Right: Set error handler with dead letter recoverer */
DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(kafkaTemplate);
SeekToCurrentErrorHandler errorHandler = new SeekToCurrentErrorHandler(recoverer, new FixedBackOff(1000L, 2));
factory.setErrorHandler(errorHandler);

Quick Reference

Dead Letter Topic Setup Tips:

• Use DeadLetterPublishingRecoverer with SeekToCurrentErrorHandler for retries and dead letter forwarding.
• Configure your ConcurrentKafkaListenerContainerFactory with the error handler.
• Set retry count and backoff delay in FixedBackOff.
• Ensure dead letter topic exists or enable auto-creation in Kafka.
• Dead letter topic name defaults to {original-topic}.DLT.

Key Takeaways

Configure DeadLetterPublishingRecoverer with SeekToCurrentErrorHandler to handle failed messages.

Set the error handler on your KafkaListenerContainerFactory to enable dead letter topic usage.

Use FixedBackOff to control retry attempts and delay before sending to dead letter topic.

Ensure the dead letter topic exists or Kafka auto-creates it to avoid errors.

Dead letter topics help isolate and analyze messages that fail processing after retries.