Dead letter queue pattern in Kafka - Mini Project: Build & Apply

Dead letter queue pattern

📖 Scenario: You are building a message processing system using Kafka. Sometimes, messages fail to process correctly. To handle these failures gracefully, you want to send the failed messages to a special topic called a dead letter queue (DLQ). This helps you keep track of problematic messages and fix them later without losing data.

🎯 Goal: Create a Kafka consumer that reads messages from a main topic, tries to process them, and if processing fails, sends the failed messages to a dead letter queue topic.

📋 What You'll Learn

Create a Kafka consumer to read messages from the main_topic.

Create a Kafka producer to send messages to the dead_letter_queue topic.

Implement a simple message processing function that can fail for certain messages.

If processing fails, send the message to the dead letter queue.

Print the result of processing or failure for each message.

💡 Why This Matters

🌍 Real World

Dead letter queues help production systems handle message failures without losing data. They allow developers to inspect and fix problematic messages later.

💼 Career

Understanding dead letter queues is important for roles in backend development, data engineering, and system reliability engineering where message processing systems are common.

1

Set up Kafka consumer and producer

Create a Kafka consumer subscribed to the main_topic and a Kafka producer for sending messages to the dead_letter_queue. Use the variable names consumer and producer respectively.

Kafka

# Your code here to create consumer and producer

Need a hint?

Use KafkaConsumer to subscribe to 'main_topic' and KafkaProducer to send messages.

2

Create a message processing function

Define a function called process_message that takes a message string. It should raise a ValueError if the message contains the word 'fail', otherwise return 'Processed: ' plus the message.

Kafka

def process_message(message: str) -> str:
    if 'fail' in message:
        raise ValueError('Processing failed')
    return 'Processed: ' + message
# Your code here to process messages and send failures to DLQ

Need a hint?

Check if 'fail' is in the message string and raise an error if so.

3

Process messages and send failures to DLQ

Write a for loop that reads messages from consumer. For each message, decode it to a string, then try to process it using process_message. If processing raises an error, send the original message bytes to the dead_letter_queue topic using producer.send.

Kafka

for msg in consumer:
    message_str = msg.value.decode('utf-8')
    try:
        result = process_message(message_str)
    except ValueError:
        producer.send('dead_letter_queue', msg.value)
        result = 'Sent to DLQ: ' + message_str
    # Your code here to print the result

Need a hint?

Use try and except to catch errors and send failed messages to the DLQ.

4

Print processing results

Inside the for loop, after processing or sending to DLQ, print the result variable to show what happened to each message.

Kafka

Need a hint?

Use print(result) to show the processing outcome.