Dead letter queue pattern in Kafka - Mini Project: Build & Apply

Dead letter queue pattern

📖 Scenario: You are building a message processing system using Kafka. Sometimes, messages fail to process correctly. To handle these failures gracefully, you want to send the failed messages to a special topic called a dead letter queue (DLQ). This helps you keep track of problematic messages and fix them later without losing data.

🎯 Goal: Create a Kafka consumer that reads messages from a main topic, tries to process them, and if processing fails, sends the failed messages to a dead letter queue topic.

📋 What You'll Learn

Create a Kafka consumer to read messages from the main_topic.

Create a Kafka producer to send messages to the dead_letter_queue topic.

Implement a simple message processing function that can fail for certain messages.

If processing fails, send the message to the dead letter queue.

Print the result of processing or failure for each message.

💡 Why This Matters

🌍 Real World

Dead letter queues help production systems handle message failures without losing data. They allow developers to inspect and fix problematic messages later.

💼 Career

Understanding dead letter queues is important for roles in backend development, data engineering, and system reliability engineering where message processing systems are common.

1

Set up Kafka consumer and producer

Create a Kafka consumer subscribed to the main_topic and a Kafka producer for sending messages to the dead_letter_queue. Use the variable names consumer and producer respectively.

Kafka

# Your code here to create consumer and producer

Hint

Use KafkaConsumer to subscribe to 'main_topic' and KafkaProducer to send messages.

2

Create a message processing function

Define a function called process_message that takes a message string. It should raise a ValueError if the message contains the word 'fail', otherwise return 'Processed: ' plus the message.

Kafka

def process_message(message: str) -> str:
    if 'fail' in message:
        raise ValueError('Processing failed')
    return 'Processed: ' + message
# Your code here to process messages and send failures to DLQ

Hint

Check if 'fail' is in the message string and raise an error if so.

3

Process messages and send failures to DLQ

Write a for loop that reads messages from consumer. For each message, decode it to a string, then try to process it using process_message. If processing raises an error, send the original message bytes to the dead_letter_queue topic using producer.send.

Kafka

for msg in consumer:
    message_str = msg.value.decode('utf-8')
    try:
        result = process_message(message_str)
    except ValueError:
        producer.send('dead_letter_queue', msg.value)
        result = 'Sent to DLQ: ' + message_str
    # Your code here to print the result

Hint

Use try and except to catch errors and send failed messages to the DLQ.

4

Print processing results

Inside the for loop, after processing or sending to DLQ, print the result variable to show what happened to each message.

Kafka

Hint

Use print(result) to show the processing outcome.

Practice

1. What is the main purpose of a Dead Letter Queue (DLQ) in Kafka?

easy

A. To store messages that failed processing for later analysis

B. To speed up message processing by duplicating messages

C. To permanently delete messages after processing

D. To encrypt messages before sending to consumers

Solution

1. Step 1: Understand the role of DLQ

   A Dead Letter Queue is used to hold messages that cannot be processed successfully by the main consumer.

2. Step 2: Identify the purpose of storing failed messages

   Storing failed messages allows developers to analyze and fix issues without blocking the main processing flow.

3. Final Answer:

   To store messages that failed processing for later analysis -> Option A

4. Quick Check:

   DLQ = store failed messages [OK]

Hint: DLQ holds failed messages to avoid blocking main flow [OK]

Common Mistakes:

• Thinking DLQ speeds up processing by duplicating messages
• Confusing DLQ with message deletion
• Assuming DLQ encrypts messages

2. Which Kafka configuration is commonly used to specify the Dead Letter Queue topic for a consumer?

easy

A. enable.auto.commit

B. auto.offset.reset

C. max.poll.records

D. dead.letter.queue.topic.name

Solution

1. Step 1: Identify DLQ related configuration

   The configuration dead.letter.queue.topic.name is used to specify the topic where failed messages are sent.

2. Step 2: Differentiate from other common configs

   Other options like enable.auto.commit and auto.offset.reset control offset behavior, not DLQ.

3. Final Answer:

   dead.letter.queue.topic.name -> Option D

4. Quick Check:

   DLQ topic config = dead.letter.queue.topic.name [OK]

Hint: DLQ topic config usually contains 'dead.letter.queue' phrase [OK]

Common Mistakes:

• Confusing offset configs with DLQ topic config
• Using unrelated consumer configs for DLQ
• Assuming DLQ is automatic without config

3. Given the following Kafka consumer code snippet, what happens when message processing fails?

try {
  processMessage(record);
} catch (Exception e) {
  producer.send(new ProducerRecord<>("dlq-topic", record.key(), record.value()));
}

medium

A. The message is silently dropped without notification

B. The message is retried immediately in the same consumer loop

C. The failed message is sent to the "dlq-topic" for later review

D. The consumer stops processing all messages

Solution

1. Step 1: Analyze the catch block behavior

   When an exception occurs, the code sends the failed message to the "dlq-topic" using a producer.

2. Step 2: Understand the effect on message processing

   The message is not retried or dropped silently; it is stored in the DLQ topic for later analysis.

3. Final Answer:

   The failed message is sent to the "dlq-topic" for later review -> Option C

4. Quick Check:

   Exception handling sends message to DLQ [OK]

Hint: Catch block sends failed messages to DLQ topic [OK]

Common Mistakes:

• Assuming immediate retry happens automatically
• Thinking consumer stops on failure
• Believing message is dropped silently

4. Identify the error in this Kafka consumer code snippet that tries to send failed messages to a DLQ:

try {
  process(record);
} catch (Exception e) {
  producer.send("dlq-topic", record.key(), record.value());
}

medium

A. The catch block should rethrow the exception

B. The producer.send method is missing a ProducerRecord object

C. The process method should not be inside try-catch

D. The record.key() cannot be used as a message key

Solution

1. Step 1: Check the producer.send method usage

   The producer.send method requires a ProducerRecord object, not separate parameters.

2. Step 2: Identify correct usage

   The correct call is producer.send(new ProducerRecord<>("dlq-topic", record.key(), record.value()));

3. Final Answer:

   The producer.send method is missing a ProducerRecord object -> Option B

4. Quick Check:

   producer.send needs ProducerRecord object [OK]

Hint: producer.send requires ProducerRecord, not separate args [OK]

Common Mistakes:

• Passing topic and data as separate arguments to send
• Ignoring exception handling best practices
• Misusing record key as invalid

5. You want to implement a Dead Letter Queue pattern in Kafka that retries failed messages up to 3 times before sending them to the DLQ. Which approach correctly achieves this?

hard

A. Use a retry counter in message headers; on failure increment and retry, else send to DLQ after 3 tries

B. Send all failed messages immediately to DLQ without retries

C. Configure Kafka broker to automatically retry messages 3 times before DLQ

D. Use a separate consumer that deletes messages after 3 retries

Solution

1. Step 1: Understand retry logic with DLQ

   Retries require tracking how many times a message was processed; using headers to count retries is common.

2. Step 2: Implement conditional retry or DLQ send

   If retry count is less than 3, retry processing; if 3 or more, send message to DLQ topic.

3. Step 3: Evaluate other options

   Kafka broker does not handle retries automatically; immediate DLQ send skips retries; deleting messages after retries is incorrect.

4. Final Answer:

   Use a retry counter in message headers; on failure increment and retry, else send to DLQ after 3 tries -> Option A

5. Quick Check:

   Retry count in headers controls DLQ after 3 tries [OK]

Hint: Track retries in headers; send to DLQ after max retries [OK]

Common Mistakes:

• Expecting Kafka broker to retry automatically
• Sending all failures immediately to DLQ
• Deleting messages instead of sending to DLQ