Dead letter queue pattern in Kafka - Step-by-Step Execution

Process Flow - Dead letter queue pattern

Message Produced

↓

Message Consumed

↓

Process Message

↓

Success

↓

Commit

↓

Dead Letter Queue

Messages are produced and consumed. If processing succeeds, message is committed. If it fails, message is sent to a Dead Letter Queue (DLQ) for later inspection.

Execution Sample

Kafka

message = consumer.poll()
try:
  process(message)
  consumer.commit()
except Exception:
  producer.send('DLQ', message)
  consumer.commit()

Consume a message, try to process it, commit if success, else send to DLQ.

Process Table

Step	Action	Message Content	Processing Result	Queue Destination	Output
1	Poll message	Order #123	N/A	Main Topic	Message received
2	Process message	Order #123	Success	Main Topic	Processed successfully
3	Commit offset	Order #123	N/A	Main Topic	Offset committed
4	Poll message	Order #124	N/A	Main Topic	Message received
5	Process message	Order #124	Failure	Main Topic	Processing failed
6	Send to DLQ	Order #124	N/A	Dead Letter Queue	Message sent to DLQ
7	Poll message	Order #125	N/A	Main Topic	Message received
8	Process message	Order #125	Success	Main Topic	Processed successfully
9	Commit offset	Order #125	N/A	Main Topic	Offset committed

💡 No more messages to consume

Status Tracker

Variable	Start	After Step 3	After Step 6	After Step 9	Final
message	None	Order #123	Order #124	Order #125	Order #125
processing_result	None	Success	Failure	Success	Success
queue_destination	Main Topic	Main Topic	Dead Letter Queue	Main Topic	Main Topic

Key Moments - 3 Insights

Why do we send the message to the Dead Letter Queue instead of just discarding it?

Because the message failed processing and we want to keep it for later inspection or reprocessing, as shown in execution_table step 6.

What happens if the message processing succeeds?

The message offset is committed to mark it as processed, so it won't be consumed again, as shown in execution_table steps 3 and 9.

Does sending to the DLQ stop the consumer from processing other messages?

No, the consumer continues polling and processing other messages normally, as seen in steps 7-9.

Visual Quiz - 3 Questions

Look at the execution_table, what is the queue destination for the message 'Order #124' after processing fails?

A Main Topic

B Dead Letter Queue

C Committed Offsets

D Discarded

💡 Hint

Check step 6 in the execution_table where the message is sent after failure.

At which step does the consumer commit the offset for a successfully processed message?

A Step 2

B Step 5

C Step 3

D Step 6

💡 Hint

Look for 'Commit offset' action in the execution_table.

If the processing of 'Order #125' failed, how would the queue destination change in the execution_table?

A It would change to 'Dead Letter Queue'

B It would be 'Committed Offsets'

C It would remain 'Main Topic'

D It would be 'Discarded'

💡 Hint

Refer to how failure is handled for 'Order #124' in steps 5 and 6.

Concept Snapshot

Dead Letter Queue Pattern:
- Consume messages from main topic
- Try processing each message
- On success: commit offset
- On failure: send message to DLQ
- DLQ stores failed messages for later handling

Full Transcript

This visual trace shows how the Dead Letter Queue pattern works in Kafka. Messages are consumed from the main topic. Each message is processed. If processing succeeds, the consumer commits the offset so the message won't be reprocessed. If processing fails, the message is sent to a special topic called the Dead Letter Queue (DLQ). This DLQ holds failed messages for later inspection or retry. The consumer continues processing other messages normally. This pattern helps handle errors without losing messages or blocking the consumer.