Bird
Raised Fist0

Dead Letter Queues and Retry Strategies for Failure Handling

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/8
Kafka Topic

Message published

Initial message is published to the main Kafka topic

Protocol:Kafka
BeforeMessage exists only on client side, not yet sent
AfterMessage durably stored in Kafka Topic, ready for consumers
Retry Worker

Consumes message and attempts processing

Retry Worker reads message from Kafka Topic and processes it

Protocol:Kafka Consumer
BeforeMessage is in Kafka Topic, unprocessed
AfterProcessing failed; message sent to Retry Queue for retry
Retry Queue (Kafka Topic)

Message queued for retry

Failed message is placed in Retry Queue for delayed reprocessing

Protocol:Kafka
BeforeMessage failed processing once, not yet retried
AfterMessage stored in Retry Queue, awaiting retry
Retry Worker

Retries message processing

Retry Worker consumes message from Retry Queue and retries processing

Protocol:Kafka Consumer
BeforeMessage is in Retry Queue, retry count below max
AfterProcessing failed again; message sent to Dead Letter Queue
Dead Letter Queue (Kafka Topic)

Message stored after max retries

Message is stored in Dead Letter Queue after exceeding retry limit

Protocol:Kafka
BeforeMessage failed retry processing multiple times
AfterMessage durably stored in Dead Letter Queue for analysis
DLQ Processor

Consumes dead letter messages

DLQ Processor reads messages from Dead Letter Queue for analysis

Protocol:Kafka Consumer
BeforeMessage is in Dead Letter Queue, unprocessed
AfterMessage analyzed; failure reason logged and metrics updated
Monitoring & Metrics (Prometheus + Grafana)

Collects failure metrics

Metrics on retries and dead letters are collected and visualized

Protocol:Metrics scraping
BeforeFailure events occurred but not yet monitored
AfterMetrics updated with retry and DLQ failure counts
Alerting System

Triggers alert on DLQ threshold breach

Alerting system sends notifications when DLQ messages exceed threshold

Protocol:Alerting API
BeforeDLQ message count below alert threshold
AfterAlert sent to ops team for investigation
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────────┐
│       Kafka Topic         │
│  (Main message queue)     │
└─────────────┬─────────────┘
              │ Kafka
              ▼
┌───────────────────────────┐
│      Retry Queue          │
│   (Kafka Topic)           │
└─────────────┬─────────────┘
              │ Kafka
              ▼
┌───────────────────────────┐
│      Dead Letter Queue    │
│     (Kafka Topic)         │
└─────────────┬─────────────┘
              │ Kafka
              ▼
┌───────────────────────────┐
│       Retry Worker        │
│ (Consumes from Retry Q)   │
└─────────────┬─────────────┘
              │ Processes & routes failed messages
              ▼
┌───────────────────────────┐
│       DLQ Processor       │
│ (Consumes from DLQ)       │
└─────────────┬─────────────┘
              │ Processes dead letters for analysis
              ▼
┌───────────────────────────┐
│ Monitoring & Metrics      │
│ (Prometheus + Grafana)    │
└─────────────┬─────────────┘
              │ Metrics data
              ▼
┌───────────────────────────┐
│     Alerting System       │
│ (Triggers alerts on errors) │
└───────────────────────────┘
Design Rationale

This architecture uses Kafka topics to implement retry and dead letter queues for robust failure handling. Messages first enter the main Kafka Topic. If processing fails, the Retry Worker moves them to the Retry Queue for delayed reprocessing. After exceeding retry limits, messages are routed to the Dead Letter Queue. The DLQ Processor consumes these for failure analysis. Monitoring via Prometheus and Grafana tracks retry and DLQ metrics, while the Alerting System notifies operators when failure thresholds are breached. This design decouples failure handling from main processing, ensuring durability and observability of failed messages.

Primary Focus

✓

Practice

(1/5)
1. In a collaborative editing system, at step 4 of the request flow, the OT engine receives two concurrent operations from users Alice and Bob. What does the OT engine do at this step to maintain document consistency?
easy
A. Transforms Bob's operation against Alice's operation to preserve intention and order
B. Rejects Bob's operation and requests a retry to avoid conflicts
C. Queues Bob's operation until Alice's operation is fully applied to the document
D. Applies Bob's operation directly, overwriting Alice's changes

Solution

  1. Step 1: Understand OT engine's role at concurrent operation receipt

    The OT engine must handle concurrent operations arriving out of order or simultaneously and merge them without losing user intent.

  2. Step 2: Recognize OT transformation process

    OT transforms one operation against another to adjust positions and effects, preserving both users' intentions and maintaining consistency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    OT transforms concurrent ops to preserve intention and order [check]
Hint: OT transforms concurrent ops to preserve intention and order [OK]
Common Mistakes:
2. You must design an index for a high-write table where queries mostly perform equality lookups on a single column with high cardinality. Which index type is the best trade-off for write performance and query speed?
medium
A. Hash index, because it provides constant-time equality lookups and faster writes
B. B-Tree index, because it supports all query types with balanced tree structure
C. Composite index including multiple columns to cover more queries
D. Bitmap index, because it compresses well and speeds up equality queries

Solution

  1. Step 1: Identify index type for high-write equality lookups

    Hash indexes provide O(1) equality lookups and have simpler insertions compared to B-Trees, improving write throughput.

  2. Step 2: Eliminate other options

    B-Tree indexes have higher write overhead due to tree balancing. Composite indexes add complexity and overhead without query need. Bitmap indexes are inefficient for high-cardinality columns and high write rates.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Hash index optimizes equality lookups and write speed [check]
Hint: Hash index excels at equality lookups with high write load [check]
Common Mistakes:
3. If the Kafka cluster responsible for fan-out message delivery crashes, what is the exact failure observed in the News Feed system?
medium
A. New posts are lost permanently and never delivered to followers
B. The system blocks new post writes until Kafka recovers, causing user-facing delays
C. Followers receive duplicate feed updates due to Kafka replaying messages
D. Delivery Workers cannot fetch fan-out messages, so followers see stale feeds until recovery

Solution

  1. Step 1: Understand Kafka's role in fan-out delivery

    Kafka acts as a durable queue for fan-out messages. If it crashes, Delivery Workers cannot fetch new messages, causing stale feeds.

  2. Step 2: Evaluate failure modes

    Posts are not lost immediately (A) due to Kafka's durability; writes are asynchronous so no blocking (B); duplicates (D) occur only on replay, not crash.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Kafka crash stalls delivery, causing stale feeds [check]
Hint: Kafka crash stalls delivery, causing stale feeds [check]
Common Mistakes:
4. For a faceted search system with frequent filter updates and millions of documents, which architecture better balances update latency and query freshness?
medium
A. Use a near-real-time indexing system that streams document changes to shards asynchronously
B. Use a batch indexing pipeline that rebuilds the entire inverted index every hour
C. Use a write-through cache that updates inverted indexes synchronously on every document change
D. Use a static precomputed index refreshed monthly with manual invalidation

Solution

  1. Step 1: Compare update latency and query freshness trade-offs

    Batch rebuilds have high latency; synchronous writes hurt write throughput; static monthly refresh is too stale.

  2. Step 2: Identify near-real-time indexing benefits

    Streaming updates asynchronously balances freshness and throughput, allowing queries to see recent changes with low delay.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Near-real-time streaming -> balanced freshness and update latency [check]
Hint: Near-real-time streaming balances freshness and throughput [check]
Common Mistakes:
5. At 10x scale, which component is most likely to become the first bottleneck in the stock exchange system, and why?
hard
A. The network layer, because TCP connections for order submission exceed server socket limits
B. The Market Data Feed, because broadcasting to millions of subscribers saturates network bandwidth
C. The Order Book storage, because the relational database cannot handle the write throughput
D. The Matching Engine, because partitioned order books require cross-partition coordination for large orders

Solution

  1. Step 1: Analyze scaling challenges

    Matching Engine partitions order books per stock symbol, but large cross-symbol or large orders require coordination, causing latency and bottlenecks at scale.

  2. Step 2: Evaluate other components

    Market Data Feed scales with multicast or sharding; Order Book storage uses scalable Kafka or NoSQL; network layer scales with load balancers and horizontal scaling.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Cross-partition coordination -> Matching Engine bottleneck at scale [check]
Hint: Cross-partition coordination bottlenecks matching engine [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard