Bird
Raised Fist0

Dead Letter Queues and Retry Strategies for Failure Handling

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Dead Letter Queues and Retry Strategies for Failure Handling
Robust failure handling in event-driven systems using DLQs and retries
mediumMessage Queues and Event StreamingFreq: common
⚙️
Functional Requirements
  • Capture and isolate failed messages that cannot be processed after retries into a Dead Letter Queue (DLQ) for later inspection.
  • Implement configurable retry strategies (exponential backoff, fixed delay) for transient failures before moving messages to DLQ.
  • Support message visibility timeout and poison pill detection to avoid infinite retry loops.
  • Provide monitoring and alerting on DLQ size and retry failure rates for operational awareness.
  • Allow manual or automated reprocessing of messages from DLQ after root cause resolution.
  • Ensure message ordering and idempotency guarantees are preserved during retries and DLQ handling.
📊
Non-Functional Requirements
  • System must handle 2 billion daily active users generating 40 messages per user per day, totaling 80 billion messages/day.
  • End-to-end message processing latency P99 must be under 100ms excluding retries.
  • System availability must be 99.99% to avoid message loss or duplication.
  • DLQ storage must retain failed messages for at least 7 days with replication for durability.
  • Retry mechanisms must not cause message processing throughput degradation beyond 5% during peak load.
❓
Clarifying Questions to Ask
  • What types of failures are expected: transient (network, downstream) or permanent (schema errors)?
  • What is the maximum retry count and retry delay policy per message?
  • Are messages ordered per key, and must ordering be preserved during retries?
  • What visibility timeout or lease duration is expected for message processing?
  • Is there a requirement for automatic DLQ message reprocessing or manual only?
  • What monitoring and alerting tools are currently used or preferred?
Out of Scope
✗ Design of the primary message queue or event streaming platform itself (assumed Kafka).
✗ Business logic of message processing or message schema design.
✗ Authentication and authorization of message producers or consumers.
✗ Long-term archival or analytics on DLQ messages beyond 7 days retention.
📌
Assumptions
  • 2 billion DAU
  • 40 messages per user per day
  • 100 bytes average message size
  • 7 days DLQ retention
  • Replication factor 3 for durability
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 msg/sec
peak write qpsAssuming 3× peak factor: 925,926 × 3 ≈ 2,777,778 msg/sec
read qpsAssuming 1.5× read for retries and consumers: 925,926 × 1.5 ≈ 1,388,889 msg/sec
💾
Storage Estimate
daily storage gb(2B × 40 × 100 bytes) / (1024³) ≈ 7.45 TB/day
retention storage gb7.45 TB × 7 days ≈ 52.15 TB
with replication gb52.15 TB × 3 replicas ≈ 156.45 TB
🖥️
Server Estimate
kafka brokersTo handle 2.7M msg/sec peak with 100 bytes/msg, assuming 100K msg/sec per broker, need ~28 brokers
retry worker nodesAssuming each worker handles 10K retries/sec, for 1.4M read QPS, need ~140 workers
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          | Message Producer|
          +--------+-------+
                   |
                   v
          +----------------+
          |   Kafka Topic  |
          +--------+-------+
                   |
                   v
          +----------------+
          |  Consumer App  |
          +--------+-------+
                   |
          +--------+--------+
          |                 |
          v                 v
   +-------------+    +--------------+
   | Retry Queue |    | Dead Letter  |
   |  (Kafka)   |    | Queue (Kafka)|
   +------+------+    +------+-------+
          |                   |
          v                   v
   +-------------+     +--------------+
   | Retry Worker|     | DLQ Processor|
   +-------------+     +--------------+
          |                   |
          v                   v
   +----------------+  +----------------+
   | Monitoring &   |  | Alerting System|
   | Metrics (Prom) |  +----------------+
   +----------------+
🧩
Components
Kafka Topic
Primary message queue for event streaming and message delivery
Why thisKafka supports high throughput (millions msg/sec), partitioning for scalability, and exactly-once semantics with idempotent producers
Why notRabbitMQ lacks Kafka's horizontal scalability and partitioning at 2.7M msg/sec peak
Retry Queue (Kafka Topic)
Separate Kafka topic to hold messages scheduled for retry with delay semantics
Why thisKafka's log compaction and partitioning enable scalable delayed retries with retention policies
Why notRedis Streams lacks native delayed message support and scales poorly beyond 100K msg/sec
Dead Letter Queue (Kafka Topic)
Kafka topic to isolate messages that failed all retries for offline inspection and reprocessing
Why thisKafka's durability and replication ensure no message loss and easy integration with monitoring tools
Why notSQS DLQ lacks native ordering and high throughput needed at 2.7M msg/sec
Retry Worker
Service that consumes from Retry Queue, applies retry logic with backoff, and re-injects messages into main topic or DLQ
Why thisCustom worker allows flexible retry policies and poison pill detection with visibility timeout management
Why notKafka Connect lacks fine-grained retry control and visibility timeout handling
DLQ Processor
Service to analyze, alert, and optionally reprocess messages from DLQ after manual or automated fixes
Why thisDedicated processor enables operational workflows and root cause analysis integration
Why notManual inspection only is error-prone and does not scale for 52TB DLQ data
Monitoring & Metrics (Prometheus + Grafana)
Collect and visualize metrics on retry rates, DLQ size, and processing latencies
Why thisPrometheus supports high cardinality metrics and alerting rules for operational visibility
Why notCloudWatch or Datadog may not handle custom Kafka metrics at scale without additional cost
Alerting System
Trigger alerts based on DLQ growth or retry failure thresholds to notify SREs
Why thisIntegration with PagerDuty or Opsgenie enables rapid incident response
Why notEmail-only alerts cause delayed response and lack escalation capabilities

Practice

(1/5)
1. For a global service requiring low latency and high availability, which DNS load balancing approach is best when client IPs are often masked by VPNs or proxies?
medium
A. Use DNS-based latency measurements to route clients dynamically
B. Use GeoDNS with IP-based geolocation to route clients
C. Use Anycast IP routing combined with DNS round-robin for global distribution
D. Use a centralized DNS server that manually assigns IPs based on user profiles

Solution

  1. Step 1: Analyze IP masking impact

    GeoDNS relying on client IP (A) fails when IPs are masked by VPNs/proxies, causing inaccurate geolocation.

  2. Step 2: Evaluate alternatives

    Anycast IP routing (B) routes clients to nearest data center at network level, unaffected by IP masking. DNS latency-based routing (C) adds complexity and latency. Centralized DNS (D) is a bottleneck and not scalable.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Anycast routes at network layer, bypassing IP masking issues [check]
Hint: Anycast routes clients despite IP masking, GeoDNS fails [check]
Common Mistakes:
2. You need to model a social media feed with millions of users and high write throughput. Which data modeling approach is best to optimize for write scalability and read performance?
medium
A. Embed all posts and comments inside each user document to minimize joins.
B. Use a graph database to model user relationships and posts for efficient traversal.
C. Store all posts in a single collection with a compound index on user and timestamp for fast queries.
D. Use a referencing pattern with separate collections for users, posts, and comments, and denormalize popular posts into user feed documents.

Solution

  1. Step 1: Analyze write scalability and read patterns

    High write throughput and millions of users require scalable writes and efficient feed reads.

  2. Step 2: Evaluate referencing with denormalization

    Referencing separates concerns; denormalizing popular posts into feeds optimizes read performance without large documents.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Referencing plus denormalization balances write scale and read speed [check]
Hint: Reference plus denormalize popular data for scalable writes and fast reads [check]
Common Mistakes:
3. For a high-throughput e-commerce system requiring strong consistency and complex joins, which database choice best balances performance and transactional integrity?
medium
A. Use a sharded relational database with two-phase commit for distributed transactions
B. Use an in-memory cache with asynchronous writes to a relational DB
C. Use a single-node relational database with vertical scaling
D. Use a distributed NoSQL key-value store with eventual consistency

Solution

  1. Step 1: Analyze requirements

    Strong consistency and complex joins require relational capabilities and transactional integrity across shards.

  2. Step 2: Evaluate options

    Sharded relational DB with two-phase commit supports distributed transactions and joins, unlike NoSQL or cache-based approaches.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Sharded RDB + 2PC balances scale and ACID [check]
Hint: Distributed transactions need 2PC, not eventual consistency stores [check]
Common Mistakes:
4. At 10x scale, which component in Spotify's streaming architecture is most likely to fail first and why?
hard
A. The metadata database due to increased write contention on user playlists
B. The streaming servers because maintaining millions of concurrent WebSocket connections is stateful and hard to shard
C. The CDN edge caches due to limited storage capacity per region
D. The offline download manager due to device storage limits

Solution

  1. Step 1: Identify stateful bottlenecks at scale

    Streaming servers maintain WebSocket connections which are stateful and tied to server memory.

  2. Step 2: Evaluate scaling challenges

    Metadata DB (A) can be sharded; CDN caches (B) scale by adding regions; offline manager (D) limited by device, not server scale. Streaming servers (C) face connection limits and resharding complexity.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Stateful WebSocket servers fail first at 10x scale [check]
Hint: Stateful WebSocket servers bottleneck at scale [check]
Common Mistakes:
5. At 10x scale, which component in the SLO error budget tracking system is most likely to become the bottleneck first, and why?
hard
A. The metrics aggregation service, because it maintains stateful counters and cannot shard error budget calculations easily
B. The dashboard visualization layer, because rendering 10x more data overwhelms client browsers
C. The raw event ingestion pipeline, because Kafka partitions can be increased indefinitely without limit
D. The alerting system, because it can batch alerts and scale horizontally without state

Solution

  1. Step 1: Identify stateful components that limit horizontal scaling

    The metrics aggregation service maintains counters and error budget state, which are hard to shard without losing accuracy.

  2. Step 2: Evaluate other components' scalability

    Dashboards (A) are client-side and less critical; Kafka (C) scales well with partitions; alerting (D) can batch and scale horizontally.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Stateful aggregation -> sharding limits -> bottleneck at scale [check]
Hint: Stateful aggregation limits sharding, causing first bottleneck at scale [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard