Bird
Raised Fist0

Webhook Delivery System - Reliability, Retry, Ordering Guarantees

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Webhook Delivery System - Reliability, Retry, Ordering Guarantees
Design a highly reliable, ordered webhook delivery system with retry and failure handling at scale
hardModern Infrastructure DesignFreq: medium
Prerequisites:System Design FundamentalsCAP TheoremDistributed Messaging SystemsIdempotency and Exactly-Once Delivery Concepts
⚙️
Functional Requirements
  • Accept webhook event requests from upstream services and persist them reliably.
  • Deliver webhook events to client endpoints with at-least-once delivery guarantee.
  • Support configurable retry policies with exponential backoff for failed deliveries.
  • Maintain strict ordering guarantees per webhook subscription or client endpoint.
  • Provide monitoring and alerting on delivery success, failures, and retries.
  • Allow clients to register, update, and delete webhook endpoints.
📊
Non-Functional Requirements
  • Support 2 billion daily webhook events (DAU) with peak write QPS ~925K.
  • Ensure 99.99% delivery success rate within retry window.
  • End-to-end delivery latency P99 under 100ms for successful deliveries.
  • System availability target of 99.999% (five nines).
  • Retry window configurable up to 24 hours with exponential backoff.
❓
Clarifying Questions to Ask
  • What is the maximum payload size per webhook event?
  • Are webhook endpoints expected to acknowledge delivery synchronously or asynchronously?
  • Is ordering required globally or scoped per client or subscription?
  • What is the expected retry policy and maximum retry duration?
  • Are duplicate deliveries acceptable or must we guarantee exactly-once?
  • What failure modes should the system tolerate (network, client downtime, etc.)?
Out of Scope
✗ Authentication and authorization of webhook event producers.
✗ Payload transformation or enrichment before delivery.
✗ Client-side webhook endpoint implementation or SDKs.
✗ Long-term archival of webhook event payloads beyond retention.
📌
Assumptions
  • 2 billion daily webhook events (DAU).
  • Each user generates 40 webhook events per day on average.
  • Average webhook event payload size is 100 bytes.
  • Retention period for retry and audit logs is 7 days.
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 writes/sec
peak write qpsAssuming 2× peak factor: 925,926 × 2 ≈ 1.85M writes/sec
read qpsAssuming 1.5× write QPS for retries and monitoring: 1.85M × 1.5 ≈ 2.78M reads/sec
💾
Storage Estimate
daily storage2B × 100 bytes = 200 GB/day
retention storage200 GB × 7 days = 1.4 TB
with replication1.4 TB × 3 replicas = 4.2 TB total storage
🖥️
Server Estimate
write pathKafka cluster with 50 brokers to handle 1.85M writes/sec (each broker ~40K writes/sec capacity)
delivery workers500 delivery worker instances (each handles ~3.7K deliveries/sec) for webhook dispatch and retries
🏗️
Architecture Diagram
Architecture Diagram
  +------------+       +------------+       +----------------+
  | Webhook    |       | Kafka      |       | Delivery       |
  | Producer   | ----> | Cluster    | ----> | Worker Pool    |
  +------------+       +------------+       +----------------+
                             |                      |
                             |                      v
                             |               +-------------+
                             |               | Client      |
                             |               | Webhook     |
                             |               | Endpoint    |
                             |               +-------------+
                             v
                      +--------------+
                      | Metadata     |
                      | Store        |
                      +--------------+
                             |
                             v
                      +--------------+
                      | Retry &      |
                      | Ordering     |
                      | Coordinator  |
                      +--------------+
                             |
                             v
                      +--------------+
                      | Monitoring & |
                      | Alerting     |
                      +--------------+
🧩
Components
Kafka Cluster
Durably ingest and buffer webhook events at high throughput with partitioning by client ID
Why thisKafka supports 10M+ writes/sec with partitioning and replication, ensuring durability and ordering per partition
Why notRabbitMQ lacks horizontal scalability and partition-level ordering at this scale; Redis Streams cannot handle 1.85M writes/sec reliably
Delivery Worker Pool
Consume events from Kafka, perform webhook delivery with retries and exponential backoff
Why thisStateless workers horizontally scale and can be restarted without losing state; use HTTP client libraries with timeout and circuit breaker support
Why notMonolithic delivery service would limit scalability and fault isolation; serverless functions have cold start latency and limited concurrency
Metadata Store (Cassandra)
Store webhook endpoint configurations, delivery status, and ordering metadata with low latency
Why thisCassandra provides linear scalability, tunable consistency, and efficient writes for high QPS workloads
Why notPostgreSQL requires complex manual sharding at 1.85M writes/sec; MongoDB's consistency model complicates ordering guarantees
Retry & Ordering Coordinator
Track delivery attempts, schedule retries with exponential backoff, and enforce ordering per client subscription
Why thisUse Apache Flink or Kafka Streams for stateful stream processing with exactly-once semantics and event-time timers
Why notCron jobs or external schedulers cannot guarantee ordering or handle dynamic retry windows at scale
Monitoring & Alerting System
Collect metrics on delivery success, failures, latency, and retry counts; trigger alerts on SLA breaches
Why thisPrometheus + Grafana stack supports high cardinality metrics and alerting with minimal overhead
Why notCustom monitoring solutions add operational complexity and lack community support
🔄
Data Flow
1
Webhook Producer → Kafka Cluster
Publish webhook event with client ID as partition key
Ensures ordering per client by partitioning
2
Kafka Cluster → Delivery Worker Pool
Workers consume events from assigned partitions
Parallel consumption with ordering guarantees per partition
3
Delivery Worker Pool → Client Webhook Endpoint
Send HTTP POST request with webhook event payload; await acknowledgment
Retries triggered on failure with exponential backoff
4
Delivery Worker Pool → Metadata Store
Update delivery status and ordering metadata
Ensures delivery tracking and ordering enforcement
5
Retry & Ordering Coordinator → Delivery Worker Pool
Schedule retries based on failure and ordering constraints
Uses event-time timers for precise retry scheduling
6
Monitoring & Alerting System → Operations Team
Send alerts on SLA breaches or abnormal failure rates
Enables proactive incident response

Practice

(1/5)
1. In a system using asynchronous replication, at step 4 the Replication Worker reads the write-ahead log entry for a committed transaction. What is the state of the replica after this step if the worker crashes before applying the entry?
easy
A. Replica has applied the transaction but not acknowledged it
B. Replica has not applied the transaction and will retry on restart
C. Replica has partially applied the transaction causing inconsistency
D. Replica has acknowledged the transaction despite crash

Solution

  1. Step 1: Trace Replication Worker reading WAL entry

    Worker reads the log but crashes before applying, so replica state unchanged.

  2. Step 2: Understand retry behavior on restart

    Since transaction not applied, worker retries applying on restart ensuring consistency.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Unapplied transaction retried after crash [check]
Hint: Unapplied logs retried after crash, no partial state [check]
Common Mistakes:
2. In the TLS handshake trace, at step 4 the server sends its certificate chain to the client. What does the client do immediately after receiving this chain at step 5?
easy
A. The client closes the connection if the certificate is expired without any retries
B. The client immediately sends encrypted application data without verification
C. The client requests a new certificate from the server
D. The client verifies the certificate chain against trusted root CAs

Solution

  1. Step 1: Understand client certificate validation

    After receiving the server's certificate chain, the client validates it against its trusted root certificate authorities to ensure authenticity.

  2. Step 2: Eliminate incorrect options

    The client immediately sends encrypted application data without verification is wrong because the client must verify before sending encrypted data. The client requests a new certificate from the server is incorrect; clients do not request certificates from servers. The client closes the connection if the certificate is expired without any retries is partially true but the client may attempt other steps like checking revocation before closing.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Client verifies certificate chain before proceeding [check]
Hint: Client verifies certificate chain before encrypted data [check]
Common Mistakes:
3. A distributed system requires a lock mechanism that ensures mutual exclusion across multiple Redis instances with minimal risk of deadlocks. Which component or approach best handles this requirement?
easy
A. Implement Redlock algorithm coordinating locks across multiple Redis masters
B. Use a relational database row lock with distributed transactions
C. Use a single Redis instance with SETNX and expire for locking
D. Use a centralized ZooKeeper lock service for distributed locking

Solution

  1. Step 1: Understand distributed lock requirements

    Single Redis instance with SETNX risks single point of failure and split-brain issues.

  2. Step 2: Evaluate Redlock's multi-instance coordination

    Redlock uses multiple Redis masters to reduce risk of deadlocks and ensure safety under partial failures.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Redlock coordinates multiple Redis masters for safer distributed locks [check]
Hint: Multiple Redis masters coordinated -> safer distributed locks [check]
Common Mistakes:
4. A web application experiences increasing CPU load on a single database server, causing query latency spikes. Which scaling approach best addresses this bottleneck without changing the application code?
easy
A. Distribute the database load by sharding data across multiple servers (horizontal scaling)
B. Add more CPU cores and RAM to the existing database server (vertical scaling)
C. Use a caching layer in front of the database to reduce read load
D. Switch from SQL to NoSQL database to improve write throughput

Solution

  1. Step 1: Identify the bottleneck as CPU and memory on a single DB server

    Vertical scaling adds resources to the existing server to handle more load without architectural changes.

  2. Step 2: Evaluate horizontal scaling and other options

    Horizontal scaling requires application changes for sharding; caching reduces reads but not CPU-bound writes; switching DB types is a different trade-off.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Vertical scaling adds resources to one server without code changes [check]
Hint: Vertical scaling adds power to one machine [check]
Common Mistakes:
5. If the Booking Service loses connectivity to the Calendar Availability Service during booking confirmation, what is the most likely failure mode experienced by the user?
medium
A. The booking is queued locally and confirmed later without user notification
B. The booking request times out and the user sees an error without confirmation
C. The booking is confirmed but the calendar is not updated, causing double bookings
D. The system automatically retries and confirms booking without user delay

Solution

  1. Step 1: Trace failure impact on booking flow

    Loss of connectivity prevents immediate calendar update, risking inconsistency.

  2. Step 2: Analyze failure handling

    Immediate confirmation without calendar update (A) risks double bookings; timeout with error (B) hurts UX; automatic retry without delay (D) is unrealistic under network failure. Local queuing (C) allows eventual consistency with delayed confirmation.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Local queuing delays confirmation but prevents data loss [check]
Hint: Booking queues locally on calendar service failure, delaying confirmation [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard