Bird
Raised Fist0

Webhook Delivery System - Reliability, Retry, Ordering Guarantees

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/5
Kafka Cluster

Event enqueued

Webhook event is durably stored in Kafka partition ordered by webhook ID

Protocol:Kafka protocol
BeforeWebhook event generated but not yet persisted
AfterEvent durably committed to Kafka with ordering key
Delivery Worker Pool

Consume event and attempt delivery

Worker fetches event from Kafka and sends HTTP webhook request

Protocol:Internal RPC
BeforeEvent available in Kafka, not yet delivered
AfterDelivery attempt made; success or failure recorded
Retry & Ordering Coordinator

Check delivery result and schedule retry if failed

Coordinator updates retry metadata and enforces ordering constraints

Protocol:CQL
BeforeDelivery result unknown, ordering state unchanged
AfterRetry scheduled in Metadata Store; ordering preserved
Metadata Store (Cassandra)

Persist delivery and retry state

Stores updated offsets, retry counts, and ordering info

Protocol:CQL
BeforePrevious delivery state stored, retry info outdated
AfterMetadata updated with latest delivery and retry info
Monitoring & Alerting System

Analyze metrics and alert on repeated failures

Monitors retry counts and triggers alerts if thresholds exceeded

Protocol:Metrics API
BeforeDelivery metrics collected but no alert triggered
AfterAlert triggered due to repeated delivery failures
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────────────┐
│        Kafka Cluster           │
│  (Durable event queue, ordered)│
└───────────────┬───────────────┘
                │ Kafka protocol
                ▼
┌───────────────────────────────┐
│    Delivery Worker Pool        │
│ (Consumes events, sends hooks)│
└───────────────┬───────────────┘
                │ Internal RPC
                ▼
┌───────────────────────────────┐
│ Retry & Ordering Coordinator   │
│ (Manages retries, enforces order)│
└───────────────┬───────────────┘
                │ CQL (Cassandra Query Language)
                ▼
┌───────────────────────────────┐
│ Metadata Store (Cassandra)     │
│ (Stores delivery state, offsets)│
└───────────────┬───────────────┘
                │ Metrics API
                ▼
┌───────────────────────────────┐
│ Monitoring & Alerting System   │
│ (Tracks failures, triggers alerts)│
└───────────────────────────────┘
Design Rationale

This architecture ensures reliable webhook delivery with ordering and retry guarantees. Events are durably stored in Kafka, preserving order by webhook ID. Delivery Workers consume events and attempt HTTP delivery. Failures are detected by the Retry & Ordering Coordinator, which updates retry metadata in Cassandra to schedule retries without violating ordering. The Metadata Store maintains delivery state and retry counts, enabling coordination across workers. The Monitoring & Alerting System tracks delivery health and triggers alerts on repeated failures, enabling rapid operational response. This design balances durability, ordering, and retry logic across components to ensure webhook events are delivered reliably and in order.

Primary Focus

✓

Practice

(1/5)
1. A payment system must ensure that duplicate payment requests from clients do not result in multiple charges. Which component is primarily responsible for handling idempotency in this scenario?
easy
A. The load balancer, by filtering duplicate requests before they reach backend services
B. The database, by enforcing unique constraints on payment amounts and timestamps
C. The client SDK, by retrying requests only once and caching responses locally
D. The transaction processing service, by storing and checking unique request IDs before processing

Solution

  1. Step 1: Identify idempotency enforcement point

    The transaction processing service must track unique request IDs to prevent duplicate charges, as it controls payment execution.

  2. Step 2: Eliminate other options

    The load balancer cannot reliably detect duplicates due to statelessness; the database unique constraints cannot cover all idempotency cases; client SDK retries are unreliable and cannot guarantee global idempotency.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Transaction service stores request IDs -> prevents duplicate processing [check]
Hint: Idempotency enforced where payment logic executes [check]
Common Mistakes:
2. In a system generating KSUIDs, at step 4 the ID generator component appends a random payload after the timestamp. What is the state of the generated ID at this step?
easy
A. The ID is a UUIDv4 embedded inside the KSUID format
B. The ID is fully deterministic and only contains the timestamp
C. The ID contains a timestamp prefix followed by a cryptographically secure random payload
D. The ID is a sequential number incremented per node

Solution

  1. Step 1: Understand KSUID structure at step 4

    KSUIDs combine a 4-byte timestamp prefix with a 16-byte random payload to ensure uniqueness and sortability.

  2. Step 2: Eliminate incorrect states

    Deterministic timestamp only (A) lacks uniqueness; UUIDv4 (C) is unrelated; sequential number (D) is not part of KSUID design.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    KSUID = timestamp + random payload for uniqueness and sortability [check]
Hint: KSUID = timestamp prefix + random payload [check]
Common Mistakes:
3. If the event ingestion pipeline fails to persist incoming metric data, what is the exact failure observed in the alerting system?
medium
A. Threshold and anomaly detectors receive incomplete data, causing missed or false alerts
B. Alerts are generated with delayed timestamps but eventually delivered
C. Composite alerts continue to function normally using cached inputs
D. The system switches to a backup data source automatically without alert impact

Solution

  1. Step 1: Role of event ingestion pipeline

    This pipeline persists raw metric data for downstream alerting components.

  2. Step 2: Consequence of pipeline failure

    Without persistence, threshold and anomaly detectors get incomplete or missing data, leading to missed alerts or false positives.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Missing data -> detectors miss or misfire alerts [check]
Hint: No data persistence -> detectors get incomplete data, miss alerts [check]
Common Mistakes:
4. If the BFF layer crashes and is unavailable, what is the immediate failure impact on client requests?
medium
A. Microservices handle requests directly, bypassing BFF automatically
B. Clients' requests fail immediately as BFF is the only entry point for client-specific aggregation
C. API Gateway retries requests indefinitely until BFF recovers
D. Clients receive stale cached data from API Gateway without error

Solution

  1. Step 1: Understand BFF's role in request flow

    BFF is the dedicated layer for client-specific aggregation; if it crashes, requests cannot be processed as usual.

  2. Step 2: Analyze fallback behavior

    Microservices handle requests directly, bypassing BFF automatically is correct because some architectures allow API Gateway to fallback to direct microservice calls, though without client-specific aggregation. Clients receive stale cached data from API Gateway without error is wrong as API Gateway typically does not cache full responses. Clients' requests fail immediately as BFF is the only entry point for client-specific aggregation is incorrect because requests do not fail immediately if fallback exists. API Gateway retries requests indefinitely until BFF recovers is wrong since indefinite retries cause client timeouts.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    API Gateway fallback to microservices bypasses BFF on failure [check]
Hint: API Gateway fallback bypasses BFF if it crashes [check]
Common Mistakes:
5. If the Kafka cluster responsible for connection event streaming goes down, what exactly breaks in the system's connection request processing?
medium
A. Connection requests are lost immediately with no retry or buffering
B. Connection requests are redirected to a backup database that duplicates all writes
C. Connection requests block the user synchronously until Kafka recovers, causing UI timeouts
D. Connection requests are accepted but downstream services do not receive updates, causing stale connection graphs

Solution

  1. Step 1: Understand Kafka's role in event streaming

    Kafka buffers and reliably delivers connection events to downstream services asynchronously.

  2. Step 2: Trace failure impact

    If Kafka is down, writes to Kafka fail but connection requests may still be accepted at the API layer; however, downstream consumers do not get updates, causing stale graph state.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Kafka down -> downstream stale, no immediate request loss [check]
Hint: Kafka failure stalls downstream, not immediate request loss [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard