Bird
Raised Fist0

Webhook Delivery System - Reliability, Retry, Ordering Guarantees

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design a Webhook Delivery System - Reliability, Retry, Ordering Guarantees
45 min · senior
senior

You are building a webhook delivery system for a startup with 50 million daily active users. The system must deliver webhooks reliably to third-party endpoints with strict ordering guarantees per webhook subscription. It must handle offline endpoints gracefully with retries and exponential backoff. All webhook payloads must be encrypted in transit and at rest. The system should maintain a P99 latency under 200ms for successful deliveries. The system must scale to handle millions of webhook events per second.

Constraints
  • All webhook payloads must be encrypted at rest and in transit using TLS and AES-256.
  • Strict ordering guarantees per webhook subscription: events must be delivered in the order they were generated.
  • Offline endpoint handling with retry and exponential backoff up to 24 hours.
  • P99 delivery latency under 200ms for online endpoints.
  • Support at least 10 million webhook events per second at peak.
📋
Guided Steps
1/5
Step 1
Event Ingestion and Ordering
How would you design the ingestion pipeline to ensure strict ordering per webhook subscription at scale?
💡 Hint
Consider partitioning events by subscription ID to preserve order and use a scalable distributed log.
Key Decision
Use Apache Kafka with webhook_subscription_id as the partition key - Kafka partitions guarantee ordered delivery within a partition and scale linearly with partitions.
Interviewer Expectation
Mid-level: Names Kafka for event ingestion. Senior: Explains why webhook_subscription_id as partition key ensures ordering and why using user_id or random keys breaks ordering.
⚠️Common Wrong Answer
Using a relational database with auto-increment IDs for ordering - this causes write bottlenecks and does not scale to millions of events per second.
Step 2
Reliable Delivery and Retry Mechanism
How would you implement reliable delivery with retries and exponential backoff for offline endpoints?
💡 Hint
Think about a durable queue with delayed retry capabilities and a mechanism to track delivery attempts.
Key Decision
Use Apache Kafka for initial event storage and Redis Streams with TTL-based delayed queues for retry scheduling, combined with a Delivery Worker that tracks attempts and applies exponential backoff.
Interviewer Expectation
Mid-level: Mentions retry queues and backoff. Senior: Details how Redis Streams TTL and Kafka offset management coordinate retries and prevent duplicate deliveries.
⚠️Common Wrong Answer
Implementing retries with simple in-memory timers - this fails at scale due to lack of durability and coordination across distributed workers.
Step 3
Encryption and Security
How do you ensure webhook payloads are encrypted both in transit and at rest?
💡 Hint
Consider TLS for transport and a secure encryption scheme for stored payloads with key management.
Key Decision
Use TLS 1.3 for all HTTP webhook deliveries and encrypt payloads at rest using AES-256 with keys managed by AWS KMS or HashiCorp Vault integrated into the storage layer.
Interviewer Expectation
Mid-level: Mentions TLS and encryption at rest. Senior: Explains key rotation, envelope encryption, and how to integrate KMS with Kafka Connect or storage systems.
⚠️Common Wrong Answer
Relying only on TLS without encrypting payloads at rest - this exposes sensitive data if storage is compromised.
Step 4
Delivery Status Tracking and Acknowledgments
How would you design the system to track delivery status and handle acknowledgments from webhook endpoints?
💡 Hint
Think about idempotent acknowledgments and a scalable storage for delivery metadata.
Key Decision
Use Cassandra with partition key = webhook_subscription_id and clustering key = event_sequence_number to store delivery status, enabling efficient range queries and idempotent updates.
Interviewer Expectation
Mid-level: Suggests storing status in a database. Senior: Explains Cassandra’s LSM-tree write scalability and how partitioning by subscription_id supports ordered status updates and fast lookups.
⚠️Common Wrong Answer
Using a relational database with global locks for status updates - this causes contention and latency spikes at scale.
Step 5
API Design for Webhook Management
Sketch the 3 core API endpoints for clients to register webhooks, ingest events, and acknowledge delivery status.
💡 Hint
Consider RESTful endpoints with authentication and payload validation.
Key Decision
POST /webhooks/register to create/update webhook subscriptions with OAuth2 authentication; POST /events to ingest events with webhook_subscription_id; POST /delivery/ack to receive delivery acknowledgments with event_id and status.
Interviewer Expectation
Mid-level: Names basic REST endpoints. Senior: Details authentication, idempotency keys, payload schemas, and error handling strategies.
⚠️Common Wrong Answer
Using unauthenticated endpoints or polling-based status checks - this risks security and increases load unnecessarily.
🔌
API Design Challenge
Sketch 3 core API endpoints for the webhook delivery system: how clients register webhooks, how events are ingested, and how delivery status is acknowledged.
Hints
  • How does a client securely register or update a webhook subscription?
  • How does the system accept and enqueue webhook events for delivery?
  • How does the system receive delivery acknowledgments or failures from webhook endpoints?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use Kafka with webhook_subscription_id partition key to guarantee ordering and scale.
  • Implement retry with durable delayed queues (Redis Streams) and exponential backoff.
  • Encrypt payloads in transit with TLS and at rest with AES-256 and KMS-managed keys.
  • Track delivery status in Cassandra partitioned by subscription for efficient updates.
  • Design secure, idempotent REST APIs for webhook lifecycle management.
Good to Have - 30%
  • Explain key rotation and envelope encryption integration with KMS.
  • Describe how Delivery Workers coordinate Kafka offset commits and retries.
  • Discuss monitoring and alerting for delivery failures and SLA breaches.
Bonus - 10%
  • Propose optimizations for P99 latency under 200ms using connection pooling and HTTP/2.
  • Suggest multi-region deployment for geo-redundancy and disaster recovery.
Anti-patterns to Avoid
Using relational databases for event ingestion causes write bottlenecks and loss of ordering at scale, leading to system failures under high load.
In-memory retry timers without durability lead to lost retries and inconsistent delivery state when workers restart or fail.
Relying only on TLS without encrypting payloads at rest exposes sensitive data if storage is compromised.
Using relational databases with global locks for delivery status updates causes contention and latency spikes, reducing system throughput.
HTTP polling at 50M DAU results in 50 million reconnects per minute just to check for messages, collapsing the system under load.
✅
Model Answer
▼ Reveal

The webhook delivery system uses Apache Kafka partitioned by webhook_subscription_id to ensure strict ordering and scalability. Events are ingested into Kafka, then Delivery Workers consume events per partition to deliver webhooks reliably. Retries for offline endpoints use Redis Streams with TTL-based delayed queues implementing exponential backoff up to 24 hours. Payloads are encrypted in transit with TLS 1.3 and at rest using AES-256 with keys managed by AWS KMS. Delivery status is tracked in Cassandra partitioned by webhook_subscription_id and clustered by event_sequence_number for efficient, ordered updates. The system exposes OAuth2-secured REST APIs for webhook registration, event ingestion, and delivery acknowledgment, supporting idempotency and error handling.

Key Decisions
  • Use Kafka with webhook_subscription_id as partition key to guarantee ordering and scale linearly with partitions.
  • Implement retries using Redis Streams delayed queues with TTL and exponential backoff to handle offline endpoints reliably.
  • Encrypt all webhook payloads in transit using TLS 1.3 and at rest using AES-256 with keys managed by AWS KMS or HashiCorp Vault.
  • Track delivery status in Cassandra with partition key webhook_subscription_id and clustering key event_sequence_number for efficient range queries and idempotent updates.
  • Design REST APIs secured with OAuth2, supporting idempotency keys and payload validation to ensure secure and reliable webhook management.
  • Apache Kafka with webhook_subscription_id as partition key for ordered, scalable event ingestion.
  • Redis Streams with TTL-based delayed queues for implementing reliable retries with exponential backoff.
  • TLS 1.3 for secure transport and AES-256 encryption at rest with AWS KMS for key management.
  • Cassandra with partition key webhook_subscription_id and clustering key event_sequence_number for scalable delivery status tracking.
  • Delivery Workers consuming Kafka partitions to ensure ordered delivery and retry logic per subscription.
  • OAuth2-secured REST APIs for webhook registration, event ingestion, and delivery acknowledgment.
  • Idempotent delivery acknowledgment handling to prevent duplicate processing and maintain consistency.
  • Exponential backoff retry strategy with maximum retry window of 24 hours to handle offline endpoints.

Practice

(1/5)
1. In the request flow, at Step 4 the Delivery Worker checks the message queue's visibility timeout before processing. What happens if the visibility timeout expires before the message is acknowledged?
easy
A. The message is moved to a dead-letter queue without retry
B. The message is deleted immediately to prevent duplicates
C. The message is archived permanently without redelivery
D. The message becomes visible again and can be redelivered to another worker

Solution

  1. Step 1: Understand visibility timeout semantics

    Visibility timeout hides the message during processing; if not acknowledged, it reappears for redelivery.

  2. Step 2: Analyze options

    Immediate deletion (A) risks data loss; archiving without retry (C) prevents recovery; dead-letter queue (D) is for failed retries, not timeout expiry.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Visibility timeout expiry -> message reappears for retry [check]
Hint: Visibility timeout expiry -> message requeued for retry [check]
Common Mistakes:
2. In the driver matching flow, at step 4 the Matching Service queries the Surge Pricing Service for the current multiplier. What happens if the Surge Pricing Service responds with a stale multiplier due to cache lag?
easy
A. The Matching Service rejects the request and retries immediately
B. The Matching Service blocks until fresh surge data is available
C. The Matching Service falls back to a default multiplier of 1.0
D. The Matching Service uses the stale multiplier, causing temporary incorrect pricing

Solution

  1. Step 1: Trace Matching Service behavior at step 4

    Matching Service queries Surge Pricing Service for multiplier to calculate fare.

  2. Step 2: Understand stale data impact

    If surge multiplier is stale due to cache lag, Matching Service proceeds with returned value, causing temporary fare mispricing but no blocking or retries.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Matching Service tolerates stale surge multiplier for availability [check]
Hint: Matching tolerates stale surge data to avoid blocking [check]
Common Mistakes:
3. In a message queue system, at Step 4 the Delivery Worker reads a message from Kafka and checks the consumer's offset commit status. What happens if the offset commit is missing for that consumer group?
easy
A. The Delivery Worker skips the message and moves to the next offset
B. The Delivery Worker delivers the message and waits for the consumer to commit offset later
C. The Delivery Worker retries reading the message until offset commit appears
D. The Delivery Worker deletes the message from the partition to avoid duplicates

Solution

  1. Step 1: Understand offset commit role

    Offset commit tracks consumer progress; missing commit means consumer hasn't acknowledged message yet.

  2. Step 2: Delivery Worker behavior

    Delivery Worker must deliver the message and rely on consumer to commit offset asynchronously; skipping (A) or deleting (D) would cause message loss; retrying (C) is unnecessary as delivery is decoupled.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Delivery Worker delivers message, offset commit is consumer's responsibility [check]
Hint: Delivery Worker delivers before offset commit [check]
Common Mistakes:
4. For ensuring strong consistency in a distributed digital wallet system, which architecture is better suited given the need to support high throughput and fault tolerance?
medium
A. Use a distributed event sourcing architecture with Kafka as the event log and materialized views for queries
B. Use a NoSQL database with eventual consistency and background reconciliation jobs
C. Use a single centralized relational database with ACID transactions for all wallet operations
D. Use a microservices architecture with synchronous REST calls between services for each transaction

Solution

  1. Step 1: Analyze consistency and throughput requirements

    Strong consistency with high throughput and fault tolerance requires decoupling writes and reads with durable event logs.

  2. Step 2: Evaluate architectures

    Centralized DB (A) limits throughput and is a single point of failure; NoSQL eventual consistency (C) violates strong consistency; synchronous REST calls (D) increase latency and reduce fault tolerance. Event sourcing with Kafka (B) provides durable, ordered events and scalable materialized views.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Event sourcing with Kafka -> strong consistency and scalability [check]
Hint: Event sourcing with Kafka balances consistency and scale [OK]
Common Mistakes:
5. At 10x traffic scale, which component in a Layer 7 load balancing architecture is most likely to become the bottleneck first, and why?
hard
A. Layer 7 load balancer CPU, due to expensive HTTP header parsing and SSL termination
B. Network switches, because increased packet forwarding saturates bandwidth
C. Backend servers, because they cannot handle increased HTTP request volume
D. DNS servers, because client-side resolution delays increase exponentially

Solution

  1. Step 1: Identify Layer 7 load balancer workload at scale

    Layer 7 load balancers perform CPU-intensive tasks like HTTP header parsing and SSL termination, which scale poorly with traffic.

  2. Step 2: Compare with other components

    Backend servers can be scaled horizontally; network switches handle packet forwarding efficiently; DNS servers are less impacted by traffic volume.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Layer 7 LB CPU bottleneck due to HTTP parsing and SSL termination [check]
Hint: Layer 7 LB CPU chokes first at high traffic scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard