Bird
Raised Fist0

Design an Event-Driven Architecture - Saga Pattern, Outbox Pattern, Idempotency

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design an Event-Driven Architecture - Saga Pattern, Outbox Pattern, Idempotency
45 min · senior
senior

You are designing a distributed microservices system for a 50 million daily active user (DAU) startup. The system must coordinate complex multi-step business transactions across services using the Saga pattern to maintain data consistency without distributed transactions. Messages between services must be reliably delivered even if recipients are offline, and all inter-service communication must be encrypted end-to-end. The system must guarantee idempotent processing of events to handle retries without side effects. The P99 end-to-end latency for transaction completion must be under 200ms.

Constraints
  • End-to-end encryption of all messages in transit and at rest
  • Support offline delivery with eventual consistency guarantees
  • P99 end-to-end latency under 200ms for transaction completion
  • Idempotent event processing to handle retries safely
  • Use open-source technologies only
📋
Guided Steps
1/5
Step 1
Choosing the Event Bus for Saga Coordination
Which messaging system will you use to implement the event-driven saga coordination and why?
💡 Hint
Consider a distributed log system that supports partitioning and ordering guarantees for events.
Key Decision
Use Apache Kafka with saga_id as the partition key to ensure ordered, durable event streams per saga instance.
Interviewer Expectation
Mid-level: Names Kafka as the event bus. Senior: Explains using saga_id as Kafka partition key to guarantee ordering and co-locate related events, avoiding out-of-order processing.
⚠️Common Wrong Answer
Using RabbitMQ without partitioning - leads to unordered event delivery and complicates saga step sequencing at scale.
Step 2
Implementing the Outbox Pattern for Reliable Event Publishing
How will you ensure atomicity between local database updates and event publishing to Kafka?
💡 Hint
Think about a pattern that writes events to a local durable store within the same transaction as business data updates.
Key Decision
Use the Outbox pattern with PostgreSQL’s transactional tables to atomically write business data and outgoing events, then a separate process reads and publishes events to Kafka.
Interviewer Expectation
Mid-level: Mentions Outbox pattern for atomic writes. Senior: Details using PostgreSQL transactional tables and a dedicated Outbox publisher service to guarantee exactly-once event publishing.
⚠️Common Wrong Answer
Directly publishing to Kafka inside the business transaction - risks losing events on transaction rollback or duplicating events on retries.
Step 3
Ensuring Idempotency in Event Processing
How will you design services to process events idempotently to handle retries and duplicates?
💡 Hint
Consider storing processed event IDs and using unique constraints or deduplication caches.
Key Decision
Implement idempotency by persisting processed event IDs in Redis with TTL and enforcing unique constraints in service databases to prevent duplicate side effects.
Interviewer Expectation
Mid-level: Mentions idempotency keys or deduplication. Senior: Explains combining Redis-based deduplication cache with database-level unique constraints to handle both in-memory and persistent idempotency.
⚠️Common Wrong Answer
Ignoring idempotency - leads to inconsistent state and side effects when events are retried or duplicated.
Step 4
Handling Offline Delivery and Eventual Consistency
How will your system guarantee delivery of events to services that may be temporarily offline?
💡 Hint
Think about durable event storage and replay mechanisms.
Key Decision
Leverage Kafka’s durable log retention and consumer offset tracking to allow offline services to replay missed events upon reconnection.
Interviewer Expectation
Mid-level: Mentions Kafka retention and consumer groups. Senior: Explains how consumer offset management enables exactly-once processing and offline replay without message loss.
⚠️Common Wrong Answer
Using ephemeral queues without persistence - causes message loss if consumers are offline.
Step 5
Securing Event Data with Encryption
How will you ensure end-to-end encryption of events both in transit and at rest?
💡 Hint
Consider encryption at the transport layer and encrypting event payloads before publishing.
Key Decision
Use TLS for Kafka transport encryption and encrypt event payloads at the application layer using AES-256 before writing to the Outbox and Kafka.
Interviewer Expectation
Mid-level: Mentions TLS for Kafka. Senior: Details application-layer payload encryption to protect data at rest and in transit, including key management strategies.
⚠️Common Wrong Answer
Relying solely on TLS - leaves event data unencrypted at rest in Kafka and Outbox tables.
🔌
API Design Challenge
Sketch 3 core API endpoints to support the event-driven saga workflow: initiating a saga, reporting saga step completion, and querying saga status.
Hints
  • How does the client initiate a multi-step transaction that spans services?
  • How do services report the completion or failure of individual saga steps?
  • How can clients query the current status and outcome of a saga?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use Kafka with saga_id partition key for ordered event streams
  • Implement Outbox pattern with PostgreSQL for atomic event and data writes
  • Design idempotent event processing using Redis and database constraints
  • Support offline delivery via Kafka consumer offset replay
  • Encrypt events end-to-end with TLS and application-layer AES-256
Good to Have - 30%
  • Explain saga orchestration as a durable state machine
  • Describe API endpoints for saga lifecycle management
  • Detail key management for encryption
  • Discuss scaling Kafka partitions and Outbox processing
Bonus - 10%
  • Propose monitoring and alerting for Outbox lag and event processing failures
  • Suggest fallback or compensation mechanisms for saga failures
Anti-patterns to Avoid
Directly publishing events inside business transactions without Outbox - causes event loss on rollback or duplicates on retries
Using RabbitMQ without partitioning for saga events - leads to unordered processing and inconsistent saga state
Ignoring idempotency - causes duplicate side effects and inconsistent data when events are retried or duplicated
Relying only on TLS without payload encryption - exposes sensitive data at rest in Kafka and Outbox tables
Using ephemeral queues without durable storage - causes message loss if consumers are offline
Polling-based event delivery - causes high load and latency spikes at scale, collapsing under 50M DAU load
✅
Model Answer
▼ Reveal

The system uses Apache Kafka with saga_id as the partition key to guarantee ordered, durable event streams per saga instance. Business data updates and outgoing events are atomically written using the Outbox pattern with PostgreSQL transactional tables. A dedicated Outbox publisher service reads events from the Outbox table and publishes them reliably to Kafka, ensuring exactly-once delivery. Services process events idempotently by storing processed event IDs in Redis with TTL and enforcing unique constraints in their databases to prevent duplicate side effects. Kafka consumer groups with offset tracking enable offline services to replay missed events, ensuring eventual consistency. End-to-end encryption is achieved by using TLS for Kafka transport and AES-256 application-layer encryption of event payloads before writing to the Outbox and Kafka. Saga orchestration is implemented as a durable state machine per saga instance stored in a persistent store. The system exposes API endpoints to initiate sagas, report step completion, and query saga status.

Key Decisions
  • Use Apache Kafka with saga_id as partition key to ensure ordered event streams per saga
  • Implement Outbox pattern using PostgreSQL transactional tables to atomically write business data and events
  • Deploy a dedicated Outbox publisher service for reliable event publishing to Kafka
  • Combine Redis-based deduplication cache with database unique constraints for idempotent event processing
  • Leverage Kafka consumer groups and offset tracking to support offline event replay and eventual consistency
  • Apply TLS for Kafka transport encryption and AES-256 for application-layer payload encryption to secure data in transit and at rest
  • Model saga orchestration as durable state machines stored persistently
  • Provide API endpoints to initiate sagas, report step completion, and query saga status
  • Apache Kafka with saga_id as partition key to guarantee ordered event streams per saga
  • PostgreSQL Outbox pattern to atomically write business data and outgoing events
  • Dedicated Outbox publisher service to read and publish events reliably to Kafka
  • Redis with TTL for in-memory idempotency cache combined with database unique constraints
  • Kafka consumer groups with offset tracking to enable offline event replay and eventual consistency
  • TLS for Kafka transport encryption plus AES-256 application-layer payload encryption for end-to-end security
  • Saga orchestration implemented as a state machine per saga instance stored in a durable store
  • API endpoints for initiating sagas, reporting step completion, and querying saga status

Practice

(1/5)
1. A global CDN operator needs to prevent a misconfiguration from propagating incorrect BGP routes that could cause widespread internet outages. Which component is primarily responsible for detecting and containing such route leaks?
easy
A. Using a centralized DNS firewall to block suspicious IP prefixes
B. Implementing BGP route filters and prefix limits on edge routers
C. Relying on client-side validation of IP addresses before routing
D. Deploying a distributed cache to store valid routing tables

Solution

  1. Step 1: Identify BGP route filtering role

    BGP route filters and prefix limits on edge routers prevent invalid or excessive route announcements, containing route leaks.

  2. Step 2: Eliminate other options

    DNS firewalls (A) do not control BGP routing; client-side validation (C) is ineffective for routing control; distributed caches (D) do not enforce routing policies.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    BGP route filters contain route leaks at the network edge [check]
Hint: Edge routers enforce BGP filters to block bad routes [check]
Common Mistakes:
2. You need to handle millions of photo uploads per minute with immediate thumbnail generation and metadata extraction without blocking the user upload flow. Which component is best suited to decouple upload ingestion from processing?
easy
A. Use a distributed message queue like Kafka to buffer uploads and process asynchronously
B. Use a relational database to store uploads synchronously and trigger processing via triggers
C. Use a CDN edge server to generate thumbnails immediately upon upload
D. Use a synchronous REST API that processes images inline before responding

Solution

  1. Step 1: Identify upload ingestion requirements

    Uploads must not block user flow and require asynchronous processing for thumbnails and metadata extraction.

  2. Step 2: Evaluate decoupling mechanisms

    Relational DB triggers (A) cause synchronous blocking and poor scalability. CDN edge (C) is for delivery, not upload processing. Synchronous REST (D) blocks user upload. Kafka (B) buffers uploads asynchronously, enabling scalable processing.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka buffers high-throughput uploads asynchronously [check]
Hint: Buffer uploads asynchronously with Kafka for scalable processing [OK]
Common Mistakes:
3. For delivering videos globally with low latency, should the system use a centralized origin server with global CDN caching, or a peer-to-peer (P2P) delivery network? Assume millions of users worldwide and variable network conditions.
medium
A. Use a centralized origin server with a global CDN to ensure consistent video availability and latency
B. Use P2P delivery combined with CDN fallback to optimize bandwidth and reliability
C. Use P2P delivery exclusively to reduce CDN costs and improve scalability
D. Use multiple regional origin servers without CDN to reduce latency

Solution

  1. Step 1: Evaluate delivery approaches for global scale and latency

    P2P (A) alone is unreliable due to variable user upload bandwidth and NAT traversal issues. Multiple regional origins without CDN (D) add complexity and don't scale well globally. P2P with CDN fallback (C) adds complexity and inconsistent performance.

  2. Step 2: Recognize CDN benefits

    A centralized origin with global CDN (B) provides consistent availability, low latency via edge caching, and simplifies management, making it the best fit for millions of users worldwide.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Global CDN ensures consistent low-latency delivery [check]
Hint: Global CDN with origin beats P2P for consistent delivery [check]
Common Mistakes:
4. At 10x scale, which component in a real-time chat system is most likely to fail first and why?
hard
A. Database because increased writes cause disk I/O bottlenecks
B. Load balancer because it cannot handle increased TCP connections
C. Chat servers because WebSocket connections are stateful and cannot be resharded easily
D. Message queue because partition count limits throughput

Solution

  1. Step 1: Identify stateful bottleneck at scale

    WebSocket connections are stateful and bound to specific chat servers, limiting horizontal scaling.

  2. Step 2: Compare other components

    Databases and queues can be sharded or scaled horizontally; load balancers can be scaled with proxies.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Stateful WebSocket servers limit scaling more than DB or queues [check]
Hint: Stateful WebSocket servers block scaling before DB or queues [check]
Common Mistakes:
5. At 10x current scale, which component in the YouTube view count system is most likely to fail first, and why?
hard
A. The CDN edge caches because they cannot handle the increased video delivery requests
B. The view count database because it is a centralized bottleneck for high write throughput
C. The transcoding pipeline because it cannot process 10x more video uploads in time
D. The upload service because it cannot handle 10x concurrent upload connections

Solution

  1. Step 1: Analyze component scalability at 10x load

    CDN edge caches (A) scale horizontally and are designed for massive read throughput. Transcoding pipeline (C) can be scaled with more workers. Upload service (D) can be scaled with load balancers and stateless servers.

  2. Step 2: Identify view count database bottleneck

    View count DB is a centralized write hotspot with high contention and latency sensitivity. At 10x scale, it becomes the first bottleneck due to write amplification and consistency requirements.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Centralized view count DB write bottleneck fails first [check]
Hint: Centralized DB write bottleneck breaks first at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard