Bird
Raised Fist0

Design an Event-Driven Architecture - Saga Pattern, Outbox Pattern, Idempotency

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design an Event-Driven Architecture - Saga Pattern, Outbox Pattern, Idempotency
Design a scalable, reliable event-driven system using Saga, Outbox, and Idempotency patterns for distributed transactions
hardModern Infrastructure DesignFreq: medium
Prerequisites:System Design FundamentalsCAP TheoremDistributed TransactionsEvent-Driven Architecture
⚙️
Functional Requirements
  • Support distributed transactions across multiple microservices using Saga pattern
  • Guarantee eventual consistency with reliable event delivery via Outbox pattern
  • Ensure idempotent processing of events to handle retries and duplicates
  • Provide monitoring and alerting for failed or stuck saga transactions
  • Support compensating transactions for rollback scenarios
  • Allow asynchronous communication between services with guaranteed ordering per saga
📊
Non-Functional Requirements
  • Handle 2 billion daily active users generating 40 events per user per day
  • Achieve 99.99% availability for event processing pipelines
  • Maintain end-to-end event processing latency under 100ms P99
  • Support 7-day event retention for replay and audit purposes
  • Scale to peak write throughput of approximately 925K events per second
❓
Clarifying Questions to Ask
  • What is the maximum acceptable latency for saga completion?
  • Are all services owned by the same team or multiple independent teams?
  • What is the failure tolerance for partial saga failures?
  • Is strict ordering required across all events or only per saga instance?
  • What is the expected size and schema of each event message?
  • Are there any regulatory or compliance requirements for data retention?
Out of Scope
✗ Design of individual microservices business logic
✗ User authentication and authorization mechanisms
✗ UI/UX design for monitoring dashboards
✗ Data analytics or reporting on event data
📌
Assumptions
  • 2 billion daily active users (DAU)
  • Each user generates 40 events per day on average
  • Average event size is 100 bytes
  • 7 days event retention for replay and audit
⚡
QPS Calculation
write qps2,000,000,000 × 40 / 86,400 ≈ 925,926 events/sec
peak write qpsAssuming 2× peak factor: 925,926 × 2 ≈ 1,851,852 events/sec
read qpsAssuming 3× read amplification for event consumers: 925,926 × 3 ≈ 2,777,778 events/sec
💾
Storage Estimate
daily storage gb(2B × 40 × 100 bytes) / (1024³) ≈ 7.45 TB/day
retention storage gb7.45 TB × 7 days ≈ 52.15 TB
with replication gb52.15 TB × 3 replicas ≈ 156.45 TB
🖥️
Server Estimate
kafka brokersTo handle 1.85M peak writes/sec and 2.77M reads/sec, ~50 Kafka brokers (each 40K writes/sec capacity)
processing workersAssuming each worker handles 5K events/sec, need ~370 workers for event processing
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |  Client / UI   |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |  API Gateway   |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |  Microservice  |
          |  (Business)    |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |  Outbox Table  |
          |  (PostgreSQL)  |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |  Outbox Poller |
          |  (Debezium)    |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |  Kafka Cluster |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |  Saga Workers  |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |  Downstream    |
          |  Microservices |
          +----------------+
🧩
Components
API Gateway
Entry point for client requests, routes to microservices
Why thisNginx or Envoy provides high throughput, low latency routing with TLS termination
Why notDirect client-to-service communication lacks centralized control and observability
Microservice (Business Logic)
Handles business operations, writes to local DB and Outbox table atomically
Why thisSpring Boot with PostgreSQL supports transactional outbox pattern with ACID guarantees
Why notNoSQL DBs like MongoDB lack multi-document ACID transactions needed for atomic outbox writes
Outbox Table (PostgreSQL)
Stores events atomically with business data changes for reliable event publishing
Why thisPostgreSQL supports transactional writes and logical decoding for change data capture
Why notKafka transactions alone cannot guarantee atomicity with business DB updates
Outbox Poller (Debezium)
Reads Outbox table changes via CDC and publishes events to Kafka
Why thisDebezium provides reliable, low-latency CDC with exactly-once semantics integration with Kafka
Why notCustom polling risks missing events or duplicates, lacks scalability
Kafka Cluster
Distributed event streaming platform for decoupled communication and event persistence
Why thisKafka scales to millions of events/sec with partitioning and replication for durability
Why notRabbitMQ lacks horizontal scalability and partitioning at this scale
Saga Workers
Consume events from Kafka, execute local transactions, and emit compensating events if needed
Why thisKubernetes-based stateless workers with Kafka consumer groups enable scalable parallel processing
Why notMonolithic services cannot horizontally scale or isolate failures effectively
Downstream Microservices
React to saga events to update their own state or trigger further actions
Why thisIndependent services enable bounded contexts and fault isolation
Why notTightly coupled services increase blast radius and reduce deployment agility
🔄
Data Flow
1
Client / UI → API Gateway
Send user request to initiate saga
Client triggers a distributed transaction
2
API Gateway → Microservice (Business Logic)
Forward request for processing
Microservice handles local transaction and writes to Outbox
3
Microservice → Outbox Table
Atomically write business data and event to Outbox
Ensures event is not lost if transaction commits
4
Outbox Poller (Debezium) → Kafka Cluster
Capture Outbox changes and publish events
Decouples event publishing from business logic
5
Kafka Cluster → Saga Workers
Deliver events to saga processors
Workers process events idempotently and trigger next saga steps
6
Saga Workers → Downstream Microservices
Emit events for downstream services to update state
Completes saga step or triggers compensating transactions
7
Downstream Microservices → Kafka Cluster
Publish events for further saga steps or completion
Enables chained saga orchestration

Practice

(1/5)
1. For a system requiring strict consistency on command handling but eventual consistency on query side, which architecture best fits?
medium
A. Use CQRS with event sourcing and asynchronous read model updates
B. Use a single monolithic database for both commands and queries
C. Use eventual consistency everywhere with no command validation
D. Use synchronous replication between command and query databases

Solution

  1. Step 1: Understand consistency requirements

    Strict consistency on commands requires immediate validation; eventual consistency on queries allows asynchronous updates.

  2. Step 2: Evaluate architectures

    CQRS with event sourcing separates command and query models, enabling strict command consistency and eventual query consistency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    CQRS + event sourcing -> strict command, eventual query consistency [check]
Hint: CQRS splits command and query for tailored consistency [check]
Common Mistakes:
2. If the client fails to validate the server's certificate chain during the TLS handshake, what is the most likely failure observed in the connection?
medium
A. The client proceeds with encrypted communication but logs a warning
B. The client aborts the handshake and the connection is terminated without data exchange
C. The server retries sending the certificate chain automatically
D. The client falls back to unencrypted HTTP automatically

Solution

  1. Step 1: Understand client certificate validation failure

    If the client detects an invalid certificate chain, it aborts the handshake and terminates the connection to prevent insecure communication.

  2. Step 2: Analyze options

    The client proceeds with encrypted communication but logs a warning is incorrect because clients do not proceed with encryption if validation fails. The server retries sending the certificate chain automatically is incorrect; servers do not retry sending certificates automatically. The client falls back to unencrypted HTTP automatically is incorrect because clients do not downgrade to HTTP automatically.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Client aborts handshake on certificate validation failure [check]
Hint: Certificate validation failure -> handshake abort, no fallback [check]
Common Mistakes:
3. At 10x scale, which component in a real-time chat system fails first and why?
hard
A. The database fails first due to increased query volume and locking
B. The message queue fails first due to partition key hotspots causing uneven load
C. The load balancer fails first due to inability to distribute traffic evenly
D. The chat servers fail first because WebSocket connections are stateful and cannot be resharded easily

Solution

  1. Step 1: Identify stateful bottlenecks at scale

    Chat servers maintain WebSocket connections bound to server memory, making horizontal scaling and resharding difficult.

  2. Step 2: Compare other components

    Databases (The database fails first due to increased query volume and locking) can be scaled with sharding and replicas. Load balancers (The load balancer fails first due to inability to distribute traffic evenly) scale horizontally. Message queues (The message queue fails first due to partition key hotspots causing uneven load) can rebalance partitions.

  3. Step 3: Conclude first failure point

    Chat servers hit connection limits first due to stateful nature and sticky sessions.

  4. Final Answer:

    Option D -> Option D

  5. Quick Check:

    Stateful WebSocket servers -> scaling bottleneck [check]
Hint: Stateful WebSocket servers block scaling; stateless scale easier [check]
Common Mistakes:
4. At 10x traffic scale, which component in HTTPS connections is most likely to become the bottleneck first, and why?
hard
A. The DNS resolver, because it cannot cache enough domain lookups
B. The TLS handshake CPU overhead on the server, due to expensive asymmetric cryptography per new connection
C. The web application server's database, due to increased query volume
D. The client browsers, because they cannot handle multiple simultaneous TLS sessions

Solution

  1. Step 1: Identify TLS handshake cost at scale

    Each new HTTPS connection requires a TLS handshake involving CPU-intensive asymmetric cryptography, which becomes a bottleneck as connection volume grows.

  2. Step 2: Eliminate other options

    The DNS resolver, because it cannot cache enough domain lookups is less likely as DNS caching mitigates lookup load. The web application server's database, due to increased query volume is unrelated to TLS overhead. The client browsers, because they cannot handle multiple simultaneous TLS sessions is client-side and not the first bottleneck in server scaling.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    TLS handshake CPU cost limits server scaling first [check]
Hint: TLS handshake CPU cost -> first bottleneck at scale [check]
Common Mistakes:
5. At 10x scale, which component in a relational database system is most likely to fail first and why?
hard
A. The query optimizer, because it cannot handle complex plans at scale
B. The storage engine, because disk I/O bandwidth saturates first
C. The transaction coordinator, due to increased contention and locking overhead
D. The connection pool, because client connections exceed configured limits

Solution

  1. Step 1: Identify scaling bottlenecks

    At 10x scale, contention on locks and transaction coordination overhead increase dramatically.

  2. Step 2: Why transaction coordinator fails first

    Locking and coordination cause blocking and deadlocks, limiting throughput before storage or connection limits.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Transaction coordinator bottlenecks due to lock contention at scale [check]
Hint: Lock contention chokes transaction coordinator first at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard