Bird
Raised Fist0

Design an Event-Driven Architecture - Saga Pattern, Outbox Pattern, Idempotency

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/5
API Gateway

Receives client HTTP request

Forwards request to Microservice (Business Logic)

Protocol:HTTP
BeforeClient request is sent to API Gateway
AfterAPI Gateway forwards request to Microservice
Microservice (Business Logic)

Processes request and writes event to Outbox Table

Inserts event record in Outbox Table within same DB transaction

Protocol:JDBC Insert
BeforeNo event exists in Outbox Table for this request
AfterEvent record durably stored in Outbox Table
Outbox Poller (Debezium)

Reads CDC stream from Outbox Table

Detects new event and publishes it to Kafka Cluster

Protocol:CDC Stream / Kafka Produce
BeforeEvent exists in Outbox Table but not yet published
AfterEvent published to Kafka topic
Saga Workers

Consumes event from Kafka and coordinates saga

Executes saga step, calls Downstream Microservices with idempotency checks

Protocol:Kafka Consume / REST/gRPC
BeforeEvent is available in Kafka topic
AfterSaga step executed; downstream services invoked
Downstream Microservices

Executes local transaction and replies status

Returns success or failure to Saga Workers

Protocol:REST/gRPC Response
BeforeSaga Workers sent request to downstream service
AfterDownstream service completed local step and replied
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────┐
│ API Gateway   │
└───────┬───────┘
        │ HTTP Request
        ▼
┌───────────────────────────────┐
│ Microservice (Biz Logic)       │
│ - Handles request              │
│ - Writes to Outbox Table       │
└───────┬───────────────────────┘
        │ JDBC Insert (Outbox Tbl)
        ▼
┌───────────────────────┐
│ Outbox Table          │
│ (PostgreSQL)          │
└───────┬───────────────┘
        │ CDC Stream (Debezium)
        ▼
┌────────────────────────┐
│ Outbox Poller          │
│ (Debezium)             │
│ - Reads CDC events     │
│ - Publishes to Kafka   │
└───────┬────────────────┘
        │ Kafka Produce
        ▼
┌───────────────┐
│ Kafka Cluster │
└───────┬───────┘
        │ Kafka Consume
        ▼
┌───────────────┐
│ Saga Workers  │
│ - Consume events
│ - Coordinate saga
│ - Ensure idempotency
└───────┬───────┘
        │ REST/gRPC Calls
        ▼
┌───────────────────────┐
│ Downstream Microservices│
│ - Execute local steps  │
│ - Reply with status    │
└───────────────────────┘
Design Rationale

This architecture uses the Outbox Pattern to ensure reliable event publishing by writing events to a durable Outbox Table within the business logic's database transaction. Debezium acts as an Outbox Poller, reading change data capture streams and publishing events to Kafka, guaranteeing exactly-once delivery semantics. Saga Workers consume these events from Kafka to coordinate distributed transactions across downstream microservices, enforcing idempotency to handle retries safely. This design decouples business logic from event publishing, prevents lost or duplicated events, and enables robust distributed saga orchestration.

Primary Focus

✓

Practice

(1/5)
1. For a system requiring both low latency and strong consistency in user profile updates, which architecture choice best balances these constraints?
medium
A. Use eventual consistency with asynchronous replication to maximize availability
B. Use a single primary database with synchronous replication for strong consistency
C. Use a distributed cache with write-behind to the database for low latency
D. Use client-side caching to reduce read latency without server coordination

Solution

  1. Step 1: Identify consistency and latency requirements

    Strong consistency requires synchronous replication; low latency demands minimal delay.

  2. Step 2: Evaluate architecture options

    Single primary DB with sync replication ensures consistency but may increase latency slightly; best trade-off.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Synchronous replication balances strong consistency and latency [check]
Hint: Strong consistency needs sync replication, not eventual or cache-only [check]
Common Mistakes:
2. At 10x scale, which component in a Write-Through caching system is most likely to become the bottleneck first, and why?
hard
A. The cache, because synchronous writes increase latency and reduce throughput
B. The application servers, because they block waiting for cache writes to complete
C. The database, because it must handle every write synchronously from both cache and application
D. The network, because cache-to-database communication saturates bandwidth

Solution

  1. Step 1: Identify Write-Through bottleneck

    Write-Through synchronously writes to DB on every write, increasing DB load linearly with scale.

  2. Step 2: Compare other components

    Cache handles fast reads; network bandwidth usually sufficient; app servers block but DB saturation causes overall failure first.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    DB synchronous writes bottleneck Write-Through at scale [check]
Hint: Write-Through sync DB writes bottleneck at scale [check]
Common Mistakes:
3. At 10x scale, which component in the Q&A feed system is most likely to fail first and why?
hard
A. The search index cluster because it cannot shard queries across more nodes
B. The user profile database because it uses a single master node for writes
C. The feed generation service because it maintains stateful user session caches that cannot be sharded easily
D. The Kafka cluster because partition count cannot increase beyond a fixed limit

Solution

  1. Step 1: Analyze component scalability

    Search clusters (A) scale horizontally with sharding; Kafka (C) can increase partitions; user DB (D) can be sharded or use multi-master.

  2. Step 2: Stateful feed generation bottleneck

    Feed generation maintains per-user session caches and WebSocket connections, which are stateful and tied to specific servers, limiting horizontal scaling and causing early failure.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Stateful feed caches block horizontal scaling, causing first failure [check]
Hint: Stateful caches block scaling before stateless components [check]
Common Mistakes:
4. At 10x scale, which component in an idempotent payment API system is most likely to become the bottleneck first, and why?
hard
A. The payment processor, because it cannot handle increased transaction volume
B. The client SDK, as it struggles to generate unique idempotency keys at scale
C. The API gateway, because it cannot route 10x more requests efficiently
D. The idempotency key store, due to synchronous writes and read-after-write consistency requirements

Solution

  1. Step 1: Identify idempotency store workload at scale

    Idempotency store must synchronously write and read keys with strong consistency, creating a high throughput bottleneck.

  2. Step 2: Compare other components

    Payment processors can be scaled horizontally; API gateways are stateless and scalable; client SDKs generating keys is lightweight.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Synchronous idempotency store writes limit scaling first [check]
Hint: Idempotency store sync writes throttle scaling first [check]
Common Mistakes:
5. At 10x scale, which component in an async messaging chat system fails first and why?
hard
A. The database, because increased writes cause lock contention
B. The message queue brokers, due to partition leader overload
C. The chat servers, because WebSocket connections are stateful and cannot be resharded easily
D. The delivery workers, because retry queues grow exponentially

Solution

  1. Step 1: Analyze stateful components and scaling challenges

    Databases (The database, because increased writes cause lock contention) can be scaled with sharding and replicas. Message queue brokers (The message queue brokers, due to partition leader overload) scale horizontally with partitions. Delivery workers (The delivery workers, because retry queues grow exponentially) can be scaled with backpressure.

  2. Step 2: Identify chat servers as stateful WebSocket connection holders

    Chat servers maintain persistent WebSocket connections bound to server memory, making resharding or migration difficult at scale.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Stateful WebSocket servers -> first bottleneck at 10x scale [check]
Hint: Stateful WebSocket servers limit horizontal scaling first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard