Bird
Raised Fist0
Microservicessystem_design~25 mins

Outbox pattern for reliable events in Microservices - System Design Exercise

Choose your learning style10 modes available

Start learning this pattern below

Jump into concepts and practice - no test required

or
Recommended
Test this pattern10 questions across easy, medium, and hard to know if this pattern is strong
Design: Outbox Pattern for Reliable Event Delivery
Design focuses on the microservice implementing the outbox pattern and event delivery mechanism. Event consumers and their processing logic are out of scope.
Functional Requirements
FR1: Ensure events generated by a microservice are reliably delivered to other services.
FR2: Guarantee no events are lost even if the service crashes after database update but before event publishing.
FR3: Support eventual consistency between the service's database and event consumers.
FR4: Allow event consumers to process events asynchronously.
FR5: Handle high throughput of events with minimal latency.
FR6: Provide visibility into event delivery status for monitoring and debugging.
Non-Functional Requirements
NFR1: System must handle 10,000 events per second.
NFR2: Event delivery latency p99 should be under 500ms.
NFR3: System availability target is 99.9% uptime.
NFR4: Events must be delivered at least once (idempotency handled by consumers).
NFR5: Use existing relational database for service data storage.
Think Before You Design
Questions to Ask
❓ Question 1
❓ Question 2
❓ Question 3
❓ Question 4
❓ Question 5
❓ Question 6
Key Components
Service database with outbox table
Transactional write mechanism
Event publisher component
Message broker (e.g., Kafka, RabbitMQ)
Event consumer services
Monitoring and logging tools
Design Patterns
Transactional Outbox Pattern
Event Sourcing
Message Queue
Idempotent Consumer
Retry and Dead Letter Queue
Reference Architecture
 +----------------+       +----------------+       +------------------+
 |                |       |                |       |                  |
 |  Microservice  |       |  Message       |       |  Event Consumers |
 |  +----------+  |       |  Broker        |       |  (Other Services)|
 |  | Database |  |       |  (Kafka/Rabbit)|       |                  |
 |  | +------+ |  |       |                |       |                  |
 |  | |Outbox| |  |       |                |       |                  |
 |  | +------+ |  |       |                |       |                  |
 |  +----------+  |       +----------------+       +------------------+
 +-------|--------+               ^                        ^
         |                        |                        |
         | 1. Write data + event  |                        |
         |    in one DB txn      |                        |
         |---------------------->|                        |
         |                       2. Publish event         |
         |                       from outbox table        |
         |                        |----------------------->|
         |                        |                        |
         |                        |                        |
         |                        |                        |
         | 3. Mark event as sent  |                        |
         |<----------------------|                        |
         |                        |                        |
Components
Service Database
Relational DB (e.g., PostgreSQL)
Stores business data and an outbox table for events in the same transactional context.
Outbox Table
Relational DB table
Holds events generated by the service, pending publishing.
Transactional Write Mechanism
Database transactions
Ensures atomicity of business data changes and event insertion.
Event Publisher
Background worker or scheduler
Reads unsent events from outbox, publishes them to message broker, and marks them sent.
Message Broker
Kafka or RabbitMQ
Decouples event producers and consumers, ensures reliable event delivery.
Event Consumers
Other microservices
Consume and process events asynchronously.
Monitoring and Logging
Prometheus, Grafana, ELK stack
Track event publishing success, failures, and system health.
Request Flow
1. Client sends request to microservice to update data.
2. Microservice starts a database transaction.
3. Microservice updates business data and inserts corresponding event into outbox table within the same transaction.
4. Transaction commits, ensuring both data and event are saved atomically.
5. Event publisher component periodically polls the outbox table for unsent events.
6. Event publisher reads events, publishes them to the message broker.
7. After successful publish, event publisher marks events as sent in the outbox table.
8. Event consumers subscribe to the message broker and process events asynchronously.
9. Monitoring tools track event publishing metrics and alert on failures.
Database Schema
Entities: - BusinessData(id PK, data fields...) - OutboxEvent(id PK, aggregate_id FK, event_type, payload JSON, created_at, sent_at nullable) Relationships: - OutboxEvent.aggregate_id references BusinessData.id Notes: - OutboxEvent stores serialized event data. - sent_at is null until event is published.
Scaling Discussion
Bottlenecks
Outbox table grows large causing slow polling and database performance degradation.
Event publisher becomes a bottleneck under high event throughput.
Message broker saturation or slow consumers causing backpressure.
Database transaction contention due to frequent writes.
Solutions
Implement archiving or purging of sent events from outbox table periodically.
Scale event publisher horizontally with partitioned polling or sharding.
Use a high-throughput, distributed message broker like Kafka with partitioning.
Optimize database indexes and use batch inserts for outbox events.
Apply backpressure handling and consumer scaling to keep up with event load.
Interview Tips
Time: Spend 10 minutes understanding requirements and clarifying assumptions, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.
Explain the problem of atomicity between data changes and event publishing.
Describe how the outbox pattern solves this with a single database transaction.
Discuss the role of the event publisher and message broker.
Mention how eventual consistency is achieved and why at-least-once delivery is acceptable.
Highlight monitoring and retry mechanisms for reliability.
Address scaling challenges and solutions.

Practice

(1/5)
1. What is the main purpose of the Outbox pattern in microservices?
easy
A. To store user session data for microservices
B. To ensure events are stored and published reliably with data changes
C. To replace the need for message queues entirely
D. To speed up database queries by caching events

Solution

  1. Step 1: Understand the problem Outbox pattern solves

    The Outbox pattern ensures that events related to data changes are not lost and are reliably published.
  2. Step 2: Identify the main purpose

    It stores events in the same database transaction as the data change, so both succeed or fail together, ensuring consistency.
  3. Final Answer:

    To ensure events are stored and published reliably with data changes -> Option B
  4. Quick Check:

    Outbox pattern = reliable event storage and publishing [OK]
Hint: Outbox pattern links events with data changes atomically [OK]
Common Mistakes:
  • Thinking it speeds up queries
  • Believing it replaces message queues
  • Confusing it with session storage
2. Which of the following is the correct sequence in the Outbox pattern?
easy
A. Write event to outbox table, commit transaction, then publish event
B. Publish event, then write event to outbox table, then commit transaction
C. Commit transaction, then write event to outbox table, then publish event
D. Publish event and write to outbox table simultaneously outside transaction

Solution

  1. Step 1: Understand transaction order in Outbox pattern

    The event is first written to the outbox table inside the same transaction as the data change.
  2. Step 2: Commit transaction before publishing

    Only after the transaction commits successfully, a separate process reads and publishes the event.
  3. Final Answer:

    Write event to outbox table, commit transaction, then publish event -> Option A
  4. Quick Check:

    Outbox write before commit, publish after commit [OK]
Hint: Events must be saved before commit, published after [OK]
Common Mistakes:
  • Publishing before commit causes lost events
  • Writing outbox after commit breaks atomicity
  • Trying to publish and write outside transaction
3. Given this pseudocode for an Outbox pattern implementation, what will be the output if the transaction fails?
begin transaction
write data change
write event to outbox
commit transaction
publish event from outbox
medium
A. Neither data change nor event is saved or published
B. Event is published but data change is lost
C. Data change saved but event not published
D. Event published twice

Solution

  1. Step 1: Analyze transaction failure impact

    If the transaction fails, none of the writes (data change or outbox event) are committed to the database.
  2. Step 2: Understand event publishing dependency

    Since the event is published only after commit, no event will be published if commit fails.
  3. Final Answer:

    Neither data change nor event is saved or published -> Option A
  4. Quick Check:

    Failed transaction means no data or event saved [OK]
Hint: Failed transaction means no commit, no event published [OK]
Common Mistakes:
  • Assuming event publishes despite transaction failure
  • Thinking data change saves without commit
  • Believing event publishes twice
4. A developer notices some events are missing in the message queue after using the Outbox pattern. What is the most likely cause?
medium
A. Events are duplicated in the outbox table
B. Events are published before the transaction commits
C. The database transaction is too fast
D. The outbox table is not being read and events published after commit

Solution

  1. Step 1: Identify missing events cause

    If events are missing in the queue, it usually means the process that reads the outbox and publishes events is not running or failing.
  2. Step 2: Rule out other causes

    Publishing before commit risks lost events, but missing events usually mean no publishing process or failure to read outbox.
  3. Final Answer:

    The outbox table is not being read and events published after commit -> Option D
  4. Quick Check:

    Missing events = outbox not read or published [OK]
Hint: Missing events? Check outbox reader process [OK]
Common Mistakes:
  • Assuming transaction speed causes missing events
  • Thinking events publish before commit is safe
  • Confusing missing events with duplicates
5. You want to design a microservice using the Outbox pattern to handle user registrations and notify other services. Which approach best ensures no events are lost and services stay consistent?
hard
A. Publish event first, then write user data; rollback event if data write fails
B. Write user data first, then publish event immediately without transaction; retry on failure
C. Write user data and event to outbox in one transaction; use a separate reliable process to publish events asynchronously
D. Write user data and publish event in the same transaction synchronously

Solution

  1. Step 1: Ensure atomicity of data and event writes

    Writing user data and event to the outbox table in the same transaction guarantees both succeed or fail together.
  2. Step 2: Use separate process for event publishing

    Publishing events asynchronously from the outbox ensures reliable delivery without blocking the main transaction.
  3. Final Answer:

    Write user data and event to outbox in one transaction; use a separate reliable process to publish events asynchronously -> Option C
  4. Quick Check:

    Atomic write + async publish = reliable and consistent [OK]
Hint: Atomic write + async publish ensures no lost events [OK]
Common Mistakes:
  • Publishing events outside transaction without retry
  • Publishing before data write risks inconsistency
  • Trying synchronous publish inside transaction