Saga pattern for distributed transactions in Microservices - System Design Exercise

Design: Distributed Transaction Management using Saga Pattern

Design focuses on the saga pattern implementation for distributed transactions across microservices. It excludes detailed service business logic and UI design.

Functional Requirements

FR1: Support transactions that span multiple microservices

FR2: Ensure data consistency across services without using distributed locks

FR3: Handle failures by compensating transactions to rollback partial changes

FR4: Support both choreography and orchestration styles of saga

FR5: Provide visibility into transaction status for monitoring and debugging

Non-Functional Requirements

NFR1: Must handle up to 10,000 concurrent distributed transactions

NFR2: End-to-end transaction latency should be under 5 seconds p99

NFR3: System availability target is 99.9% uptime

NFR4: Services are loosely coupled and communicate asynchronously

NFR5: No single point of failure in transaction coordination

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Saga Orchestrator or Event Bus for choreography

Microservices with local transaction and compensating actions

Message broker for asynchronous communication

Saga state store (database or distributed cache)

Monitoring and logging system

Design Patterns

Saga pattern (choreography and orchestration)

Event-driven architecture

Compensating transactions

Idempotency and retry mechanisms

State machine for saga status tracking

Reference Architecture

 +----------------+       +----------------+       +----------------+
 |  Service A     |       |  Service B     |       |  Service C     |
 | (Local Txn +   |       | (Local Txn +   |       | (Local Txn +   |
 |  Compensate)   |       |  Compensate)   |       |  Compensate)   |
 +-------+--------+       +-------+--------+       +-------+--------+
         |                        |                        |
         |                        |                        |
         |                        |                        |
         |                        |                        |
         |                        |                        |
         |                        |                        |
 +-------v------------------------v------------------------v-------+
 |                      Message Broker / Event Bus                  |
 +------------------------------------------------------------------+
         |                        |                        |
         |                        |                        |
 +-------v----------------------------------------------------------v-------+
 |                         Saga Orchestrator / Coordinator                 |
 |  - Tracks saga state                                                    |
 |  - Sends commands to services                                          |
 |  - Handles compensations on failure                                    |
 +------------------------------------------------------------------------+

Legend:
- Services perform local transactions and define compensating actions.
- Message Broker enables asynchronous communication.
- Saga Orchestrator manages transaction flow and state.

Components

Microservices

Any language/framework supporting microservices

Perform local transactions and define compensating actions for rollback

Message Broker / Event Bus

Kafka, RabbitMQ, or AWS SNS/SQS

Enable asynchronous communication between services and orchestrator

Saga Orchestrator / Coordinator

Custom service or workflow engine (e.g., Temporal, Camunda)

Manage saga state, send commands, and trigger compensations on failures

Saga State Store

Relational DB or NoSQL DB (e.g., PostgreSQL, MongoDB)

Persist saga transaction states and progress for reliability and recovery

Monitoring and Logging

Prometheus, Grafana, ELK stack

Track saga execution, failures, and performance metrics

Request Flow

1. Client initiates a distributed transaction request to the Saga Orchestrator.

2. Orchestrator sends a command to Service A to perform its local transaction.

3. Service A executes local transaction and publishes success event to Message Broker.

4. Orchestrator listens for Service A's success event, then sends command to Service B.

5. Service B performs local transaction and publishes success event.

6. Orchestrator proceeds similarly with Service C.

7. If all services succeed, orchestrator marks saga as completed.

8. If any service fails, orchestrator triggers compensating transactions in reverse order.

9. Each service executes its compensating action and publishes compensation success event.

10. Orchestrator updates saga state accordingly and reports final status to client.

Database Schema

Entities: - SagaTransaction: id (PK), status (pending, completed, compensating, failed), created_at, updated_at - SagaStep: id (PK), saga_transaction_id (FK), service_name, action, status (pending, success, failed, compensated), timestamp Relationships: - One SagaTransaction has many SagaSteps representing each service's action and compensation status.

Scaling Discussion

Bottlenecks

Saga Orchestrator becomes a single point of failure and bottleneck under high load.

Message Broker throughput limits can delay event delivery.

Database contention on saga state store with many concurrent transactions.

Handling long-running sagas with many steps increases complexity and resource usage.

Solutions

Deploy multiple orchestrator instances with leader election or partition sagas by ID for horizontal scaling.

Use a high-throughput, distributed message broker like Kafka with partitioning and replication.

Optimize saga state store with indexing, sharding, or use distributed NoSQL databases.

Implement timeout and compensation policies to clean up long-running sagas and avoid resource leaks.

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying assumptions, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and failure handling, 5 minutes summarizing.

Explain the difference between choreography and orchestration saga styles.

Describe how compensating transactions maintain data consistency without distributed locks.

Discuss asynchronous communication and eventual consistency trade-offs.

Highlight how saga state tracking enables recovery and monitoring.

Address scaling challenges and solutions for orchestrator, messaging, and storage.

Practice

1. What is the main purpose of the Saga pattern in microservices?

easy

A. To replicate data across multiple databases synchronously

B. To manage distributed transactions by breaking them into smaller steps with compensations

C. To speed up database queries by caching results

D. To lock all resources until the transaction completes

Solution

1. Step 1: Understand distributed transactions challenges

   Distributed transactions across microservices are hard because locking resources is inefficient and can cause delays.

2. Step 2: Identify Saga pattern role

   The Saga pattern breaks a big transaction into smaller steps, each with a compensating action to undo if needed, avoiding locks.

3. Final Answer:

   To manage distributed transactions by breaking them into smaller steps with compensations -> Option B

4. Quick Check:

   Saga pattern = distributed transaction management [OK]

Hint: Saga means small steps with undo actions for transactions [OK]

Common Mistakes:

• Thinking Saga locks resources like traditional transactions
• Confusing Saga with caching or replication
• Assuming Saga runs all steps in parallel

2. Which of the following is the correct sequence in a Saga pattern transaction?

easy

A. Execute steps and compensations simultaneously

B. Run compensations first, then execute all steps

C. Execute only compensations without any steps

D. Execute steps sequentially, then run compensations if any step fails

Solution

1. Step 1: Understand Saga execution flow

   Saga executes each step in order. If a step fails, compensations undo previous steps.

2. Step 2: Confirm correct sequence

   Compensations run only after a failure, never before or simultaneously with steps.

3. Final Answer:

   Execute steps sequentially, then run compensations if any step fails -> Option D

4. Quick Check:

   Steps then compensations = correct Saga flow [OK]

Hint: Steps run first; compensations only if failure occurs [OK]

Common Mistakes:

• Running compensations before any step
• Running steps and compensations at the same time
• Skipping compensations on failure

3. Consider a Saga with three steps: A, B, and C. Step B fails after A succeeds. What happens next?

medium

A. Saga retries step B indefinitely without compensation

B. Step C runs regardless of failure

C. Compensation for step A runs, then Saga aborts

D. No compensation runs; Saga commits partial results

Solution

1. Step 1: Analyze failure impact in Saga

   When step B fails, Saga must undo previous successful steps to keep data consistent.

2. Step 2: Identify compensation actions

   Compensation for step A runs to rollback its changes, then Saga aborts without running step C.

3. Final Answer:

   Compensation for step A runs, then Saga aborts -> Option C

4. Quick Check:

   Failure triggers compensation rollback [OK]

Hint: Failure in middle triggers compensations backward [OK]

Common Mistakes:

• Assuming later steps run after failure
• Thinking Saga retries endlessly without rollback
• Ignoring compensation steps

4. A developer implemented a Saga but noticed data inconsistencies after failures. What is the most likely cause?

medium

A. Compensation actions are missing or incomplete

B. All steps are executed synchronously

C. Steps are too small and independent

D. Saga pattern locks all resources during execution

Solution

1. Step 1: Identify cause of inconsistencies

   Data inconsistencies after failure usually mean rollback (compensation) did not happen properly.

2. Step 2: Check compensation implementation

   If compensation actions are missing or incomplete, previous steps cannot be undone, causing inconsistency.

3. Final Answer:

   Compensation actions are missing or incomplete -> Option A

4. Quick Check:

   Missing compensation = inconsistency [OK]

Hint: Always implement full compensations for each step [OK]

Common Mistakes:

• Assuming synchronous execution causes inconsistency
• Believing small steps cause inconsistency
• Thinking Saga locks resources like traditional transactions

5. You design a payment system using Saga pattern with steps: debit account, reserve inventory, and confirm order. If inventory reservation fails, what should happen?

hard

A. Run compensation to credit back the debited amount and abort order confirmation

B. Ignore failure and proceed to confirm order

C. Retry inventory reservation indefinitely without compensation

D. Lock all services until inventory is reserved

Solution

1. Step 1: Understand Saga compensation in payment flow

   If inventory reservation fails, previous successful steps (debit account) must be undone to avoid inconsistent state.

2. Step 2: Apply compensation and abort

   Compensation credits back the debited amount, and order confirmation is aborted to maintain consistency.

3. Final Answer:

   Run compensation to credit back the debited amount and abort order confirmation -> Option A

4. Quick Check:

   Failure triggers compensation rollback and abort [OK]

Hint: Failure in middle step triggers rollback of prior steps [OK]

Common Mistakes:

• Proceeding despite failure causing inconsistent state
• Retrying endlessly without rollback
• Locking services defeats Saga benefits