Saga pattern for distributed transactions in Microservices - System Design Exercise

Design: Distributed Transaction Management using Saga Pattern

Design focuses on the saga pattern implementation for distributed transactions across microservices. It excludes detailed service business logic and UI design.

Functional Requirements

FR1: Support transactions that span multiple microservices

FR2: Ensure data consistency across services without using distributed locks

FR3: Handle failures by compensating transactions to rollback partial changes

FR4: Support both choreography and orchestration styles of saga

FR5: Provide visibility into transaction status for monitoring and debugging

Non-Functional Requirements

NFR1: Must handle up to 10,000 concurrent distributed transactions

NFR2: End-to-end transaction latency should be under 5 seconds p99

NFR3: System availability target is 99.9% uptime

NFR4: Services are loosely coupled and communicate asynchronously

NFR5: No single point of failure in transaction coordination

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Saga Orchestrator or Event Bus for choreography

Microservices with local transaction and compensating actions

Message broker for asynchronous communication

Saga state store (database or distributed cache)

Monitoring and logging system

Design Patterns

Saga pattern (choreography and orchestration)

Event-driven architecture

Compensating transactions

Idempotency and retry mechanisms

State machine for saga status tracking

Reference Architecture

 +----------------+       +----------------+       +----------------+
 |  Service A     |       |  Service B     |       |  Service C     |
 | (Local Txn +   |       | (Local Txn +   |       | (Local Txn +   |
 |  Compensate)   |       |  Compensate)   |       |  Compensate)   |
 +-------+--------+       +-------+--------+       +-------+--------+
         |                        |                        |
         |                        |                        |
         |                        |                        |
         |                        |                        |
         |                        |                        |
         |                        |                        |
 +-------v------------------------v------------------------v-------+
 |                      Message Broker / Event Bus                  |
 +------------------------------------------------------------------+
         |                        |                        |
         |                        |                        |
 +-------v----------------------------------------------------------v-------+
 |                         Saga Orchestrator / Coordinator                 |
 |  - Tracks saga state                                                    |
 |  - Sends commands to services                                          |
 |  - Handles compensations on failure                                    |
 +------------------------------------------------------------------------+

Legend:
- Services perform local transactions and define compensating actions.
- Message Broker enables asynchronous communication.
- Saga Orchestrator manages transaction flow and state.

Components

Microservices

Any language/framework supporting microservices

Perform local transactions and define compensating actions for rollback

Message Broker / Event Bus

Kafka, RabbitMQ, or AWS SNS/SQS

Enable asynchronous communication between services and orchestrator

Saga Orchestrator / Coordinator

Custom service or workflow engine (e.g., Temporal, Camunda)

Manage saga state, send commands, and trigger compensations on failures

Saga State Store

Relational DB or NoSQL DB (e.g., PostgreSQL, MongoDB)

Persist saga transaction states and progress for reliability and recovery

Monitoring and Logging

Prometheus, Grafana, ELK stack

Track saga execution, failures, and performance metrics

Request Flow

1. Client initiates a distributed transaction request to the Saga Orchestrator.

2. Orchestrator sends a command to Service A to perform its local transaction.

3. Service A executes local transaction and publishes success event to Message Broker.

4. Orchestrator listens for Service A's success event, then sends command to Service B.

5. Service B performs local transaction and publishes success event.

6. Orchestrator proceeds similarly with Service C.

7. If all services succeed, orchestrator marks saga as completed.

8. If any service fails, orchestrator triggers compensating transactions in reverse order.

9. Each service executes its compensating action and publishes compensation success event.

10. Orchestrator updates saga state accordingly and reports final status to client.

Database Schema

Entities: - SagaTransaction: id (PK), status (pending, completed, compensating, failed), created_at, updated_at - SagaStep: id (PK), saga_transaction_id (FK), service_name, action, status (pending, success, failed, compensated), timestamp Relationships: - One SagaTransaction has many SagaSteps representing each service's action and compensation status.

Scaling Discussion

Bottlenecks

Saga Orchestrator becomes a single point of failure and bottleneck under high load.

Message Broker throughput limits can delay event delivery.

Database contention on saga state store with many concurrent transactions.

Handling long-running sagas with many steps increases complexity and resource usage.

Solutions

Deploy multiple orchestrator instances with leader election or partition sagas by ID for horizontal scaling.

Use a high-throughput, distributed message broker like Kafka with partitioning and replication.

Optimize saga state store with indexing, sharding, or use distributed NoSQL databases.

Implement timeout and compensation policies to clean up long-running sagas and avoid resource leaks.

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying assumptions, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and failure handling, 5 minutes summarizing.

Explain the difference between choreography and orchestration saga styles.

Describe how compensating transactions maintain data consistency without distributed locks.

Discuss asynchronous communication and eventual consistency trade-offs.

Highlight how saga state tracking enables recovery and monitoring.

Address scaling challenges and solutions for orchestrator, messaging, and storage.