Saga pattern for distributed transactions in HLD - System Design Exercise

Design: Distributed Transaction Management using Saga Pattern

Design the saga orchestration and choreography mechanisms for distributed transactions. Out of scope: detailed microservice business logic, database schema design for individual services.

Functional Requirements

FR1: Support transactions that span multiple microservices or databases

FR2: Ensure data consistency across services without using distributed locks

FR3: Handle failures by compensating or undoing partial work

FR4: Allow concurrent transactions without blocking

FR5: Provide visibility into transaction status for monitoring

Non-Functional Requirements

NFR1: Must support at least 1000 concurrent distributed transactions

NFR2: End-to-end transaction latency should be under 2 seconds on average

NFR3: System availability target of 99.9% uptime

NFR4: Services are loosely coupled and communicate asynchronously

NFR5: No global distributed transaction coordinator allowed

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Saga Orchestrator service or event bus for choreography

Microservices participating in the transaction

Message broker or event streaming platform

Saga state store or transaction log

Compensation handlers for undo operations

Design Patterns

Saga Orchestration pattern

Saga Choreography pattern

Event-driven architecture

Compensating transactions

Idempotent message handling

Reference Architecture

          +-------------------+          
          |   Client Request  |          
          +---------+---------+          
                    |                    
                    v                    
          +-------------------+          
          | Saga Orchestrator  |          
          +----+---------+----+          
               |         |               
       +-------+         +-------+       
       |                         |       
+------+-----+           +-------+-----+
| Service A  |           | Service B   |
+------------+           +-------------+
       |                         |       
       v                         v       
+------------+           +-------------+
| DB A       |           | DB B        |
+------------+           +-------------+

Legend:
- Saga Orchestrator sends commands to services
- Services perform local transactions and reply
- On failure, orchestrator triggers compensations
- Communication via asynchronous messages

Components

Saga Orchestrator

Stateless service with persistent state store (e.g., Redis, PostgreSQL)

Coordinates the sequence of local transactions and compensations across services

Microservices (Service A, Service B, ...)

Independent services with own databases

Perform local transactions and execute compensating actions if requested

Message Broker

Kafka, RabbitMQ, or AWS SNS/SQS

Facilitates asynchronous communication between orchestrator and services

Saga State Store

Durable database like PostgreSQL or Redis

Stores current state and progress of each saga transaction

Request Flow

1. Client sends a transaction request to Saga Orchestrator

2. Orchestrator sends command to Service A to perform local transaction

3. Service A executes transaction on its database and replies success or failure

4. If success, orchestrator sends command to Service B for its local transaction

5. If any service fails, orchestrator sends compensating commands to undo previous successful steps

6. Services execute compensations and confirm completion

7. Orchestrator updates saga state and notifies client of final outcome

Database Schema

Entities: - SagaTransaction: id (PK), status (pending, completed, compensating, failed), created_at, updated_at - SagaStep: id (PK), saga_transaction_id (FK), service_name, step_name, status (pending, success, failed, compensated), started_at, finished_at Relationships: - One SagaTransaction has many SagaSteps - SagaSteps track each local transaction or compensation within the saga

Scaling Discussion

Bottlenecks

Saga Orchestrator becomes a single point of failure or bottleneck

Message broker overload with high transaction volume

State store latency impacting saga progress tracking

Handling long-running sagas with many steps increases complexity

Solutions

Deploy multiple orchestrator instances with leader election or partitioning by saga ID

Use scalable, distributed message brokers with partitioning and replication

Optimize state store with caching and efficient queries; consider event sourcing

Implement timeout and retry policies; break large sagas into smaller sub-sagas

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and failure handling, 5 minutes summarizing.

Explain why distributed transactions are hard and why two-phase commit is not ideal for microservices

Describe the difference between saga orchestration and choreography

Show how compensating transactions maintain data consistency

Discuss asynchronous communication and eventual consistency

Highlight how to handle failures and retries gracefully

Mention scalability and availability considerations

Practice

1. What is the main purpose of the Saga pattern in distributed systems?

easy

A. To replicate data across multiple servers for backup

B. To lock all resources until the transaction completes

C. To speed up database queries by caching results

D. To manage long transactions by splitting them into smaller steps with compensations

Solution

1. Step 1: Understand the problem Saga solves

   The Saga pattern handles distributed transactions by breaking them into smaller steps that can be undone if needed.

2. Step 2: Compare options with Saga's goal

   Locking resources or caching are unrelated to Saga's main goal of managing distributed transactions with compensations.

3. Final Answer:

   To manage long transactions by splitting them into smaller steps with compensations -> Option D

4. Quick Check:

   Saga pattern purpose = Manage transactions with compensations [OK]

Hint: Saga splits big tasks into steps with undo actions [OK]

Common Mistakes:

• Thinking Saga locks resources like traditional transactions
• Confusing Saga with caching or replication techniques
• Assuming Saga only works with single database systems

2. Which of the following is the correct sequence in a Saga transaction?

easy

A. Execute steps sequentially, running compensations for previous steps if any step fails

B. Run compensations first, then execute all steps

C. Execute steps and compensations simultaneously

D. Execute steps without any compensations

Solution

1. Step 1: Recall Saga transaction flow

   Saga executes steps one by one. If a step fails, compensations undo previous steps.

2. Step 2: Eliminate incorrect sequences

   Running compensations before steps or simultaneously is incorrect. Skipping compensations breaks consistency.

3. Final Answer:

   Execute steps sequentially, running compensations for previous steps if any step fails -> Option A

4. Quick Check:

   Saga sequence = Steps then compensations on failure [OK]

Hint: Steps run first; compensations only if failure occurs [OK]

Common Mistakes:

• Running compensations before any step executes
• Assuming compensations run regardless of success
• Thinking steps and compensations run at the same time

3. Consider a Saga with three steps: A, B, and C. Step B fails after A succeeds. What happens next?

medium

A. Compensate step A, then abort the Saga

B. Retry step B indefinitely

C. Proceed to step C despite failure

D. Ignore failure and commit all steps

Solution

1. Step 1: Identify failure handling in Saga

   If step B fails, Saga triggers compensations for all previous successful steps, here step A.

2. Step 2: Understand why other options fail

   Retrying indefinitely can cause blocking; proceeding ignores failure; ignoring failure breaks consistency.

3. Final Answer:

   Compensate step A, then abort the Saga -> Option A

4. Quick Check:

   Failure in step B triggers compensation of A [OK]

Hint: Failure triggers undo of prior successful steps [OK]

Common Mistakes:

• Assuming Saga retries failed steps endlessly
• Skipping compensations and continuing steps
• Ignoring failure and committing partial results

4. A developer implemented a Saga but noticed data inconsistencies after failures. What is a likely cause?

medium

A. Saga uses asynchronous messaging

B. Compensation actions are missing or incomplete

C. Steps are idempotent and retry safe

D. All steps are executed sequentially

Solution

1. Step 1: Analyze cause of inconsistencies

   Missing or incomplete compensation means failed steps do not undo prior changes, causing inconsistency.

2. Step 2: Evaluate other options

   Sequential execution, idempotency, and async messaging are good practices and do not cause inconsistencies alone.

3. Final Answer:

   Compensation actions are missing or incomplete -> Option B

4. Quick Check:

   Missing compensations cause inconsistencies [OK]

Hint: Check if compensations are properly implemented [OK]

Common Mistakes:

• Blaming sequential execution for inconsistency
• Ignoring importance of compensation actions
• Assuming async messaging causes inconsistency

5. You design a Saga for an e-commerce order process with payment, inventory, and shipping services. Which approach best ensures data consistency across these services?

hard

A. Use a global database lock across all services during the order process

B. Allow each service to commit independently without rollback

C. Implement compensating transactions for each service step to rollback on failure

D. Retry failed steps indefinitely without compensation

Solution

1. Step 1: Understand distributed transaction challenges

   Locking globally is impractical; independent commits without rollback cause inconsistency.

2. Step 2: Apply Saga pattern best practice

   Compensating transactions allow rollback of previous steps if any step fails, ensuring consistency.

3. Final Answer:

   Implement compensating transactions for each service step to rollback on failure -> Option C

4. Quick Check:

   Compensations ensure consistency in distributed Saga [OK]

Hint: Use compensations, not global locks, for distributed consistency [OK]

Common Mistakes:

• Trying to lock all services globally
• Ignoring rollback on failure
• Relying on infinite retries without undo