Saga pattern for distributed transactions in Kafka - Mini Project: Build & Apply

Implementing the Saga Pattern for Distributed Transactions with Kafka

📖 Scenario: Your e-commerce platform uses separate microservices for orders, payments, and inventory. When a customer places an order, all three services must succeed or the entire transaction must roll back. You need to coordinate this distributed transaction using the saga pattern with Kafka as the event bus.

🎯 Goal: Build a Kafka-based saga coordination system by creating the required topics, publishing an order event, simulating a saga step, and implementing a compensation event for rollback.

📋 What You'll Learn

Create Kafka topics for saga events and compensation events

Publish an order-created event to start the saga

Consume the event and simulate a payment processing step

Publish a compensation event to simulate rollback on failure

Verify the full saga flow using console tools

💡 Why This Matters

🌍 Real World

The saga pattern coordinates distributed transactions across microservices without distributed locks. Kafka provides reliable event delivery for saga steps and compensation actions.

💼 Career

Backend and DevOps engineers working with microservices architectures implement saga patterns using Kafka to maintain data consistency across services.

1

Create Kafka topics for saga coordination

Create two Kafka topics: saga-events with 3 partitions for saga step events, and saga-compensation with 3 partitions for rollback compensation events. Use replication factor 1 for local development.

Kafka

# Create saga event topics
# Your code here

Hint

Use kafka-topics.sh --create twice — once for saga-events and once for saga-compensation.

2

Publish an order-created event to start the saga

Use the Kafka console producer to publish a JSON event to the saga-events topic. The event should contain an order ID, step name, and status: {"order_id":"ORD-001","step":"order-created","status":"success"}.

Kafka

kafka-topics.sh --create --topic saga-events --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1
kafka-topics.sh --create --topic saga-compensation --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1
# Publish order-created event here

Hint

Pipe a JSON string into kafka-console-producer.sh targeting the saga-events topic.

3

Consume the event and publish a payment failure for compensation

First consume from saga-events to verify the order event arrived. Then simulate a payment failure by publishing a compensation event to saga-compensation topic: {"order_id":"ORD-001","step":"payment-failed","action":"rollback-order"}.

Kafka

kafka-topics.sh --create --topic saga-events --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1
kafka-topics.sh --create --topic saga-compensation --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1
echo '{"order_id":"ORD-001","step":"order-created","status":"success"}' | kafka-console-producer.sh --bootstrap-server localhost:9092 --topic saga-events
# Consume saga event and publish compensation here

Hint

Use kafka-console-consumer.sh --from-beginning --max-messages 1 to read the event, then pipe a compensation JSON into the compensation topic.

4

Verify the compensation event was received

Consume from the saga-compensation topic to verify the rollback event is available for the order service to process.

Kafka

kafka-topics.sh --create --topic saga-events --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1
kafka-topics.sh --create --topic saga-compensation --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1
echo '{"order_id":"ORD-001","step":"order-created","status":"success"}' | kafka-console-producer.sh --bootstrap-server localhost:9092 --topic saga-events
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic saga-events --from-beginning --max-messages 1
echo '{"order_id":"ORD-001","step":"payment-failed","action":"rollback-order"}' | kafka-console-producer.sh --bootstrap-server localhost:9092 --topic saga-compensation
# Verify compensation event here

Hint

Use kafka-console-consumer.sh --from-beginning --max-messages 1 on the saga-compensation topic.

Practice

1. What is the main purpose of the Saga pattern in distributed systems?

easy

A. To encrypt messages between microservices

B. To speed up database queries by caching results

C. To break a large transaction into smaller steps with compensations

D. To create backups of all data in real-time

Solution

1. Step 1: Understand the Saga pattern concept

   The Saga pattern divides a big transaction into smaller, manageable steps that can be undone if needed.

2. Step 2: Identify the main goal

   Its goal is to maintain data consistency across services without locking resources for a long time.

3. Final Answer:

   To break a large transaction into smaller steps with compensations -> Option C

4. Quick Check:

   Saga pattern = smaller steps with undo [OK]

Hint: Saga splits big tasks into small reversible steps [OK]

Common Mistakes:

• Confusing Saga with caching or encryption
• Thinking Saga creates backups
• Assuming Saga speeds up queries

2. Which Kafka feature is most important for implementing the Saga pattern?

easy

A. Kafka Streams for data encryption

B. Kafka Consumer groups for load balancing

C. Kafka Connect for database backups

D. Kafka topics for event communication

Solution

1. Step 1: Identify Kafka components used in Saga

   Saga relies on events to coordinate steps, so Kafka topics are used to send these events.

2. Step 2: Exclude unrelated Kafka features

   Encryption, backups, and load balancing are not the main focus for Saga coordination.

3. Final Answer:

   Kafka topics for event communication -> Option D

4. Quick Check:

   Kafka topics = event communication [OK]

Hint: Saga uses Kafka topics to send events between services [OK]

Common Mistakes:

• Confusing Kafka Streams with event messaging
• Thinking Kafka Connect handles Saga coordination
• Mixing consumer groups with transaction steps

3. Given a Saga with three steps: A, B, and C, where each step publishes an event to Kafka, what happens if step B fails after step A succeeds?

medium

A. Step A is compensated by sending a rollback event

B. Step C executes anyway to complete the transaction

C. The entire Saga is aborted without compensation

D. Step B retries indefinitely without affecting others

Solution

1. Step 1: Understand failure handling in Saga

   If a step fails, previous successful steps must be undone using compensating actions.

2. Step 2: Apply to the scenario

   Since step B failed after A succeeded, step A must be compensated by sending a rollback event.

3. Final Answer:

   Step A is compensated by sending a rollback event -> Option A

4. Quick Check:

   Failure in step B triggers compensation for step A [OK]

Hint: Failed step triggers compensation for prior steps [OK]

Common Mistakes:

• Assuming next steps run despite failure
• Thinking Saga aborts without compensation
• Believing retries never stop

4. A developer implemented a Saga using Kafka but notices that compensating events are not triggered on failure. What is the most likely cause?

medium

A. The service does not listen to failure events to trigger compensation

B. Kafka topics are not partitioned correctly

C. Kafka consumer group ID is missing

D. Kafka brokers are overloaded

Solution

1. Step 1: Identify compensation trigger mechanism

   Compensation happens when a service listens for failure events and reacts by sending rollback events.

2. Step 2: Analyze the problem

   If compensations are missing, likely the service is not listening or handling failure events properly.

3. Final Answer:

   The service does not listen to failure events to trigger compensation -> Option A

4. Quick Check:

   Missing compensation = no failure event listener [OK]

Hint: Compensation needs failure event listeners active [OK]

Common Mistakes:

• Blaming Kafka partitions for compensation logic
• Confusing consumer group ID with compensation triggers
• Assuming broker overload causes missing compensations

5. You design a Saga with multiple microservices communicating via Kafka. To ensure scalability and fault tolerance, which approach is best?

hard

A. Store Saga state only in a centralized database without event logs

B. Use separate Kafka topics per Saga step and implement idempotent consumers

C. Use a single Kafka topic for all Saga events without consumer groups

D. Send synchronous HTTP calls between services instead of Kafka events

Solution

1. Step 1: Consider scalability and fault tolerance needs

   Separate topics per step allow independent scaling and isolation of events.

2. Step 2: Ensure idempotency in consumers

   Idempotent consumers prevent duplicate processing on retries, improving fault tolerance.

3. Step 3: Evaluate other options

   Single topic without consumer groups limits scalability; centralized DB without events loses Saga benefits; synchronous calls reduce fault tolerance.

4. Final Answer:

   Use separate Kafka topics per Saga step and implement idempotent consumers -> Option B

5. Quick Check:

   Separate topics + idempotent consumers = scalable Saga [OK]

Hint: Separate topics and idempotent consumers scale Saga well [OK]

Common Mistakes:

• Using one topic for all events limits scaling
• Ignoring idempotency causes duplicate side effects
• Relying on synchronous calls reduces fault tolerance