Saga pattern for distributed transactions in Kafka - Commands & Configuration

Introduction

When multiple services need to update data together, it can be hard to keep everything consistent if one part fails. The Saga pattern breaks a big transaction into smaller steps, each with a way to undo it if needed. This helps keep data correct across many services without locking everything.

When you have an order service, payment service, and inventory service that must update data together.

When you want to avoid long database locks in a multi-service environment.

When you need to handle failures gracefully by rolling back partial changes.

When you want to coordinate distributed transactions using asynchronous messaging.

When you want to track the state of a multi-step business process across services.

Config File - saga-topics-config.json

saga-topics-config.json

{
  "topics": [
    {
      "name": "order-commands",
      "partitions": 3,
      "replicationFactor": 2
    },
    {
      "name": "payment-commands",
      "partitions": 3,
      "replicationFactor": 2
    },
    {
      "name": "inventory-commands",
      "partitions": 3,
      "replicationFactor": 2
    },
    {
      "name": "saga-events",
      "partitions": 3,
      "replicationFactor": 2
    }
  ]
}

This JSON file defines Kafka topics used in the Saga pattern.

• order-commands, payment-commands, and inventory-commands are topics where each service listens for commands to perform their part of the transaction.
• saga-events is a topic where services publish events about the success or failure of each step.
• partitions and replicationFactor ensure scalability and fault tolerance.

Commands

Create the Kafka topic 'order-commands' where the order service will receive commands to start or update orders.

Terminal

kafka-topics --create --topic order-commands --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

Expected Output

Created topic order-commands.

→

--partitions - Number of partitions for parallelism

→

--replication-factor - Number of copies for fault tolerance

Create the 'saga-events' topic where all services publish events about transaction steps success or failure.

Terminal

kafka-topics --create --topic saga-events --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

Expected Output

Created topic saga-events.

→

--partitions - Number of partitions for parallelism

→

--replication-factor - Number of copies for fault tolerance

Send a command message to start a new order transaction. This triggers the saga workflow.

Terminal

kafka-console-producer --topic order-commands --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Topic to send messages to

→

--bootstrap-server - Kafka server address

Listen to all saga events from the beginning to monitor the progress and outcome of distributed transactions.

Terminal

kafka-console-consumer --topic saga-events --from-beginning --bootstrap-server localhost:9092

Expected Output

order-created-success

payment-processed-success

inventory-updated-success

saga-completed

→

--from-beginning - Read all messages from the start

→

--topic - Topic to consume messages from

Key Concept

If you remember nothing else from this pattern, remember: the Saga pattern manages distributed transactions by breaking them into steps with compensating actions to keep data consistent without locking.

Common Mistakes

Not creating separate Kafka topics for commands and events

This mixes command and event messages, making it hard to track transaction state and causing confusion between services.

Create distinct topics for commands each service listens to and a separate topic for saga events to track progress.

Not handling failure events with compensating actions

If a step fails and no rollback is done, data becomes inconsistent across services.

Implement compensating transactions that undo previous steps when a failure event is received.

Using only synchronous calls instead of Kafka messaging

Synchronous calls can cause tight coupling and blocking, losing the benefits of the Saga pattern's asynchronous flow.

Use Kafka topics to send commands and events asynchronously between services.

Summary

Create Kafka topics for commands and saga events to separate concerns.

Send commands to start transaction steps and listen for events to track progress.

Use compensating actions on failure events to keep data consistent across services.

Practice

1. What is the main purpose of the Saga pattern in distributed systems?

easy

A. To encrypt messages between microservices

B. To speed up database queries by caching results

C. To break a large transaction into smaller steps with compensations

D. To create backups of all data in real-time

Solution

1. Step 1: Understand the Saga pattern concept

   The Saga pattern divides a big transaction into smaller, manageable steps that can be undone if needed.

2. Step 2: Identify the main goal

   Its goal is to maintain data consistency across services without locking resources for a long time.

3. Final Answer:

   To break a large transaction into smaller steps with compensations -> Option C

4. Quick Check:

   Saga pattern = smaller steps with undo [OK]

Hint: Saga splits big tasks into small reversible steps [OK]

Common Mistakes:

• Confusing Saga with caching or encryption
• Thinking Saga creates backups
• Assuming Saga speeds up queries

2. Which Kafka feature is most important for implementing the Saga pattern?

easy

A. Kafka Streams for data encryption

B. Kafka Consumer groups for load balancing

C. Kafka Connect for database backups

D. Kafka topics for event communication

Solution

1. Step 1: Identify Kafka components used in Saga

   Saga relies on events to coordinate steps, so Kafka topics are used to send these events.

2. Step 2: Exclude unrelated Kafka features

   Encryption, backups, and load balancing are not the main focus for Saga coordination.

3. Final Answer:

   Kafka topics for event communication -> Option D

4. Quick Check:

   Kafka topics = event communication [OK]

Hint: Saga uses Kafka topics to send events between services [OK]

Common Mistakes:

• Confusing Kafka Streams with event messaging
• Thinking Kafka Connect handles Saga coordination
• Mixing consumer groups with transaction steps

3. Given a Saga with three steps: A, B, and C, where each step publishes an event to Kafka, what happens if step B fails after step A succeeds?

medium

A. Step A is compensated by sending a rollback event

B. Step C executes anyway to complete the transaction

C. The entire Saga is aborted without compensation

D. Step B retries indefinitely without affecting others

Solution

1. Step 1: Understand failure handling in Saga

   If a step fails, previous successful steps must be undone using compensating actions.

2. Step 2: Apply to the scenario

   Since step B failed after A succeeded, step A must be compensated by sending a rollback event.

3. Final Answer:

   Step A is compensated by sending a rollback event -> Option A

4. Quick Check:

   Failure in step B triggers compensation for step A [OK]

Hint: Failed step triggers compensation for prior steps [OK]

Common Mistakes:

• Assuming next steps run despite failure
• Thinking Saga aborts without compensation
• Believing retries never stop

4. A developer implemented a Saga using Kafka but notices that compensating events are not triggered on failure. What is the most likely cause?

medium

A. The service does not listen to failure events to trigger compensation

B. Kafka topics are not partitioned correctly

C. Kafka consumer group ID is missing

D. Kafka brokers are overloaded

Solution

1. Step 1: Identify compensation trigger mechanism

   Compensation happens when a service listens for failure events and reacts by sending rollback events.

2. Step 2: Analyze the problem

   If compensations are missing, likely the service is not listening or handling failure events properly.

3. Final Answer:

   The service does not listen to failure events to trigger compensation -> Option A

4. Quick Check:

   Missing compensation = no failure event listener [OK]

Hint: Compensation needs failure event listeners active [OK]

Common Mistakes:

• Blaming Kafka partitions for compensation logic
• Confusing consumer group ID with compensation triggers
• Assuming broker overload causes missing compensations

5. You design a Saga with multiple microservices communicating via Kafka. To ensure scalability and fault tolerance, which approach is best?

hard

A. Store Saga state only in a centralized database without event logs

B. Use separate Kafka topics per Saga step and implement idempotent consumers

C. Use a single Kafka topic for all Saga events without consumer groups

D. Send synchronous HTTP calls between services instead of Kafka events

Solution

1. Step 1: Consider scalability and fault tolerance needs

   Separate topics per step allow independent scaling and isolation of events.

2. Step 2: Ensure idempotency in consumers

   Idempotent consumers prevent duplicate processing on retries, improving fault tolerance.

3. Step 3: Evaluate other options

   Single topic without consumer groups limits scalability; centralized DB without events loses Saga benefits; synchronous calls reduce fault tolerance.

4. Final Answer:

   Use separate Kafka topics per Saga step and implement idempotent consumers -> Option B

5. Quick Check:

   Separate topics + idempotent consumers = scalable Saga [OK]

Hint: Separate topics and idempotent consumers scale Saga well [OK]

Common Mistakes:

• Using one topic for all events limits scaling
• Ignoring idempotency causes duplicate side effects
• Relying on synchronous calls reduces fault tolerance