Saga pattern for distributed transactions in Kafka - Commands & Configuration

Introduction

When multiple services need to update data together, it can be hard to keep everything consistent if one part fails. The Saga pattern breaks a big transaction into smaller steps, each with a way to undo it if needed. This helps keep data correct across many services without locking everything.

When you have an order service, payment service, and inventory service that must update data together.

When you want to avoid long database locks in a multi-service environment.

When you need to handle failures gracefully by rolling back partial changes.

When you want to coordinate distributed transactions using asynchronous messaging.

When you want to track the state of a multi-step business process across services.

Config File - saga-topics-config.json

saga-topics-config.json

{
  "topics": [
    {
      "name": "order-commands",
      "partitions": 3,
      "replicationFactor": 2
    },
    {
      "name": "payment-commands",
      "partitions": 3,
      "replicationFactor": 2
    },
    {
      "name": "inventory-commands",
      "partitions": 3,
      "replicationFactor": 2
    },
    {
      "name": "saga-events",
      "partitions": 3,
      "replicationFactor": 2
    }
  ]
}

This JSON file defines Kafka topics used in the Saga pattern.

• order-commands, payment-commands, and inventory-commands are topics where each service listens for commands to perform their part of the transaction.
• saga-events is a topic where services publish events about the success or failure of each step.
• partitions and replicationFactor ensure scalability and fault tolerance.

Commands

Create the Kafka topic 'order-commands' where the order service will receive commands to start or update orders.

Terminal

kafka-topics --create --topic order-commands --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

Expected Output

Created topic order-commands.

→

--partitions - Number of partitions for parallelism

→

--replication-factor - Number of copies for fault tolerance

Create the 'saga-events' topic where all services publish events about transaction steps success or failure.

Terminal

kafka-topics --create --topic saga-events --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

Expected Output

Created topic saga-events.

→

--partitions - Number of partitions for parallelism

→

--replication-factor - Number of copies for fault tolerance

Send a command message to start a new order transaction. This triggers the saga workflow.

Terminal

kafka-console-producer --topic order-commands --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Topic to send messages to

→

--bootstrap-server - Kafka server address

Listen to all saga events from the beginning to monitor the progress and outcome of distributed transactions.

Terminal

kafka-console-consumer --topic saga-events --from-beginning --bootstrap-server localhost:9092

Expected Output

order-created-success

payment-processed-success

inventory-updated-success

saga-completed

→

--from-beginning - Read all messages from the start

→

--topic - Topic to consume messages from

Key Concept

If you remember nothing else from this pattern, remember: the Saga pattern manages distributed transactions by breaking them into steps with compensating actions to keep data consistent without locking.

Common Mistakes

Not creating separate Kafka topics for commands and events

This mixes command and event messages, making it hard to track transaction state and causing confusion between services.

Create distinct topics for commands each service listens to and a separate topic for saga events to track progress.

Not handling failure events with compensating actions

If a step fails and no rollback is done, data becomes inconsistent across services.

Implement compensating transactions that undo previous steps when a failure event is received.

Using only synchronous calls instead of Kafka messaging

Synchronous calls can cause tight coupling and blocking, losing the benefits of the Saga pattern's asynchronous flow.

Use Kafka topics to send commands and events asynchronously between services.

Summary

Create Kafka topics for commands and saga events to separate concerns.

Send commands to start transaction steps and listen for events to track progress.

Use compensating actions on failure events to keep data consistent across services.