Saga Pattern with Kafka: How It Works and When to Use

The saga pattern with Kafka is a way to manage long business transactions across multiple microservices by breaking them into smaller steps and using Kafka events to coordinate success or rollback. Each service publishes events to Kafka topics to trigger the next step or compensate if something fails, ensuring data consistency without locking resources.

How It Works

Imagine you want to buy a product online. This process involves several steps like reserving stock, charging payment, and confirming the order. Each step is handled by a different microservice. The saga pattern breaks this big task into smaller pieces that happen one after another.

Kafka acts like a message bus where each microservice listens for events and publishes new events when it finishes its part. If a step fails, the saga triggers compensating actions to undo previous steps, like refunding payment or releasing stock. This way, the system stays consistent without locking everything at once.

Example

This example shows a simple saga using Kafka where an order service triggers a payment service and then a shipping service. Each service listens to Kafka topics and publishes success or failure events.

java

import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Collections;
import java.util.Properties;

public class OrderService {
    private KafkaProducer<String, String> producer;
    private KafkaConsumer<String, String> consumer;

    public OrderService(Properties producerProps, Properties consumerProps) {
        producer = new KafkaProducer<>(producerProps);
        consumer = new KafkaConsumer<>(consumerProps);
        consumer.subscribe(Collections.singletonList("payment-result"));
    }

    public void createOrder(String orderId) {
        // Step 1: Send payment request
        producer.send(new ProducerRecord<>("payment-request", orderId, "pay"));

        // Step 2: Listen for payment result
        consumer.poll(java.time.Duration.ofMillis(1000)).forEach(record -> {
            if (record.key().equals(orderId)) {
                if (record.value().equals("success")) {
                    // Proceed to shipping
                    producer.send(new ProducerRecord<>("shipping-request", orderId, "ship"));
                } else {
                    // Compensate or cancel order
                    System.out.println("Payment failed, cancelling order " + orderId);
                }
            }
        });
    }
}

// Similar PaymentService and ShippingService would consume and produce events accordingly.

Output

Payment failed, cancelling order 12345

When to Use

Use the saga pattern with Kafka when you have multiple microservices that must work together to complete a business process and you want to avoid locking resources or using distributed transactions.

It is ideal for systems where eventual consistency is acceptable and you want to handle failures gracefully by compensating previous steps. Common use cases include order processing, payment workflows, and inventory management in e-commerce platforms.

Key Points

• Saga breaks a big transaction into smaller, independent steps.
• Kafka is used to send and receive events between microservices.
• Each step publishes success or failure events to trigger next actions or compensations.
• Helps maintain data consistency without locking resources.
• Best for distributed systems where eventual consistency is acceptable.

Key Takeaways

The saga pattern manages distributed transactions by splitting them into smaller steps coordinated via Kafka events.

Kafka topics carry success or failure messages to trigger next steps or compensations in microservices.

Use saga with Kafka when you want eventual consistency without locking resources in distributed systems.

Compensating actions undo previous steps if a failure occurs, keeping data consistent.

Ideal for complex workflows like order processing and payment in microservice architectures.