CQRS pattern in Kafka - Commands & Configuration

Introduction

CQRS helps separate how data is read and written in a system. This makes it easier to scale and manage complex data flows, especially when using Kafka to handle messages.

When you want to handle commands (writes) and queries (reads) differently for better performance.

When your system has complex business logic that changes data and simple queries that read data.

When you want to scale read and write workloads independently using Kafka topics.

When you want to keep your write operations fast and your read operations optimized for queries.

When you want to use Kafka to stream changes and update read models asynchronously.

Config File - kafka-cqrs-config.yaml

kafka-cqrs-config.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: commands-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 3
  replicas: 1
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: events-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 3
  replicas: 1
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: queries-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 3
  replicas: 1

This file creates three Kafka topics:

• commands-topic for write operations (commands).
• events-topic for events generated after commands are processed.
• queries-topic for read operations (queries).

Each topic has 3 partitions for parallel processing and 1 replica for simplicity.

Commands

This command creates the Kafka topics needed for the CQRS pattern: commands, events, and queries topics.

Terminal

kubectl apply -f kafka-cqrs-config.yaml

Expected Output

kafkatopic.kafka.strimzi.io/commands-topic created

kafkatopic.kafka.strimzi.io/events-topic created

kafkatopic.kafka.strimzi.io/queries-topic created

This command opens a console to send command messages to the commands-topic. Commands represent write requests.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic commands-topic

Expected Output

No output (command runs silently)

→

--broker-list - Specifies the Kafka broker address to connect to.

→

--topic - Specifies the Kafka topic to send messages to.

This command reads all event messages from the events-topic. Events are generated after commands are processed and represent state changes.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic events-topic --from-beginning

Expected Output

UserCreatedEvent {"userId":"123","name":"Alice"}

OrderPlacedEvent {"orderId":"456","amount":100}

→

--bootstrap-server - Specifies the Kafka broker address to connect to.

→

--topic - Specifies the Kafka topic to read messages from.

→

--from-beginning - Reads all messages from the start of the topic.

This command reads query results or read model updates from the queries-topic, which is optimized for fast reads.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic queries-topic --from-beginning

Expected Output

{"userId":"123","name":"Alice","status":"active"}

{"orderId":"456","amount":100,"status":"confirmed"}

→

--bootstrap-server - Specifies the Kafka broker address to connect to.

→

--topic - Specifies the Kafka topic to read messages from.

→

--from-beginning - Reads all messages from the start of the topic.

Key Concept

If you remember nothing else from this pattern, remember: separate write commands and read queries into different Kafka topics to improve scalability and clarity.

Common Mistakes

Using a single Kafka topic for both commands and queries.

This mixes write and read operations, causing confusion and performance issues.

Create separate Kafka topics for commands (writes) and queries (reads) as shown in the config file.

Not consuming events from the events-topic to update read models.

Without processing events, the read side will not reflect the latest state.

Set up consumers to listen to events-topic and update the queries-topic or read database accordingly.

Summary

Create separate Kafka topics for commands, events, and queries to implement CQRS.

Send write requests as commands to the commands-topic using a Kafka producer.

Consume events from the events-topic to track state changes.

Consume queries from the queries-topic to serve read requests efficiently.