CQRS pattern in Kafka - Commands & Configuration

Introduction

CQRS helps separate how data is read and written in a system. This makes it easier to scale and manage complex data flows, especially when using Kafka to handle messages.

When you want to handle commands (writes) and queries (reads) differently for better performance.

When your system has complex business logic that changes data and simple queries that read data.

When you want to scale read and write workloads independently using Kafka topics.

When you want to keep your write operations fast and your read operations optimized for queries.

When you want to use Kafka to stream changes and update read models asynchronously.

Config File - kafka-cqrs-config.yaml

kafka-cqrs-config.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: commands-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 3
  replicas: 1
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: events-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 3
  replicas: 1
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: queries-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 3
  replicas: 1

This file creates three Kafka topics:

• commands-topic for write operations (commands).
• events-topic for events generated after commands are processed.
• queries-topic for read operations (queries).

Each topic has 3 partitions for parallel processing and 1 replica for simplicity.

Commands

This command creates the Kafka topics needed for the CQRS pattern: commands, events, and queries topics.

Terminal

kubectl apply -f kafka-cqrs-config.yaml

Expected Output

kafkatopic.kafka.strimzi.io/commands-topic created

kafkatopic.kafka.strimzi.io/events-topic created

kafkatopic.kafka.strimzi.io/queries-topic created

This command opens a console to send command messages to the commands-topic. Commands represent write requests.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic commands-topic

Expected Output

No output (command runs silently)

→

--broker-list - Specifies the Kafka broker address to connect to.

→

--topic - Specifies the Kafka topic to send messages to.

This command reads all event messages from the events-topic. Events are generated after commands are processed and represent state changes.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic events-topic --from-beginning

Expected Output

UserCreatedEvent {"userId":"123","name":"Alice"}

OrderPlacedEvent {"orderId":"456","amount":100}

→

--bootstrap-server - Specifies the Kafka broker address to connect to.

→

--topic - Specifies the Kafka topic to read messages from.

→

--from-beginning - Reads all messages from the start of the topic.

This command reads query results or read model updates from the queries-topic, which is optimized for fast reads.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic queries-topic --from-beginning

Expected Output

{"userId":"123","name":"Alice","status":"active"}

{"orderId":"456","amount":100,"status":"confirmed"}

→

--bootstrap-server - Specifies the Kafka broker address to connect to.

→

--topic - Specifies the Kafka topic to read messages from.

→

--from-beginning - Reads all messages from the start of the topic.

Key Concept

If you remember nothing else from this pattern, remember: separate write commands and read queries into different Kafka topics to improve scalability and clarity.

Common Mistakes

Using a single Kafka topic for both commands and queries.

This mixes write and read operations, causing confusion and performance issues.

Create separate Kafka topics for commands (writes) and queries (reads) as shown in the config file.

Not consuming events from the events-topic to update read models.

Without processing events, the read side will not reflect the latest state.

Set up consumers to listen to events-topic and update the queries-topic or read database accordingly.

Summary

Create separate Kafka topics for commands, events, and queries to implement CQRS.

Send write requests as commands to the commands-topic using a Kafka producer.

Consume events from the events-topic to track state changes.

Consume queries from the queries-topic to serve read requests efficiently.

Practice

1. What is the main idea behind the CQRS pattern in Kafka-based systems?

easy

A. Separating commands (writes) and queries (reads) into different parts

B. Using a single topic for both commands and queries

C. Storing all data in one database without separation

D. Ignoring events and focusing only on commands

Solution

1. Step 1: Understand CQRS basics

   CQRS stands for Command Query Responsibility Segregation, which means separating write and read operations.

2. Step 2: Apply to Kafka context

   In Kafka, commands and queries are handled separately, often using different topics for clarity and scalability.

3. Final Answer:

   Separating commands (writes) and queries (reads) into different parts -> Option A

4. Quick Check:

   CQRS = Separate commands and queries [OK]

Hint: Remember CQRS splits writes and reads separately [OK]

Common Mistakes:

• Thinking commands and queries share the same topic
• Confusing CQRS with event sourcing
• Assuming CQRS means no events are used

2. Which Kafka topic setup correctly follows the CQRS pattern?

easy

A. One topic for all commands, events, and queries

B. One topic for commands and a separate topic for events

C. Multiple topics for queries only

D. No topics, just direct database writes

Solution

1. Step 1: Recall CQRS topic separation

   CQRS uses separate topics for commands (writes) and events (results of commands) to keep communication clear.

2. Step 2: Evaluate options

   One topic for commands and a separate topic for events correctly separates commands and events into different topics, matching CQRS principles.

3. Final Answer:

   One topic for commands and a separate topic for events -> Option B

4. Quick Check:

   Separate topics for commands and events = One topic for commands and a separate topic for events [OK]

Hint: Separate commands and events into different topics [OK]

Common Mistakes:

• Mixing commands and queries in one topic
• Ignoring event topics
• Using no Kafka topics at all

3. Given this Kafka setup for CQRS:

commands_topic = ['CreateOrder', 'UpdateOrder']
events_topic = ['OrderCreated', 'OrderUpdated']

Which topic should a read service subscribe to for query data?

medium

A. commands_topic

B. Both commands_topic and events_topic

C. events_topic

D. Neither, it reads directly from the database

Solution

1. Step 1: Identify read service role

   Read services in CQRS listen to events that represent state changes, not commands which are requests.

2. Step 2: Match topic to read service

   Events_topic contains 'OrderCreated' and 'OrderUpdated' which reflect changes; read service subscribes here.

3. Final Answer:

   events_topic -> Option C

4. Quick Check:

   Read service subscribes to events_topic [OK]

Hint: Reads listen to events, not commands [OK]

Common Mistakes:

• Subscribing read service to commands_topic
• Subscribing to both topics unnecessarily
• Assuming reads bypass Kafka

4. You have this Kafka CQRS code snippet:

producer.send('commands', 'CreateOrder')
consumer.subscribe(['commands', 'events'])

What is the main problem here?

medium

A. Consumer should not subscribe to both 'commands' and 'events' topics

B. Producer should send to 'events' topic, not 'commands'

C. Producer must send JSON, not strings

D. Consumer must subscribe only to 'commands' topic

Solution

1. Step 1: Understand CQRS topic roles

   Commands topic is for writes; events topic is for read updates. Consumers usually subscribe to one role.

2. Step 2: Analyze consumer subscription

   Subscribing to both commands and events mixes responsibilities and can cause confusion or errors.

3. Final Answer:

   Consumer should not subscribe to both 'commands' and 'events' topics -> Option A

4. Quick Check:

   Consumer subscribes to one topic type only [OK]

Hint: Consumers should subscribe to either commands or events, not both [OK]

Common Mistakes:

• Thinking producer must send events on commands topic
• Assuming consumer can handle both topics safely
• Ignoring message format requirements

5. How can you design a Kafka-based CQRS system to handle high read traffic without slowing down writes?

hard

A. Use only one consumer group for both commands and events

B. Combine commands and events in one topic to reduce overhead

C. Write all data directly to the database and skip Kafka for reads

D. Use separate Kafka topics for commands and events, and scale read consumers independently

Solution

1. Step 1: Apply CQRS scalability principle

   Separating commands and events allows independent scaling of write and read sides.

2. Step 2: Design Kafka topics and consumers

   Using separate topics for commands and events lets read consumers scale without affecting write performance.

3. Final Answer:

   Use separate Kafka topics for commands and events, and scale read consumers independently -> Option D

4. Quick Check:

   Separate topics and scale reads independently [OK]

Hint: Separate topics and scale read consumers for high traffic [OK]

Common Mistakes:

• Combining commands and events causing bottlenecks
• Skipping Kafka for reads losing scalability
• Using one consumer group for all messages