CQRS pattern in Kafka - Step-by-Step Execution

Process Flow - CQRS pattern

Client sends Command

↓

Command Handler processes

↓

Write Model updates DB

↓

Event published to Kafka

↓

Read Model consumes Event

↓

Read Model updates Query DB

↓

Client sends Query

↓

Query Handler reads from Query DB

↓

Response

The CQRS pattern splits commands (writes) and queries (reads) into separate flows, using Kafka events to keep read data updated.

Execution Sample

Kafka

1. Client sends command: CreateOrder
2. Command Handler writes to Orders DB
3. Publish OrderCreated event to Kafka
4. Read Model consumes event
5. Update Read DB
6. Client queries order status

This shows how a command triggers a write and event, which updates the read model for queries.

Process Table

Step	Action	Component	Data State	Kafka Event	Output
1	Send CreateOrder command	Client	No new order	None	Command sent
2	Process command	Command Handler	Order saved in Write DB	OrderCreated event published	Event published
3	Consume event	Read Model	Read DB updated with order	OrderCreated event consumed	Read DB updated
4	Send Query for order status	Client	Read DB has order data	None	Query sent
5	Process query	Query Handler	Read DB accessed	None	Order status returned
6	End	System	Consistent read and write DBs	No new events	Response sent

💡 Process ends after query response; read model is updated asynchronously from write model via Kafka events.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 5	Final
Write DB (Orders)	Empty	Order created	Order created	Order created	Order created
Kafka Event Queue	Empty	OrderCreated event published	OrderCreated event consumed	Empty	Empty
Read DB (Query)	Empty	Empty	Order data updated	Order data updated	Order data updated
Client Output	None	None	None	Order status response	Order status response

Key Moments - 3 Insights

Why does the read database update after the write database?

Because the read model updates asynchronously by consuming Kafka events after the write model saves data, as shown in steps 2 and 3 in the execution_table.

What happens if the read model is not updated yet when a query arrives?

The query handler reads from the read DB which may be stale until the event is consumed; this eventual consistency is normal in CQRS, as seen in step 4 and 5.

Why do we use Kafka events between write and read models?

Kafka events decouple the write and read sides, allowing asynchronous updates and scalability, shown by the 'OrderCreated event published' and 'consumed' in steps 2 and 3.

Visual Quiz - 3 Questions

Look at the execution_table, at which step is the OrderCreated event published to Kafka?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Check the Kafka Event column in execution_table rows for when the event is published.

According to variable_tracker, what is the state of the Read DB after step 3?

A Order data updated

B Order created

C Empty

D Order status response

💡 Hint

Look at the Read DB (Query) row after Step 3 in variable_tracker.

If the Kafka event is delayed, which step's output would be affected?

A Step 5: Query Handler reading data

B Step 2: Command Handler processing

C Step 3: Read Model consuming event

D Step 1: Client sending command

💡 Hint

Delayed Kafka event affects when the Read Model updates, see Step 3 in execution_table.

Concept Snapshot

CQRS splits write (commands) and read (queries) into separate models.
Commands update the write DB and publish events to Kafka.
Read model consumes events to update the read DB asynchronously.
Queries read from the read DB for fast, scalable reads.
This pattern enables scalability and separation of concerns.

Full Transcript

The CQRS pattern separates commands and queries into different flows. When a client sends a command, the command handler updates the write database and publishes an event to Kafka. The read model listens to Kafka events and updates the read database asynchronously. When the client sends a query, the query handler reads from the read database and returns the response. This separation allows the system to scale reads and writes independently and keeps data eventually consistent. The execution table shows each step from command sending to query response, tracking data states and Kafka events. The variable tracker shows how the write DB, Kafka event queue, read DB, and client output change over time. Key moments clarify why the read DB updates after the write DB and the role of Kafka events. The visual quiz tests understanding of event publishing, read DB state, and effects of event delays.

Practice

1. What is the main idea behind the CQRS pattern in Kafka-based systems?

easy

A. Separating commands (writes) and queries (reads) into different parts

B. Using a single topic for both commands and queries

C. Storing all data in one database without separation

D. Ignoring events and focusing only on commands

Solution

1. Step 1: Understand CQRS basics

   CQRS stands for Command Query Responsibility Segregation, which means separating write and read operations.

2. Step 2: Apply to Kafka context

   In Kafka, commands and queries are handled separately, often using different topics for clarity and scalability.

3. Final Answer:

   Separating commands (writes) and queries (reads) into different parts -> Option A

4. Quick Check:

   CQRS = Separate commands and queries [OK]

Hint: Remember CQRS splits writes and reads separately [OK]

Common Mistakes:

• Thinking commands and queries share the same topic
• Confusing CQRS with event sourcing
• Assuming CQRS means no events are used

2. Which Kafka topic setup correctly follows the CQRS pattern?

easy

A. One topic for all commands, events, and queries

B. One topic for commands and a separate topic for events

C. Multiple topics for queries only

D. No topics, just direct database writes

Solution

1. Step 1: Recall CQRS topic separation

   CQRS uses separate topics for commands (writes) and events (results of commands) to keep communication clear.

2. Step 2: Evaluate options

   One topic for commands and a separate topic for events correctly separates commands and events into different topics, matching CQRS principles.

3. Final Answer:

   One topic for commands and a separate topic for events -> Option B

4. Quick Check:

   Separate topics for commands and events = One topic for commands and a separate topic for events [OK]

Hint: Separate commands and events into different topics [OK]

Common Mistakes:

• Mixing commands and queries in one topic
• Ignoring event topics
• Using no Kafka topics at all

3. Given this Kafka setup for CQRS:

commands_topic = ['CreateOrder', 'UpdateOrder']
events_topic = ['OrderCreated', 'OrderUpdated']

Which topic should a read service subscribe to for query data?

medium

A. commands_topic

B. Both commands_topic and events_topic

C. events_topic

D. Neither, it reads directly from the database

Solution

1. Step 1: Identify read service role

   Read services in CQRS listen to events that represent state changes, not commands which are requests.

2. Step 2: Match topic to read service

   Events_topic contains 'OrderCreated' and 'OrderUpdated' which reflect changes; read service subscribes here.

3. Final Answer:

   events_topic -> Option C

4. Quick Check:

   Read service subscribes to events_topic [OK]

Hint: Reads listen to events, not commands [OK]

Common Mistakes:

• Subscribing read service to commands_topic
• Subscribing to both topics unnecessarily
• Assuming reads bypass Kafka

4. You have this Kafka CQRS code snippet:

producer.send('commands', 'CreateOrder')
consumer.subscribe(['commands', 'events'])

What is the main problem here?

medium

A. Consumer should not subscribe to both 'commands' and 'events' topics

B. Producer should send to 'events' topic, not 'commands'

C. Producer must send JSON, not strings

D. Consumer must subscribe only to 'commands' topic

Solution

1. Step 1: Understand CQRS topic roles

   Commands topic is for writes; events topic is for read updates. Consumers usually subscribe to one role.

2. Step 2: Analyze consumer subscription

   Subscribing to both commands and events mixes responsibilities and can cause confusion or errors.

3. Final Answer:

   Consumer should not subscribe to both 'commands' and 'events' topics -> Option A

4. Quick Check:

   Consumer subscribes to one topic type only [OK]

Hint: Consumers should subscribe to either commands or events, not both [OK]

Common Mistakes:

• Thinking producer must send events on commands topic
• Assuming consumer can handle both topics safely
• Ignoring message format requirements

5. How can you design a Kafka-based CQRS system to handle high read traffic without slowing down writes?

hard

A. Use only one consumer group for both commands and events

B. Combine commands and events in one topic to reduce overhead

C. Write all data directly to the database and skip Kafka for reads

D. Use separate Kafka topics for commands and events, and scale read consumers independently

Solution

1. Step 1: Apply CQRS scalability principle

   Separating commands and events allows independent scaling of write and read sides.

2. Step 2: Design Kafka topics and consumers

   Using separate topics for commands and events lets read consumers scale without affecting write performance.

3. Final Answer:

   Use separate Kafka topics for commands and events, and scale read consumers independently -> Option D

4. Quick Check:

   Separate topics and scale reads independently [OK]

Hint: Separate topics and scale read consumers for high traffic [OK]

Common Mistakes:

• Combining commands and events causing bottlenecks
• Skipping Kafka for reads losing scalability
• Using one consumer group for all messages