CQRS pattern in Kafka - Mini Project: Build & Apply

Implementing CQRS Pattern with Kafka

📖 Scenario: You are building a simple order management system. You want to separate the commands (actions that change data) from queries (actions that read data) using the CQRS pattern. Kafka will be used to handle the communication between the command side and the query side.

🎯 Goal: Build a basic CQRS setup where commands to create orders are sent to a Kafka topic, and the query side consumes these commands to update a read model.

📋 What You'll Learn

Create a Kafka topic named orders-commands to receive order creation commands.

Create a Kafka topic named orders-queries to send updated order read models.

Produce a command message to orders-commands with order details.

Consume the command message from orders-commands and update the read model.

Produce the updated read model message to orders-queries.

Print the final read model message.

💡 Why This Matters

🌍 Real World

CQRS pattern is used in systems where separating write and read operations improves scalability and performance. Kafka helps by providing a reliable messaging backbone.

💼 Career

Understanding CQRS with Kafka is valuable for roles in backend development, system architecture, and DevOps where event-driven and scalable systems are built.

1

Create Kafka topics for commands and queries

Create two Kafka topics named orders-commands and orders-queries using the Kafka CLI commands exactly as shown.

Kafka

# Your code here
# Create Kafka topics orders-commands and orders-queries

Hint

Use kafka-topics --create command with --topic, --bootstrap-server, --partitions, and --replication-factor options.

2

Produce an order creation command message

Use the Kafka console producer to send a JSON message with order details {"orderId": "123", "product": "Book", "quantity": 2} to the orders-commands topic.

Kafka

kafka-topics --create --topic orders-commands --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
kafka-topics --create --topic orders-queries --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
# Your code here
# Produce order creation command JSON message to orders-commands topic

Hint

Use kafka-console-producer with --topic orders-commands and send the JSON message.

3

Consume command and produce updated read model

Consume the message from orders-commands topic using Kafka console consumer, then produce an updated read model JSON message {"orderId": "123", "status": "Created", "product": "Book", "quantity": 2} to the orders-queries topic using Kafka console producer.

Kafka

kafka-topics --create --topic orders-commands --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
kafka-topics --create --topic orders-queries --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
kafka-console-producer --topic orders-commands --bootstrap-server localhost:9092 <<EOF
{"orderId": "123", "product": "Book", "quantity": 2}
EOF
# Your code here
# Consume from orders-commands and produce updated read model to orders-queries

Hint

Use kafka-console-consumer to read one message from orders-commands, then use kafka-console-producer to send the updated read model JSON to orders-queries.

4

Display the updated read model message

Consume the updated read model message from the orders-queries topic and print it to the console.

Kafka

kafka-topics --create --topic orders-commands --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
kafka-topics --create --topic orders-queries --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
kafka-console-producer --topic orders-commands --bootstrap-server localhost:9092 <<EOF
{"orderId": "123", "product": "Book", "quantity": 2}
EOF
kafka-console-consumer --topic orders-commands --from-beginning --bootstrap-server localhost:9092 --max-messages 1
kafka-console-producer --topic orders-queries --bootstrap-server localhost:9092 <<EOF
{"orderId": "123", "status": "Created", "product": "Book", "quantity": 2}
EOF
# Your code here
# Consume and print the updated read model from orders-queries topic

Hint

Use kafka-console-consumer with --topic orders-queries and --max-messages 1 to print the updated read model.

Practice

1. What is the main idea behind the CQRS pattern in Kafka-based systems?

easy

A. Separating commands (writes) and queries (reads) into different parts

B. Using a single topic for both commands and queries

C. Storing all data in one database without separation

D. Ignoring events and focusing only on commands

Solution

1. Step 1: Understand CQRS basics

   CQRS stands for Command Query Responsibility Segregation, which means separating write and read operations.

2. Step 2: Apply to Kafka context

   In Kafka, commands and queries are handled separately, often using different topics for clarity and scalability.

3. Final Answer:

   Separating commands (writes) and queries (reads) into different parts -> Option A

4. Quick Check:

   CQRS = Separate commands and queries [OK]

Hint: Remember CQRS splits writes and reads separately [OK]

Common Mistakes:

• Thinking commands and queries share the same topic
• Confusing CQRS with event sourcing
• Assuming CQRS means no events are used

2. Which Kafka topic setup correctly follows the CQRS pattern?

easy

A. One topic for all commands, events, and queries

B. One topic for commands and a separate topic for events

C. Multiple topics for queries only

D. No topics, just direct database writes

Solution

1. Step 1: Recall CQRS topic separation

   CQRS uses separate topics for commands (writes) and events (results of commands) to keep communication clear.

2. Step 2: Evaluate options

   One topic for commands and a separate topic for events correctly separates commands and events into different topics, matching CQRS principles.

3. Final Answer:

   One topic for commands and a separate topic for events -> Option B

4. Quick Check:

   Separate topics for commands and events = One topic for commands and a separate topic for events [OK]

Hint: Separate commands and events into different topics [OK]

Common Mistakes:

• Mixing commands and queries in one topic
• Ignoring event topics
• Using no Kafka topics at all

3. Given this Kafka setup for CQRS:

commands_topic = ['CreateOrder', 'UpdateOrder']
events_topic = ['OrderCreated', 'OrderUpdated']

Which topic should a read service subscribe to for query data?

medium

A. commands_topic

B. Both commands_topic and events_topic

C. events_topic

D. Neither, it reads directly from the database

Solution

1. Step 1: Identify read service role

   Read services in CQRS listen to events that represent state changes, not commands which are requests.

2. Step 2: Match topic to read service

   Events_topic contains 'OrderCreated' and 'OrderUpdated' which reflect changes; read service subscribes here.

3. Final Answer:

   events_topic -> Option C

4. Quick Check:

   Read service subscribes to events_topic [OK]

Hint: Reads listen to events, not commands [OK]

Common Mistakes:

• Subscribing read service to commands_topic
• Subscribing to both topics unnecessarily
• Assuming reads bypass Kafka

4. You have this Kafka CQRS code snippet:

producer.send('commands', 'CreateOrder')
consumer.subscribe(['commands', 'events'])

What is the main problem here?

medium

A. Consumer should not subscribe to both 'commands' and 'events' topics

B. Producer should send to 'events' topic, not 'commands'

C. Producer must send JSON, not strings

D. Consumer must subscribe only to 'commands' topic

Solution

1. Step 1: Understand CQRS topic roles

   Commands topic is for writes; events topic is for read updates. Consumers usually subscribe to one role.

2. Step 2: Analyze consumer subscription

   Subscribing to both commands and events mixes responsibilities and can cause confusion or errors.

3. Final Answer:

   Consumer should not subscribe to both 'commands' and 'events' topics -> Option A

4. Quick Check:

   Consumer subscribes to one topic type only [OK]

Hint: Consumers should subscribe to either commands or events, not both [OK]

Common Mistakes:

• Thinking producer must send events on commands topic
• Assuming consumer can handle both topics safely
• Ignoring message format requirements

5. How can you design a Kafka-based CQRS system to handle high read traffic without slowing down writes?

hard

A. Use only one consumer group for both commands and events

B. Combine commands and events in one topic to reduce overhead

C. Write all data directly to the database and skip Kafka for reads

D. Use separate Kafka topics for commands and events, and scale read consumers independently

Solution

1. Step 1: Apply CQRS scalability principle

   Separating commands and events allows independent scaling of write and read sides.

2. Step 2: Design Kafka topics and consumers

   Using separate topics for commands and events lets read consumers scale without affecting write performance.

3. Final Answer:

   Use separate Kafka topics for commands and events, and scale read consumers independently -> Option D

4. Quick Check:

   Separate topics and scale reads independently [OK]

Hint: Separate topics and scale read consumers for high traffic [OK]

Common Mistakes:

• Combining commands and events causing bottlenecks
• Skipping Kafka for reads losing scalability
• Using one consumer group for all messages