Message broker architecture in Kafka - Mini Project: Build & Apply

Message Broker Architecture with Kafka

📖 Scenario: You are setting up a simple message broker system using Kafka to handle messages between different parts of an application. This system will help different services communicate by sending and receiving messages through Kafka topics.

🎯 Goal: Build a basic Kafka message broker setup with a topic and produce and consume messages to understand how Kafka handles message flow.

📋 What You'll Learn

Create a Kafka topic named orders

Set a configuration variable for the number of partitions

Write a producer script to send messages to the orders topic

Write a consumer script to read messages from the orders topic

Print the consumed messages to the console

💡 Why This Matters

🌍 Real World

Kafka is widely used in real applications to handle streams of data between services, such as order processing, logging, and event tracking.

💼 Career

Understanding Kafka basics is essential for roles in DevOps, site reliability engineering, and backend development where message brokers help build scalable systems.

1

Create Kafka topic orders

Use the Kafka command line tool to create a topic called orders with 1 partition and a replication factor of 1.

Kafka

# Your command here to create the topic 'orders'
# Use kafka-topics.sh or kafka-topics.bat depending on your OS

Need a hint?

Use the kafka-topics.sh command with --create and specify the topic name, partitions, and replication factor.

2

Set number of partitions variable

In a shell script, create a variable called PARTITIONS and set it to 3 to configure the number of partitions for the topic.

Kafka

kafka-topics.sh --create --topic orders --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

# Create a variable PARTITIONS and set it to 3
# Your code here

Need a hint?

Use PARTITIONS=3 to set the variable in your shell script.

3

Produce messages to orders topic

Write a Kafka producer command that sends the message 'OrderID: 12345' to the orders topic using the kafka-console-producer.sh tool.

Kafka

kafka-topics.sh --create --topic orders --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
PARTITIONS=3

# Send 'OrderID: 12345' message to 'orders' topic
# Your code here

Need a hint?

Use echo 'OrderID: 12345' | kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092 to send the message.

4

Consume and print messages from orders topic

Use the kafka-console-consumer.sh command to read messages from the orders topic starting from the beginning and print them to the console.

Kafka

kafka-topics.sh --create --topic orders --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
PARTITIONS=3
echo 'OrderID: 12345' | kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092

# Consume messages from 'orders' topic and print
# Your code here

Need a hint?

Use kafka-console-consumer.sh --topic orders --bootstrap-server localhost:9092 --from-beginning --max-messages 1 to read and print the message.