Message broker architecture in Kafka - Mini Project: Build & Apply

Message Broker Architecture with Kafka

📖 Scenario: You are setting up a simple message broker system using Kafka to handle messages between different parts of an application. This system will help different services communicate by sending and receiving messages through Kafka topics.

🎯 Goal: Build a basic Kafka message broker setup with a topic and produce and consume messages to understand how Kafka handles message flow.

📋 What You'll Learn

Create a Kafka topic named orders

Set a configuration variable for the number of partitions

Write a producer script to send messages to the orders topic

Write a consumer script to read messages from the orders topic

Print the consumed messages to the console

💡 Why This Matters

🌍 Real World

Kafka is widely used in real applications to handle streams of data between services, such as order processing, logging, and event tracking.

💼 Career

Understanding Kafka basics is essential for roles in DevOps, site reliability engineering, and backend development where message brokers help build scalable systems.

1

Create Kafka topic orders

Use the Kafka command line tool to create a topic called orders with 1 partition and a replication factor of 1.

Kafka

# Your command here to create the topic 'orders'
# Use kafka-topics.sh or kafka-topics.bat depending on your OS

Hint

Use the kafka-topics.sh command with --create and specify the topic name, partitions, and replication factor.

2

Set number of partitions variable

In a shell script, create a variable called PARTITIONS and set it to 3 to configure the number of partitions for the topic.

Kafka

kafka-topics.sh --create --topic orders --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

# Create a variable PARTITIONS and set it to 3
# Your code here

Hint

Use PARTITIONS=3 to set the variable in your shell script.

3

Produce messages to orders topic

Write a Kafka producer command that sends the message 'OrderID: 12345' to the orders topic using the kafka-console-producer.sh tool.

Kafka

kafka-topics.sh --create --topic orders --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
PARTITIONS=3

# Send 'OrderID: 12345' message to 'orders' topic
# Your code here

Hint

Use echo 'OrderID: 12345' | kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092 to send the message.

4

Consume and print messages from orders topic

Use the kafka-console-consumer.sh command to read messages from the orders topic starting from the beginning and print them to the console.

Kafka

kafka-topics.sh --create --topic orders --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
PARTITIONS=3
echo 'OrderID: 12345' | kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092

# Consume messages from 'orders' topic and print
# Your code here

Hint

Use kafka-console-consumer.sh --topic orders --bootstrap-server localhost:9092 --from-beginning --max-messages 1 to read and print the message.

Practice

1. What is the main role of a message broker like Kafka in an application?

easy

A. To create user interfaces for applications

B. To send messages between different parts of an application reliably

C. To run complex calculations on data

D. To store large files permanently

Solution

1. Step 1: Understand the purpose of a message broker

   A message broker helps different parts of an app communicate by sending messages.

2. Step 2: Identify Kafka's role

   Kafka acts as a reliable middleman to send and store messages in order.

3. Final Answer:

   To send messages between different parts of an application reliably -> Option B

4. Quick Check:

   Message broker = reliable message passing [OK]

Hint: Message brokers connect apps by passing messages [OK]

Common Mistakes:

• Thinking Kafka stores files permanently
• Confusing message brokers with UI tools
• Assuming Kafka does data calculations

2. Which of the following is the correct way to describe a Kafka topic?

easy

A. A user who sends messages to Kafka

B. A tool to monitor Kafka servers

C. A place where messages are stored and organized in order

D. A database table for storing user data

Solution

1. Step 1: Define what a Kafka topic is

   A Kafka topic is a category or feed name where messages are stored in order.

2. Step 2: Match the correct description

   Only A place where messages are stored and organized in order correctly describes a topic as a storage place for ordered messages.

3. Final Answer:

   A place where messages are stored and organized in order -> Option C

4. Quick Check:

   Kafka topic = ordered message storage [OK]

Hint: Topic = ordered message storage in Kafka [OK]

Common Mistakes:

• Confusing topic with user or tool
• Thinking topic is a database table
• Mixing topic with monitoring tools

3. Given this Kafka setup: Producer sends messages to topic orders. Consumer reads from orders. What will the consumer receive if the producer sends messages ["order1", "order2"] in order?

medium

A. ["order1", "order2"]

B. No messages because consumer reads from a different topic

C. ["order2", "order1"]

D. An error because messages are unordered

Solution

1. Step 1: Understand message order in Kafka topics

   Kafka keeps messages in the order they are sent within a topic.

2. Step 2: Trace the messages sent and received

   Producer sends ["order1", "order2"] to topic orders. Consumer reads from the same topic, so it receives messages in the same order.

3. Final Answer:

   ["order1", "order2"] -> Option A

4. Quick Check:

   Kafka preserves message order = ["order1", "order2"] [OK]

Hint: Kafka keeps message order per topic [OK]

Common Mistakes:

• Assuming messages arrive reversed
• Thinking consumer reads from different topic
• Believing Kafka does not guarantee order

4. You wrote code where a Kafka consumer tries to read from a topic but gets no messages. What is a likely cause?

medium

A. The consumer is subscribed to the wrong topic name

B. Kafka does not support consumers

C. Messages are deleted immediately after sending

D. Producer sends messages in the wrong format

Solution

1. Step 1: Check consumer subscription

   If the consumer subscribes to a wrong or misspelled topic, it won't receive messages.

2. Step 2: Evaluate other options

   Kafka supports consumers, messages are not deleted immediately, and format issues usually cause errors, not empty reads.

3. Final Answer:

   The consumer is subscribed to the wrong topic name -> Option A

4. Quick Check:

   Wrong topic subscription = no messages [OK]

Hint: Check topic name spelling for consumer subscription [OK]

Common Mistakes:

• Assuming Kafka lacks consumer support
• Thinking messages vanish instantly
• Blaming message format without error

5. You want to ensure messages sent to Kafka are not lost even if a server crashes. Which Kafka feature helps achieve this?

hard

A. Sending messages without keys

B. Using a single broker to simplify management

C. Deleting messages immediately after consumption

D. Replication of topic partitions across multiple brokers

Solution

1. Step 1: Understand message durability in Kafka

   Kafka replicates partitions of topics across multiple brokers to avoid data loss if one fails.

2. Step 2: Evaluate options for reliability

   Single broker risks data loss, deleting messages early loses data, and keys don't affect durability.

3. Final Answer:

   Replication of topic partitions across multiple brokers -> Option D

4. Quick Check:

   Replication = message safety on server crash [OK]

Hint: Use replication to keep messages safe [OK]

Common Mistakes:

• Thinking single broker is safer
• Deleting messages early to save space
• Ignoring replication importance