Partition concept in Kafka - Mini Project: Build & Apply

Understanding Kafka Partition Concept

📖 Scenario: You are working with Apache Kafka, a system that helps send messages quickly and reliably. Kafka splits messages into parts called partitions. Each partition holds a part of the messages for a topic.Think of a topic as a big box of letters, and partitions as smaller boxes inside it. Each smaller box holds some letters. This helps many people read and write letters at the same time without waiting.

🎯 Goal: You will create a simple Kafka topic with partitions, send messages to specific partitions, and read messages from them. This will help you understand how partitions work in Kafka.

📋 What You'll Learn

Create a Kafka topic named my_topic with exactly 3 partitions.

Create a variable partition_key to decide which partition to send messages to.

Send messages to my_topic using the partition_key to select the partition.

Read and print messages from each partition separately.

💡 Why This Matters

🌍 Real World

Kafka partitions help large systems handle many messages quickly by splitting data into parts. This is like dividing work among many helpers to finish faster.

💼 Career

Understanding Kafka partitions is important for roles in data engineering, backend development, and system architecture where message streaming and real-time data processing are used.

1

Create Kafka topic with 3 partitions

Write the Kafka command to create a topic called my_topic with exactly 3 partitions.

Kafka

# Your code here

Need a hint?

Use the kafka-topics command with --create, specify --topic my_topic, and set --partitions 3.

2

Set partition key variable

In your Kafka producer script, create a variable called partition_key and set it to 1 to send messages to partition 1.

Kafka

# Your code here

Need a hint?

Just write partition_key = 1 in your script.

3

Send messages to chosen partition

Write code to send the message 'Hello Kafka' to my_topic using the partition_key to select the partition.

Kafka

# Your code here

Need a hint?

Use kafka-console-producer with --topic my_topic and send the message Hello Kafka. Use key to control partition.

4

Read and print messages from each partition

Write commands to read and print messages from each partition (0, 1, and 2) of my_topic separately.

Kafka

# Your code here

Need a hint?

Use kafka-console-consumer with --partition option for each partition.