Consumer group concept in Kafka - Mini Project: Build & Apply

Consumer group concept

📖 Scenario: You are working with Apache Kafka to process messages efficiently. Kafka uses consumer groups to allow multiple consumers to share the work of reading from topics.Imagine you have a topic with several partitions, and you want to create a consumer group to read messages in parallel.

🎯 Goal: Build a simple Kafka consumer group setup where multiple consumers join the same group and consume messages from a topic.

📋 What You'll Learn

Create a Kafka topic named test-topic with 3 partitions

Create a consumer group named my-group

Write code to create 3 consumers that join my-group

Each consumer should subscribe to test-topic and print received messages with its consumer id

💡 Why This Matters

🌍 Real World

Kafka consumer groups are used in real applications to scale message processing by distributing partitions among multiple consumers.

💼 Career

Understanding consumer groups is essential for roles involving data engineering, backend development, and real-time data processing.

1

Create Kafka topic with 3 partitions

Use the Kafka command line tool to create a topic named test-topic with exactly 3 partitions and a replication factor of 1.

Kafka

# Your command here to create the topic

Need a hint?

Use kafka-topics --create with the right flags for topic name, partitions, and replication factor.

2

Set consumer group id variable

In your consumer code, create a variable called group_id and set it to the string 'my-group'.

Kafka

kafka-topics --create --topic test-topic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
# Create a variable called group_id and set it to 'my-group'
# Your code here

Need a hint?

Just assign the string 'my-group' to a variable named group_id.

3

Create 3 consumers joining the group

Write code to create 3 Kafka consumers named consumer1, consumer2, and consumer3. Each should use the group_id variable as their group id and subscribe to 'test-topic'.

Kafka

kafka-topics --create --topic test-topic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
group_id = 'my-group'
# Create 3 consumers consumer1, consumer2, consumer3 with group_id and subscribe to 'test-topic'
# Your code here

Need a hint?

Use KafkaConsumer from the kafka Python package. Pass 'test-topic', group_id=group_id, and bootstrap_servers=['localhost:9092'].

4

Print messages consumed by each consumer

Write code to consume one message from each consumer (consumer1, consumer2, consumer3) and print the message value prefixed by the consumer name like Consumer1 received: <message>. Use next() on the consumer iterator to get one message.

Kafka

kafka-topics --create --topic test-topic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
group_id = 'my-group'

from kafka import KafkaConsumer

consumer1 = KafkaConsumer('test-topic', group_id=group_id, bootstrap_servers=['localhost:9092'])
consumer2 = KafkaConsumer('test-topic', group_id=group_id, bootstrap_servers=['localhost:9092'])
consumer3 = KafkaConsumer('test-topic', group_id=group_id, bootstrap_servers=['localhost:9092'])

# Consume one message from each consumer and print with consumer name
# Your code here

Need a hint?

Use next(consumer) to get one message. Decode the message value from bytes to string with .decode('utf-8'). Print with the consumer name prefix.