Consumer group concept in Kafka - Mini Project: Build & Apply

Consumer group concept

📖 Scenario: You are working with Apache Kafka to process messages efficiently. Kafka uses consumer groups to allow multiple consumers to share the work of reading from topics.Imagine you have a topic with several partitions, and you want to create a consumer group to read messages in parallel.

🎯 Goal: Build a simple Kafka consumer group setup where multiple consumers join the same group and consume messages from a topic.

📋 What You'll Learn

Create a Kafka topic named test-topic with 3 partitions

Create a consumer group named my-group

Write code to create 3 consumers that join my-group

Each consumer should subscribe to test-topic and print received messages with its consumer id

💡 Why This Matters

🌍 Real World

Kafka consumer groups are used in real applications to scale message processing by distributing partitions among multiple consumers.

💼 Career

Understanding consumer groups is essential for roles involving data engineering, backend development, and real-time data processing.

1

Create Kafka topic with 3 partitions

Use the Kafka command line tool to create a topic named test-topic with exactly 3 partitions and a replication factor of 1.

Kafka

# Your command here to create the topic

Hint

Use kafka-topics --create with the right flags for topic name, partitions, and replication factor.

2

Set consumer group id variable

In your consumer code, create a variable called group_id and set it to the string 'my-group'.

Kafka

kafka-topics --create --topic test-topic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
# Create a variable called group_id and set it to 'my-group'
# Your code here

Hint

Just assign the string 'my-group' to a variable named group_id.

3

Create 3 consumers joining the group

Write code to create 3 Kafka consumers named consumer1, consumer2, and consumer3. Each should use the group_id variable as their group id and subscribe to 'test-topic'.

Kafka

kafka-topics --create --topic test-topic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
group_id = 'my-group'
# Create 3 consumers consumer1, consumer2, consumer3 with group_id and subscribe to 'test-topic'
# Your code here

Hint

Use KafkaConsumer from the kafka Python package. Pass 'test-topic', group_id=group_id, and bootstrap_servers=['localhost:9092'].

4

Print messages consumed by each consumer

Write code to consume one message from each consumer (consumer1, consumer2, consumer3) and print the message value prefixed by the consumer name like Consumer1 received: <message>. Use next() on the consumer iterator to get one message.

Kafka

kafka-topics --create --topic test-topic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
group_id = 'my-group'

from kafka import KafkaConsumer

consumer1 = KafkaConsumer('test-topic', group_id=group_id, bootstrap_servers=['localhost:9092'])
consumer2 = KafkaConsumer('test-topic', group_id=group_id, bootstrap_servers=['localhost:9092'])
consumer3 = KafkaConsumer('test-topic', group_id=group_id, bootstrap_servers=['localhost:9092'])

# Consume one message from each consumer and print with consumer name
# Your code here

Hint

Use next(consumer) to get one message. Decode the message value from bytes to string with .decode('utf-8'). Print with the consumer name prefix.

Practice

1. What is the main purpose of a consumer group in Kafka?

easy

A. To allow multiple consumers to share the work of reading messages

B. To store messages permanently in Kafka

C. To produce messages to Kafka topics

D. To encrypt messages during transmission

Solution

1. Step 1: Understand consumer group role

   A consumer group lets multiple consumers work together to read messages from a topic.

2. Step 2: Compare other options

   Storing messages, producing messages, and encrypting are not roles of consumer groups.

3. Final Answer:

   To allow multiple consumers to share the work of reading messages -> Option A

4. Quick Check:

   Consumer group = share reading work [OK]

Hint: Consumer groups split message reading among consumers [OK]

Common Mistakes:

• Confusing consumer groups with producers
• Thinking consumer groups store messages
• Assuming consumer groups encrypt data

2. Which of the following is the correct way to specify a consumer group ID in Kafka consumer configuration?

easy

A. "groupName" : "my-group"

B. "group.id" : "my-group"

C. "consumer.group" : "my-group"

D. "consumerId" : "my-group"

Solution

1. Step 1: Recall Kafka consumer config keys

   The correct key to set the consumer group is "group.id".

2. Step 2: Check other options

   Other keys like "consumer.group", "groupName", or "consumerId" are incorrect or do not exist.

3. Final Answer:

   "group.id" : "my-group" -> Option B

4. Quick Check:

   Consumer group ID key = "group.id" [OK]

Hint: Remember Kafka uses "group.id" for consumer groups [OK]

Common Mistakes:

• Using incorrect config keys
• Confusing consumer ID with group ID
• Misspelling the key name

3. Given a Kafka topic with 4 partitions and a consumer group with 2 consumers, how many partitions will each consumer read from?

medium

A. Each consumer reads from 2 partitions

B. Each consumer reads from 4 partitions

C. Each consumer reads from 1 partition

D. Each consumer reads from 0 partitions

Solution

1. Step 1: Understand partition assignment

   Partitions are divided among consumers in a group. With 4 partitions and 2 consumers, partitions split evenly.

2. Step 2: Calculate partitions per consumer

   4 partitions / 2 consumers = 2 partitions per consumer.

3. Final Answer:

   Each consumer reads from 2 partitions -> Option A

4. Quick Check:

   Partitions per consumer = total partitions / consumers = 2 [OK]

Hint: Partitions divide evenly among consumers in a group [OK]

Common Mistakes:

• Assuming each consumer reads all partitions
• Thinking partitions are not shared
• Ignoring partition count

4. What is wrong with this Kafka consumer group code snippet?

props.put("group.id", "my-group");
props.put("enable.auto.commit", "false");
consumer.subscribe(Arrays.asList("my-topic"));
consumer.poll(Duration.ofMillis(100));
// Missing commit after poll

medium

A. The poll timeout is too long

B. The group.id is incorrectly set

C. The consumer subscribes to the wrong topic

D. The consumer never commits offsets, risking message reprocessing

Solution

1. Step 1: Analyze auto commit setting

   Auto commit is disabled, so offsets must be committed manually.

2. Step 2: Check code for commit call

   Code calls poll but does not commit offsets, so messages may be reprocessed on restart.

3. Final Answer:

   The consumer never commits offsets, risking message reprocessing -> Option D

4. Quick Check:

   Auto commit false needs manual commit [OK]

Hint: Disable auto commit means manual commit needed [OK]

Common Mistakes:

• Forgetting to commit offsets manually
• Assuming auto commit is enabled by default
• Ignoring poll timeout relevance

5. You have a Kafka topic with 3 partitions and a consumer group with 4 consumers. What happens to the extra consumer?

hard

A. The extra consumer causes an error in the group

B. The extra consumer reads from all partitions

C. The extra consumer remains idle and does not receive messages

D. Kafka creates a new partition for the extra consumer

Solution

1. Step 1: Understand partition to consumer assignment

   Each partition can be assigned to only one consumer in the group.

2. Step 2: Analyze consumer count vs partitions

   With 3 partitions and 4 consumers, only 3 consumers get partitions; the 4th consumer stays idle.

3. Final Answer:

   The extra consumer remains idle and does not receive messages -> Option C

4. Quick Check:

   More consumers than partitions means idle consumers [OK]

Hint: Consumers > partitions means some consumers idle [OK]

Common Mistakes:

• Thinking Kafka creates new partitions automatically
• Assuming extra consumers share partitions
• Believing extra consumers cause errors