Consumer group concept in Kafka - Step-by-Step Execution

Process Flow - Consumer group concept

Start Consumer Group

↓

Assign Partitions to Consumers

↓

Consumers Read Messages

↓

Commit Offsets

↓

Process Messages in Parallel

↓

New Consumer Joins?

Yes→Rebalance Partitions

↓

Reassign Partitions

↓

Continue Reading

This flow shows how a consumer group starts, partitions are assigned to consumers, messages are read and processed, and how rebalancing happens when consumers join or leave.

Execution Sample

Kafka

ConsumerGroup group = new ConsumerGroup("group1");
Consumer c1 = group.addConsumer("c1");
Consumer c2 = group.addConsumer("c2");
group.assignPartitions(new int[]{0,1,2,3});
c1.read(0);
c2.read(1);

This code creates a consumer group with two consumers and assigns partitions to them for reading messages.

Process Table

Step	Action	Partition Assignment	Consumer Reading	Offset Commit	Notes
1	Create Consumer Group 'group1'	None	None	None	Group initialized, no consumers yet
2	Add Consumer 'c1'	None	None	None	One consumer in group
3	Add Consumer 'c2'	None	None	None	Two consumers in group
4	Assign partitions [0,1,2,3]	c1: [0,2], c2: [1,3]	None	None	Partitions evenly assigned
5	Consumer c1 reads partition 0	c1: [0,2], c2: [1,3]	c1 reads partition 0 messages	Offset updated for partition 0	c1 processes messages from partition 0
6	Consumer c2 reads partition 1	c1: [0,2], c2: [1,3]	c2 reads partition 1 messages	Offset updated for partition 1	c2 processes messages from partition 1
7	New Consumer 'c3' joins	Trigger rebalance	None	None	Partitions reassigned among c1, c2, c3
8	After rebalance	c1: [0], c2: [2], c3: [1,3]	None	None	Consumers get new partitions
9	Consumers read assigned partitions	Same as step 8	Each consumer reads its partitions	Offsets updated	Parallel processing continues
10	Consumer c2 leaves	Trigger rebalance	None	None	Partitions reassigned to c1 and c3
11	After rebalance	c1: [0,2], c3: [1,3]	None	None	Group stabilizes with two consumers
12	Continue reading	Same as step 11	Consumers read partitions	Offsets updated	Normal operation continues

💡 Execution stops when consumers leave or group is closed.

Status Tracker

Variable	Start	After Step 3	After Step 4	After Step 7	After Step 10	Final
Consumer Group	None	group1	group1	group1	group1	group1
Consumers	None	[c1, c2]	[c1, c2]	[c1, c2, c3]	[c1, c3]	[c1, c3]
Partition Assignment	None	None	c1:[0,2], c2:[1,3]	c1:[0], c2:[2], c3:[1,3]	c1:[0,2], c3:[1,3]	c1:[0,2], c3:[1,3]
Offsets Committed	None	None	Updated for partitions 0,1	Updated after rebalance	Updated after rebalance	Updated continuously

Key Moments - 3 Insights

Why do partitions get reassigned when a new consumer joins?

When a new consumer joins (see step 7), Kafka rebalances partitions to distribute load evenly among consumers, ensuring each partition is read by only one consumer in the group.

Can two consumers read the same partition at the same time?

No, as shown in step 4 and 8, each partition is assigned to only one consumer to avoid duplicate processing.

What happens to offsets when consumers leave or join?

Offsets are committed by consumers as they read messages (steps 5 and 6). After rebalancing (steps 7 and 10), consumers continue from the last committed offsets to avoid missing or repeating messages.

Visual Quiz - 3 Questions

Look at the execution table at step 4, which partitions does consumer c1 read?

A [0, 1]

B [1, 3]

C [0, 2]

D [2, 3]

💡 Hint

Check the 'Partition Assignment' column at step 4 in the execution table.

At which step does a rebalance happen due to a new consumer joining?

A Step 7

B Step 5

C Step 10

D Step 12

💡 Hint

Look for 'New Consumer joins' and 'Trigger rebalance' in the execution table.

If consumer c2 leaves at step 10, how does partition assignment change?

A Partitions stay the same

B Partitions reassigned to c1 and c3

C Partitions assigned only to c3

D Partitions assigned to c1 only

💡 Hint

Check 'Partition Assignment' after step 10 in the execution table.

Concept Snapshot

Consumer Group Concept in Kafka:
- Consumers join a group to read partitions.
- Each partition assigned to only one consumer.
- Consumers read messages and commit offsets.
- When consumers join/leave, partitions rebalance.
- Ensures parallel, fault-tolerant message processing.

Full Transcript

This visual execution trace shows how Kafka consumer groups work. First, a consumer group is created with no consumers. Then consumers join the group. Kafka assigns partitions evenly to consumers so each partition is read by only one consumer. Consumers read messages from their assigned partitions and commit offsets to track progress. When a new consumer joins, Kafka triggers a rebalance to redistribute partitions evenly. Similarly, when a consumer leaves, partitions are reassigned to remaining consumers. This process ensures messages are processed in parallel without duplication and that the system adapts to changes in consumer membership.

Practice

1. What is the main purpose of a consumer group in Kafka?

easy

A. To allow multiple consumers to share the work of reading messages

B. To store messages permanently in Kafka

C. To produce messages to Kafka topics

D. To encrypt messages during transmission

Solution

1. Step 1: Understand consumer group role

   A consumer group lets multiple consumers work together to read messages from a topic.

2. Step 2: Compare other options

   Storing messages, producing messages, and encrypting are not roles of consumer groups.

3. Final Answer:

   To allow multiple consumers to share the work of reading messages -> Option A

4. Quick Check:

   Consumer group = share reading work [OK]

Hint: Consumer groups split message reading among consumers [OK]

Common Mistakes:

• Confusing consumer groups with producers
• Thinking consumer groups store messages
• Assuming consumer groups encrypt data

2. Which of the following is the correct way to specify a consumer group ID in Kafka consumer configuration?

easy

A. "groupName" : "my-group"

B. "group.id" : "my-group"

C. "consumer.group" : "my-group"

D. "consumerId" : "my-group"

Solution

1. Step 1: Recall Kafka consumer config keys

   The correct key to set the consumer group is "group.id".

2. Step 2: Check other options

   Other keys like "consumer.group", "groupName", or "consumerId" are incorrect or do not exist.

3. Final Answer:

   "group.id" : "my-group" -> Option B

4. Quick Check:

   Consumer group ID key = "group.id" [OK]

Hint: Remember Kafka uses "group.id" for consumer groups [OK]

Common Mistakes:

• Using incorrect config keys
• Confusing consumer ID with group ID
• Misspelling the key name

3. Given a Kafka topic with 4 partitions and a consumer group with 2 consumers, how many partitions will each consumer read from?

medium

A. Each consumer reads from 2 partitions

B. Each consumer reads from 4 partitions

C. Each consumer reads from 1 partition

D. Each consumer reads from 0 partitions

Solution

1. Step 1: Understand partition assignment

   Partitions are divided among consumers in a group. With 4 partitions and 2 consumers, partitions split evenly.

2. Step 2: Calculate partitions per consumer

   4 partitions / 2 consumers = 2 partitions per consumer.

3. Final Answer:

   Each consumer reads from 2 partitions -> Option A

4. Quick Check:

   Partitions per consumer = total partitions / consumers = 2 [OK]

Hint: Partitions divide evenly among consumers in a group [OK]

Common Mistakes:

• Assuming each consumer reads all partitions
• Thinking partitions are not shared
• Ignoring partition count

4. What is wrong with this Kafka consumer group code snippet?

props.put("group.id", "my-group");
props.put("enable.auto.commit", "false");
consumer.subscribe(Arrays.asList("my-topic"));
consumer.poll(Duration.ofMillis(100));
// Missing commit after poll

medium

A. The poll timeout is too long

B. The group.id is incorrectly set

C. The consumer subscribes to the wrong topic

D. The consumer never commits offsets, risking message reprocessing

Solution

1. Step 1: Analyze auto commit setting

   Auto commit is disabled, so offsets must be committed manually.

2. Step 2: Check code for commit call

   Code calls poll but does not commit offsets, so messages may be reprocessed on restart.

3. Final Answer:

   The consumer never commits offsets, risking message reprocessing -> Option D

4. Quick Check:

   Auto commit false needs manual commit [OK]

Hint: Disable auto commit means manual commit needed [OK]

Common Mistakes:

• Forgetting to commit offsets manually
• Assuming auto commit is enabled by default
• Ignoring poll timeout relevance

5. You have a Kafka topic with 3 partitions and a consumer group with 4 consumers. What happens to the extra consumer?

hard

A. The extra consumer causes an error in the group

B. The extra consumer reads from all partitions

C. The extra consumer remains idle and does not receive messages

D. Kafka creates a new partition for the extra consumer

Solution

1. Step 1: Understand partition to consumer assignment

   Each partition can be assigned to only one consumer in the group.

2. Step 2: Analyze consumer count vs partitions

   With 3 partitions and 4 consumers, only 3 consumers get partitions; the 4th consumer stays idle.

3. Final Answer:

   The extra consumer remains idle and does not receive messages -> Option C

4. Quick Check:

   More consumers than partitions means idle consumers [OK]

Hint: Consumers > partitions means some consumers idle [OK]

Common Mistakes:

• Thinking Kafka creates new partitions automatically
• Assuming extra consumers share partitions
• Believing extra consumers cause errors