Consumer group concept in Kafka - Step-by-Step Execution

Process Flow - Consumer group concept

Start Consumer Group

↓

Assign Partitions to Consumers

↓

Consumers Read Messages

↓

Commit Offsets

↓

Process Messages in Parallel

↓

New Consumer Joins?

Yes→Rebalance Partitions

↓

Reassign Partitions

↓

Continue Reading

This flow shows how a consumer group starts, partitions are assigned to consumers, messages are read and processed, and how rebalancing happens when consumers join or leave.

Execution Sample

Kafka

ConsumerGroup group = new ConsumerGroup("group1");
Consumer c1 = group.addConsumer("c1");
Consumer c2 = group.addConsumer("c2");
group.assignPartitions(new int[]{0,1,2,3});
c1.read(0);
c2.read(1);

This code creates a consumer group with two consumers and assigns partitions to them for reading messages.

Process Table

Step	Action	Partition Assignment	Consumer Reading	Offset Commit	Notes
1	Create Consumer Group 'group1'	None	None	None	Group initialized, no consumers yet
2	Add Consumer 'c1'	None	None	None	One consumer in group
3	Add Consumer 'c2'	None	None	None	Two consumers in group
4	Assign partitions [0,1,2,3]	c1: [0,2], c2: [1,3]	None	None	Partitions evenly assigned
5	Consumer c1 reads partition 0	c1: [0,2], c2: [1,3]	c1 reads partition 0 messages	Offset updated for partition 0	c1 processes messages from partition 0
6	Consumer c2 reads partition 1	c1: [0,2], c2: [1,3]	c2 reads partition 1 messages	Offset updated for partition 1	c2 processes messages from partition 1
7	New Consumer 'c3' joins	Trigger rebalance	None	None	Partitions reassigned among c1, c2, c3
8	After rebalance	c1: [0], c2: [2], c3: [1,3]	None	None	Consumers get new partitions
9	Consumers read assigned partitions	Same as step 8	Each consumer reads its partitions	Offsets updated	Parallel processing continues
10	Consumer c2 leaves	Trigger rebalance	None	None	Partitions reassigned to c1 and c3
11	After rebalance	c1: [0,2], c3: [1,3]	None	None	Group stabilizes with two consumers
12	Continue reading	Same as step 11	Consumers read partitions	Offsets updated	Normal operation continues

💡 Execution stops when consumers leave or group is closed.

Status Tracker

Variable	Start	After Step 3	After Step 4	After Step 7	After Step 10	Final
Consumer Group	None	group1	group1	group1	group1	group1
Consumers	None	[c1, c2]	[c1, c2]	[c1, c2, c3]	[c1, c3]	[c1, c3]
Partition Assignment	None	None	c1:[0,2], c2:[1,3]	c1:[0], c2:[2], c3:[1,3]	c1:[0,2], c3:[1,3]	c1:[0,2], c3:[1,3]
Offsets Committed	None	None	Updated for partitions 0,1	Updated after rebalance	Updated after rebalance	Updated continuously

Key Moments - 3 Insights

Why do partitions get reassigned when a new consumer joins?

When a new consumer joins (see step 7), Kafka rebalances partitions to distribute load evenly among consumers, ensuring each partition is read by only one consumer in the group.

Can two consumers read the same partition at the same time?

No, as shown in step 4 and 8, each partition is assigned to only one consumer to avoid duplicate processing.

What happens to offsets when consumers leave or join?

Offsets are committed by consumers as they read messages (steps 5 and 6). After rebalancing (steps 7 and 10), consumers continue from the last committed offsets to avoid missing or repeating messages.

Visual Quiz - 3 Questions

Look at the execution table at step 4, which partitions does consumer c1 read?

A [0, 1]

B [1, 3]

C [0, 2]

D [2, 3]

💡 Hint

Check the 'Partition Assignment' column at step 4 in the execution table.

At which step does a rebalance happen due to a new consumer joining?

A Step 7

B Step 5

C Step 10

D Step 12

💡 Hint

Look for 'New Consumer joins' and 'Trigger rebalance' in the execution table.

If consumer c2 leaves at step 10, how does partition assignment change?

A Partitions stay the same

B Partitions reassigned to c1 and c3

C Partitions assigned only to c3

D Partitions assigned to c1 only

💡 Hint

Check 'Partition Assignment' after step 10 in the execution table.

Concept Snapshot

Consumer Group Concept in Kafka:
- Consumers join a group to read partitions.
- Each partition assigned to only one consumer.
- Consumers read messages and commit offsets.
- When consumers join/leave, partitions rebalance.
- Ensures parallel, fault-tolerant message processing.

Full Transcript

This visual execution trace shows how Kafka consumer groups work. First, a consumer group is created with no consumers. Then consumers join the group. Kafka assigns partitions evenly to consumers so each partition is read by only one consumer. Consumers read messages from their assigned partitions and commit offsets to track progress. When a new consumer joins, Kafka triggers a rebalance to redistribute partitions evenly. Similarly, when a consumer leaves, partitions are reassigned to remaining consumers. This process ensures messages are processed in parallel without duplication and that the system adapts to changes in consumer membership.