Partition assignment in Kafka - Step-by-Step Execution

Process Flow - Partition assignment

Consumer Group Joins

↓

Coordinator Assigns Partitions

↓

Each Consumer Receives Partition List

↓

Consumers Consume Assigned Partitions

↓

Rebalance on Join/Leave/Failure

↓

Repeat Assignment Process

When consumers join a group, Kafka assigns partitions to them. Each consumer gets a list of partitions to read from. If consumers join or leave, Kafka rebalances and reassigns partitions.

Execution Sample

Kafka

ConsumerGroup: [C1, C2]
Partitions: [P0, P1, P2, P3]
Assignment: C1 -> [P0, P1]
            C2 -> [P2, P3]

Shows two consumers in a group assigned partitions evenly.

Process Table

Step	Event	Consumers	Partitions	Assignment	Notes
1	Group starts with C1 and C2	C1, C2	P0, P1, P2, P3	None	No assignment yet
2	Coordinator assigns partitions	C1, C2	P0, P1, P2, P3	C1 -> P0, P1; C2 -> P2, P3	Partitions split evenly
3	C3 joins group	C1, C2, C3	P0, P1, P2, P3	Rebalance triggered	Assignment invalidated
4	Coordinator reassigns partitions	C1, C2, C3	P0, P1, P2, P3	C1 -> P0; C2 -> P1, P2; C3 -> P3	Partitions reassigned to 3 consumers
5	C2 leaves group	C1, C3	P0, P1, P2, P3	Rebalance triggered	Assignment invalidated
6	Coordinator reassigns partitions	C1, C3	P0, P1, P2, P3	C1 -> P0, P1; C3 -> P2, P3	Partitions reassigned to 2 consumers
7	Execution ends	C1, C3	P0, P1, P2, P3	Final assignment as above	No more changes

💡 Execution stops after final assignment with stable consumer group

Status Tracker

Variable	Start	After Step 2	After Step 4	After Step 6	Final
Consumers	[]	[C1, C2]	[C1, C2, C3]	[C1, C3]	[C1, C3]
Partitions	[P0, P1, P2, P3]	[P0, P1, P2, P3]	[P0, P1, P2, P3]	[P0, P1, P2, P3]	[P0, P1, P2, P3]
Assignment	None	C1->[P0,P1], C2->[P2,P3]	C1->[P0], C2->[P1,P2], C3->[P3]	C1->[P0,P1], C3->[P2,P3]	C1->[P0,P1], C3->[P2,P3]

Key Moments - 3 Insights

Why does Kafka reassign partitions when a new consumer joins?

Because the assignment must be balanced among all consumers, so when a new consumer joins, Kafka triggers a rebalance to redistribute partitions fairly (see execution_table step 3 and 4).

Can two consumers read from the same partition at the same time?

No, Kafka assigns each partition to only one consumer in the group to avoid duplicate processing (see execution_table assignments at each step).

What happens if a consumer leaves the group unexpectedly?

Kafka detects the leave and triggers a rebalance to assign the partitions of the left consumer to the remaining consumers (see execution_table step 5 and 6).

Visual Quiz - 3 Questions

Look at the execution_table at step 4, which partitions does consumer C2 have?

A P0 and P1

B P1 and P2

C P2 and P3

D P3 only

💡 Hint

Check the 'Assignment' column at step 4 in execution_table

At which step does the consumer group have three consumers?

A Step 5

B Step 2

C Step 3

D Step 6

💡 Hint

Look at the 'Consumers' column in execution_table

If consumer C3 leaves after step 6, what will happen next?

A Rebalance triggered to assign all partitions to C1

B Partitions get duplicated across consumers

C No rebalance, assignments stay the same

D Kafka shuts down the group

💡 Hint

Refer to key_moments about what happens when a consumer leaves (see execution_table step 5 and 6)

Concept Snapshot

Partition assignment in Kafka:
- Consumers join a group
- Kafka coordinator assigns partitions evenly
- Each partition assigned to only one consumer
- On join/leave, rebalance triggers
- Reassignment ensures balanced load
- Prevents duplicate consumption

Full Transcript

Partition assignment in Kafka happens when consumers join a group. The Kafka coordinator assigns partitions to each consumer so that each partition is read by only one consumer. When a new consumer joins or one leaves, Kafka triggers a rebalance to redistribute partitions evenly. This process ensures that the workload is balanced and no partition is read by multiple consumers at the same time. The execution table shows how assignments change step-by-step as consumers join and leave the group.