Group coordinator in Kafka - Step-by-Step Execution

Process Flow - Group coordinator

Start Consumer Group

↓

Join Group Coordinator

↓

Coordinator Assigns Partitions

↓

Consumer Polls Messages

↓

Heartbeat to Coordinator

↓

Rebalance if Needed

↓

Continue Polling or Leave Group

The group coordinator manages consumer group membership, assigns partitions, and handles rebalances to ensure smooth message consumption.

Execution Sample

Kafka

consumer = KafkaConsumer('topic', group_id='group1')
for message in consumer:
    print(message.value)

This code creates a consumer in a group and continuously polls messages from assigned partitions.

Process Table

Step	Action	Coordinator Response	Consumer State	Output
1	Consumer starts and sends JoinGroup request	Coordinator registers consumer in group	Joining group	None
2	Coordinator assigns partitions to consumer	Sends assignment to consumer	Assigned partitions	None
3	Consumer polls messages from assigned partitions	No direct response	Consuming messages	Message values printed
4	Consumer sends heartbeat to coordinator	Acknowledges consumer is alive	Maintains membership	None
5	Another consumer joins group	Triggers rebalance	Rebalancing	None
6	Coordinator reassigns partitions	Sends new assignments	Partitions reassigned	None
7	Consumer continues polling with new assignments	No direct response	Consuming messages	Message values printed
8	Consumer leaves group	Removes consumer from group	Left group	None
9	Coordinator triggers final rebalance	Reassigns partitions to remaining consumers	Updated assignments	None

💡 Consumer leaves group or application stops, ending message consumption.

Status Tracker

Variable	Start	After Step 2	After Step 5	After Step 6	Final
Consumer State	Not started	Assigned partitions	Rebalancing	Partitions reassigned	Left group
Coordinator Group Members	Empty	1 consumer	2 consumers	2 consumers with new assignments	1 consumer

Key Moments - 3 Insights

Why does the consumer enter a rebalancing state after a new consumer joins?

When a new consumer joins, the coordinator must redistribute partitions to balance load, causing rebalancing as shown in step 5 of the execution_table.

What happens if the consumer stops sending heartbeats?

The coordinator assumes the consumer is dead and triggers a rebalance to assign its partitions to others, maintaining group health (implied between steps 4 and 5).

How does the coordinator know which partitions to assign?

The coordinator uses the group membership and partition info to assign partitions evenly among consumers, as seen in step 2 and step 6.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the consumer receive its partition assignment?

A Step 1

B Step 3

C Step 2

D Step 5

💡 Hint

Check the 'Coordinator Response' column for 'Sends assignment to consumer' in the execution_table.

According to variable_tracker, what is the consumer state after step 5?

A Assigned partitions

B Rebalancing

C Consuming messages

D Left group

💡 Hint

Look at the 'Consumer State' row under 'After Step 5' in variable_tracker.

If the consumer stops sending heartbeats, what will the coordinator most likely do next?

A Trigger a rebalance to remove the consumer

B Ignore and continue as normal

C Assign more partitions to the consumer

D Send a message to the consumer

💡 Hint

Refer to key_moments about heartbeat and rebalance behavior.

Concept Snapshot

Group Coordinator manages consumer group membership.
Consumers join group and get partition assignments.
Consumers send heartbeats to stay in group.
Coordinator triggers rebalance on membership changes.
Partitions are reassigned to balance load.
Consumers poll messages from assigned partitions.

Full Transcript

The group coordinator in Kafka manages the consumer group lifecycle. When a consumer starts, it sends a join request to the coordinator. The coordinator registers the consumer and assigns partitions to it. The consumer then polls messages from these partitions. To maintain membership, the consumer sends heartbeats to the coordinator. If a new consumer joins or one leaves, the coordinator triggers a rebalance to redistribute partitions evenly. This ensures all messages are consumed efficiently. If a consumer stops sending heartbeats, the coordinator assumes it is dead and reassigns its partitions to others. This process keeps the group healthy and balanced.