Partition assignment in Kafka - Step-by-Step Execution

Process Flow - Partition assignment

Consumer Group Joins

↓

Coordinator Assigns Partitions

↓

Each Consumer Receives Partition List

↓

Consumers Consume Assigned Partitions

↓

Rebalance on Join/Leave/Failure

↓

Repeat Assignment Process

When consumers join a group, Kafka assigns partitions to them. Each consumer gets a list of partitions to read from. If consumers join or leave, Kafka rebalances and reassigns partitions.

Execution Sample

Kafka

ConsumerGroup: [C1, C2]
Partitions: [P0, P1, P2, P3]
Assignment: C1 -> [P0, P1]
            C2 -> [P2, P3]

Shows two consumers in a group assigned partitions evenly.

Process Table

Step	Event	Consumers	Partitions	Assignment	Notes
1	Group starts with C1 and C2	C1, C2	P0, P1, P2, P3	None	No assignment yet
2	Coordinator assigns partitions	C1, C2	P0, P1, P2, P3	C1 -> P0, P1; C2 -> P2, P3	Partitions split evenly
3	C3 joins group	C1, C2, C3	P0, P1, P2, P3	Rebalance triggered	Assignment invalidated
4	Coordinator reassigns partitions	C1, C2, C3	P0, P1, P2, P3	C1 -> P0; C2 -> P1, P2; C3 -> P3	Partitions reassigned to 3 consumers
5	C2 leaves group	C1, C3	P0, P1, P2, P3	Rebalance triggered	Assignment invalidated
6	Coordinator reassigns partitions	C1, C3	P0, P1, P2, P3	C1 -> P0, P1; C3 -> P2, P3	Partitions reassigned to 2 consumers
7	Execution ends	C1, C3	P0, P1, P2, P3	Final assignment as above	No more changes

💡 Execution stops after final assignment with stable consumer group

Status Tracker

Variable	Start	After Step 2	After Step 4	After Step 6	Final
Consumers	[]	[C1, C2]	[C1, C2, C3]	[C1, C3]	[C1, C3]
Partitions	[P0, P1, P2, P3]	[P0, P1, P2, P3]	[P0, P1, P2, P3]	[P0, P1, P2, P3]	[P0, P1, P2, P3]
Assignment	None	C1->[P0,P1], C2->[P2,P3]	C1->[P0], C2->[P1,P2], C3->[P3]	C1->[P0,P1], C3->[P2,P3]	C1->[P0,P1], C3->[P2,P3]

Key Moments - 3 Insights

Why does Kafka reassign partitions when a new consumer joins?

Because the assignment must be balanced among all consumers, so when a new consumer joins, Kafka triggers a rebalance to redistribute partitions fairly (see execution_table step 3 and 4).

Can two consumers read from the same partition at the same time?

No, Kafka assigns each partition to only one consumer in the group to avoid duplicate processing (see execution_table assignments at each step).

What happens if a consumer leaves the group unexpectedly?

Kafka detects the leave and triggers a rebalance to assign the partitions of the left consumer to the remaining consumers (see execution_table step 5 and 6).

Visual Quiz - 3 Questions

Look at the execution_table at step 4, which partitions does consumer C2 have?

A P0 and P1

B P1 and P2

C P2 and P3

D P3 only

💡 Hint

Check the 'Assignment' column at step 4 in execution_table

At which step does the consumer group have three consumers?

A Step 5

B Step 2

C Step 3

D Step 6

💡 Hint

Look at the 'Consumers' column in execution_table

If consumer C3 leaves after step 6, what will happen next?

A Rebalance triggered to assign all partitions to C1

B Partitions get duplicated across consumers

C No rebalance, assignments stay the same

D Kafka shuts down the group

💡 Hint

Refer to key_moments about what happens when a consumer leaves (see execution_table step 5 and 6)

Concept Snapshot

Partition assignment in Kafka:
- Consumers join a group
- Kafka coordinator assigns partitions evenly
- Each partition assigned to only one consumer
- On join/leave, rebalance triggers
- Reassignment ensures balanced load
- Prevents duplicate consumption

Full Transcript

Partition assignment in Kafka happens when consumers join a group. The Kafka coordinator assigns partitions to each consumer so that each partition is read by only one consumer. When a new consumer joins or one leaves, Kafka triggers a rebalance to redistribute partitions evenly. This process ensures that the workload is balanced and no partition is read by multiple consumers at the same time. The execution table shows how assignments change step-by-step as consumers join and leave the group.

Practice

1.

What is the main purpose of partition assignment in Kafka?

easy

A. To encrypt messages before sending

B. To split topic data among consumers to share the workload

C. To compress data for faster transmission

D. To create new topics automatically

Solution

1. Step 1: Understand partition assignment role

   Partition assignment divides topic partitions among consumers to distribute processing.

2. Step 2: Compare options

   Only To split topic data among consumers to share the workload describes splitting data among consumers; others describe unrelated tasks.

3. Final Answer:

   To split topic data among consumers to share the workload -> Option B

4. Quick Check:

   Partition assignment = workload sharing [OK]

Hint: Partition assignment shares topic data among consumers [OK]

Common Mistakes:

• Confusing partition assignment with encryption
• Thinking it compresses data
• Assuming it creates topics

2.

Which of the following is a valid Kafka partition assignment strategy?

easy

A. RoundRobin

B. HashMap

C. QuickSort

D. BinarySearch

Solution

1. Step 1: Recall Kafka partition assignment strategies

   Kafka supports strategies like Range, RoundRobin, and Sticky.

2. Step 2: Identify valid strategy

   RoundRobin is a known Kafka strategy; others are unrelated algorithms.

3. Final Answer:

   RoundRobin -> Option A

4. Quick Check:

   RoundRobin = valid strategy [OK]

Hint: Remember Kafka uses RoundRobin, Range, Sticky strategies [OK]

Common Mistakes:

• Confusing general algorithms with Kafka strategies
• Choosing unrelated computer science terms
• Ignoring Kafka-specific terms

3.

Given a topic with 4 partitions and 2 consumers using the Range strategy, how are partitions assigned?

Partitions: 0, 1, 2, 3
Consumers: C1, C2
Strategy: Range

medium

A. Partitions are assigned randomly

B. C1 gets partitions 0 and 2; C2 gets partitions 1 and 3

C. C1 gets all partitions; C2 gets none

D. C1 gets partitions 0 and 1; C2 gets partitions 2 and 3

Solution

1. Step 1: Understand Range strategy behavior

   Range assigns consecutive partitions to each consumer in order.

2. Step 2: Apply Range to 4 partitions and 2 consumers

   Partitions 0 and 1 go to C1; partitions 2 and 3 go to C2.

3. Final Answer:

   C1 gets partitions 0 and 1; C2 gets partitions 2 and 3 -> Option D

4. Quick Check:

   Range = consecutive partition blocks [OK]

Hint: Range assigns consecutive partitions to consumers [OK]

Common Mistakes:

• Mixing RoundRobin with Range strategy
• Assuming random assignment
• Giving all partitions to one consumer

4.

Identify the error in this partition assignment code snippet using the Sticky strategy:

assignor = StickyAssignor()
assignor.assign(partitions=[0,1,2], consumers=["C1"])

medium

A. Consumers list must be integers

B. Partitions list should be empty

C. StickyAssignor requires at least two consumers for assignment

D. StickyAssignor does not support partition assignment

Solution

1. Step 1: Understand StickyAssignor requirements

   Sticky strategy balances partitions among multiple consumers to minimize movement.

2. Step 2: Check given consumers list

   Only one consumer is provided, so StickyAssignor cannot balance partitions.

3. Final Answer:

   StickyAssignor requires at least two consumers for assignment -> Option C

4. Quick Check:

   Sticky needs multiple consumers [OK]

Hint: Sticky strategy needs multiple consumers to balance partitions [OK]

Common Mistakes:

• Assuming single consumer works with Sticky
• Confusing consumer data types
• Thinking Sticky doesn't assign partitions

5.

You have 5 consumers and a topic with 10 partitions. You want to maintain processing order per partition and balance load evenly. Which partition assignment strategy should you choose?

hard

A. Sticky, because it balances partitions while minimizing reassignment

B. RoundRobin, because it distributes partitions evenly across consumers

C. Range, because it assigns consecutive partitions to consumers

D. Random, because it assigns partitions unpredictably

Solution

1. Step 1: Analyze requirements

   Need to keep processing order per partition and balance load evenly.

2. Step 2: Compare strategies

   Range assigns consecutive partitions but may imbalance load; RoundRobin balances evenly but can cause frequent partition moves; Sticky balances load and minimizes partition reassignment, preserving order.

3. Step 3: Choose best fit

   Sticky strategy best meets both order and load balance requirements.

4. Final Answer:

   Sticky, because it balances partitions while minimizing reassignment -> Option A

5. Quick Check:

   Sticky = balanced load + order preservation [OK]

Hint: Sticky balances load and keeps partition order [OK]

Common Mistakes:

• Choosing Range despite load imbalance
• Picking RoundRobin ignoring partition order
• Selecting random assignment which is unpredictable