Partition assignment in Kafka - Commands & Configuration

Introduction

Kafka splits data into parts called partitions to spread the load. Partition assignment decides which server handles which part, so data is balanced and easy to find.

When you add new servers to your Kafka cluster and want to balance the data load.

When you want to control which server processes which part of your topic for better performance.

When you need to understand how Kafka distributes data to troubleshoot slow consumers.

When you want to manually assign partitions to specific consumers in a consumer group.

When you want to optimize data locality by assigning partitions based on server location.

Commands

This command shows which partitions are assigned to each consumer in the group. It helps you see the current partition assignment.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-consumer-group

Expected Output

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID

my-topic        0          1234            1250            16              consumer-1-12345-67890                        /192.168.1.10   consumer-1

my-topic        1          1200            1250            50              consumer-2-12345-67890                        /192.168.1.11   consumer-2

→

--bootstrap-server - Specifies the Kafka server to connect to.

→

--describe - Shows detailed information about the consumer group.

→

--group - Specifies the consumer group to inspect.

This command increases the number of partitions for the topic. More partitions allow more parallel processing and finer partition assignment.

Terminal

kafka-topics --bootstrap-server localhost:9092 --alter --topic my-topic --partitions 4

Expected Output

Updated partitions for topic my-topic.

→

--alter - Modifies an existing topic.

→

--partitions - Sets the new total number of partitions.

This command reads messages from a specific partition starting at the earliest message. It helps verify data in that partition.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic my-topic --partition 2 --offset earliest

Expected Output

Message 1 from partition 2

Message 2 from partition 2

Message 3 from partition 2

→

--partition - Specifies which partition to read from.

→

--offset - Specifies where to start reading messages.

Key Concept

If you remember nothing else from this pattern, remember: partition assignment controls how Kafka spreads data and workload across servers for balance and speed.

Common Mistakes

Trying to reduce partitions after creation.

Kafka does not support decreasing partitions; it only allows increasing them.

Plan the number of partitions carefully before creating the topic or create a new topic with fewer partitions.

Assuming partitions are evenly assigned without checking consumer group status.

Consumers may not be balanced if some are slow or disconnected, causing uneven load.

Use kafka-consumer-groups --describe to check partition assignment and consumer lag.

Manually assigning partitions without understanding consumer group coordination.

Manual assignment can cause consumers to miss messages or duplicate processing.

Use Kafka's automatic partition assignment unless you have a specific reason to assign manually.

Summary

Use kafka-consumer-groups --describe to see which consumer handles which partition.

Increase partitions with kafka-topics --alter to allow more parallel processing.

Read from a specific partition using kafka-console-consumer with --partition and --offset flags.

Practice

1.

What is the main purpose of partition assignment in Kafka?

easy

A. To encrypt messages before sending

B. To split topic data among consumers to share the workload

C. To compress data for faster transmission

D. To create new topics automatically

Solution

1. Step 1: Understand partition assignment role

   Partition assignment divides topic partitions among consumers to distribute processing.

2. Step 2: Compare options

   Only To split topic data among consumers to share the workload describes splitting data among consumers; others describe unrelated tasks.

3. Final Answer:

   To split topic data among consumers to share the workload -> Option B

4. Quick Check:

   Partition assignment = workload sharing [OK]

Hint: Partition assignment shares topic data among consumers [OK]

Common Mistakes:

• Confusing partition assignment with encryption
• Thinking it compresses data
• Assuming it creates topics

2.

Which of the following is a valid Kafka partition assignment strategy?

easy

A. RoundRobin

B. HashMap

C. QuickSort

D. BinarySearch

Solution

1. Step 1: Recall Kafka partition assignment strategies

   Kafka supports strategies like Range, RoundRobin, and Sticky.

2. Step 2: Identify valid strategy

   RoundRobin is a known Kafka strategy; others are unrelated algorithms.

3. Final Answer:

   RoundRobin -> Option A

4. Quick Check:

   RoundRobin = valid strategy [OK]

Hint: Remember Kafka uses RoundRobin, Range, Sticky strategies [OK]

Common Mistakes:

• Confusing general algorithms with Kafka strategies
• Choosing unrelated computer science terms
• Ignoring Kafka-specific terms

3.

Given a topic with 4 partitions and 2 consumers using the Range strategy, how are partitions assigned?

Partitions: 0, 1, 2, 3
Consumers: C1, C2
Strategy: Range

medium

A. Partitions are assigned randomly

B. C1 gets partitions 0 and 2; C2 gets partitions 1 and 3

C. C1 gets all partitions; C2 gets none

D. C1 gets partitions 0 and 1; C2 gets partitions 2 and 3

Solution

1. Step 1: Understand Range strategy behavior

   Range assigns consecutive partitions to each consumer in order.

2. Step 2: Apply Range to 4 partitions and 2 consumers

   Partitions 0 and 1 go to C1; partitions 2 and 3 go to C2.

3. Final Answer:

   C1 gets partitions 0 and 1; C2 gets partitions 2 and 3 -> Option D

4. Quick Check:

   Range = consecutive partition blocks [OK]

Hint: Range assigns consecutive partitions to consumers [OK]

Common Mistakes:

• Mixing RoundRobin with Range strategy
• Assuming random assignment
• Giving all partitions to one consumer

4.

Identify the error in this partition assignment code snippet using the Sticky strategy:

assignor = StickyAssignor()
assignor.assign(partitions=[0,1,2], consumers=["C1"])

medium

A. Consumers list must be integers

B. Partitions list should be empty

C. StickyAssignor requires at least two consumers for assignment

D. StickyAssignor does not support partition assignment

Solution

1. Step 1: Understand StickyAssignor requirements

   Sticky strategy balances partitions among multiple consumers to minimize movement.

2. Step 2: Check given consumers list

   Only one consumer is provided, so StickyAssignor cannot balance partitions.

3. Final Answer:

   StickyAssignor requires at least two consumers for assignment -> Option C

4. Quick Check:

   Sticky needs multiple consumers [OK]

Hint: Sticky strategy needs multiple consumers to balance partitions [OK]

Common Mistakes:

• Assuming single consumer works with Sticky
• Confusing consumer data types
• Thinking Sticky doesn't assign partitions

5.

You have 5 consumers and a topic with 10 partitions. You want to maintain processing order per partition and balance load evenly. Which partition assignment strategy should you choose?

hard

A. Sticky, because it balances partitions while minimizing reassignment

B. RoundRobin, because it distributes partitions evenly across consumers

C. Range, because it assigns consecutive partitions to consumers

D. Random, because it assigns partitions unpredictably

Solution

1. Step 1: Analyze requirements

   Need to keep processing order per partition and balance load evenly.

2. Step 2: Compare strategies

   Range assigns consecutive partitions but may imbalance load; RoundRobin balances evenly but can cause frequent partition moves; Sticky balances load and minimizes partition reassignment, preserving order.

3. Step 3: Choose best fit

   Sticky strategy best meets both order and load balance requirements.

4. Final Answer:

   Sticky, because it balances partitions while minimizing reassignment -> Option A

5. Quick Check:

   Sticky = balanced load + order preservation [OK]

Hint: Sticky balances load and keeps partition order [OK]

Common Mistakes:

• Choosing Range despite load imbalance
• Picking RoundRobin ignoring partition order
• Selecting random assignment which is unpredictable