Partition assignment in Kafka - Commands & Configuration

Introduction

Kafka splits data into parts called partitions to spread the load. Partition assignment decides which server handles which part, so data is balanced and easy to find.

When you add new servers to your Kafka cluster and want to balance the data load.

When you want to control which server processes which part of your topic for better performance.

When you need to understand how Kafka distributes data to troubleshoot slow consumers.

When you want to manually assign partitions to specific consumers in a consumer group.

When you want to optimize data locality by assigning partitions based on server location.

Commands

This command shows which partitions are assigned to each consumer in the group. It helps you see the current partition assignment.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-consumer-group

Expected Output

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID

my-topic        0          1234            1250            16              consumer-1-12345-67890                        /192.168.1.10   consumer-1

my-topic        1          1200            1250            50              consumer-2-12345-67890                        /192.168.1.11   consumer-2

→

--bootstrap-server - Specifies the Kafka server to connect to.

→

--describe - Shows detailed information about the consumer group.

→

--group - Specifies the consumer group to inspect.

This command increases the number of partitions for the topic. More partitions allow more parallel processing and finer partition assignment.

Terminal

kafka-topics --bootstrap-server localhost:9092 --alter --topic my-topic --partitions 4

Expected Output

Updated partitions for topic my-topic.

→

--alter - Modifies an existing topic.

→

--partitions - Sets the new total number of partitions.

This command reads messages from a specific partition starting at the earliest message. It helps verify data in that partition.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic my-topic --partition 2 --offset earliest

Expected Output

Message 1 from partition 2

Message 2 from partition 2

Message 3 from partition 2

→

--partition - Specifies which partition to read from.

→

--offset - Specifies where to start reading messages.

Key Concept

If you remember nothing else from this pattern, remember: partition assignment controls how Kafka spreads data and workload across servers for balance and speed.

Common Mistakes

Trying to reduce partitions after creation.

Kafka does not support decreasing partitions; it only allows increasing them.

Plan the number of partitions carefully before creating the topic or create a new topic with fewer partitions.

Assuming partitions are evenly assigned without checking consumer group status.

Consumers may not be balanced if some are slow or disconnected, causing uneven load.

Use kafka-consumer-groups --describe to check partition assignment and consumer lag.

Manually assigning partitions without understanding consumer group coordination.

Manual assignment can cause consumers to miss messages or duplicate processing.

Use Kafka's automatic partition assignment unless you have a specific reason to assign manually.

Summary

Use kafka-consumer-groups --describe to see which consumer handles which partition.

Increase partitions with kafka-topics --alter to allow more parallel processing.

Read from a specific partition using kafka-console-consumer with --partition and --offset flags.