Partition count strategy in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a partition in Kafka?

A partition is a division of a Kafka topic that stores a subset of the messages. It allows Kafka to scale and distribute data across multiple brokers.

beginner

Why is the number of partitions important in Kafka?

The number of partitions affects parallelism, throughput, and fault tolerance. More partitions allow more consumers to read in parallel but can increase complexity.

intermediate

What happens if you increase the partition count after topic creation?

You can increase partitions, but existing data stays in old partitions. New partitions start empty. This can affect message ordering and consumer logic.

beginner

How does Kafka assign messages to partitions by default?

Kafka uses a partitioner that assigns messages based on the message key's hash. If no key is provided, it uses round-robin to distribute messages evenly.

intermediate

What is a good strategy to decide the number of partitions?

Consider expected throughput, number of consumers, and hardware limits. Start with enough partitions to allow parallelism but not too many to avoid overhead.

What does increasing the number of partitions in a Kafka topic do?

A Reduces the topic size

B Deletes old messages automatically

C Allows more consumers to read in parallel

D Prevents message duplication

More partitions allow more consumers to read messages in parallel, increasing throughput.

If a Kafka message has no key, how is its partition chosen?

A Round-robin distribution across partitions

B Always assigned to partition 0

C Based on the message timestamp

D Randomly assigned to any partition

Kafka uses round-robin to evenly distribute messages without keys across partitions.

Can you decrease the number of partitions in a Kafka topic after creation?

A Yes, anytime without issues

B No, Kafka does not support decreasing partitions

C Only if the topic is empty

D Only with a broker restart

Kafka does not allow decreasing the number of partitions once a topic is created.

What is a risk of having too many partitions in Kafka?

A Increased overhead and complexity

B Consumers cannot connect

C Messages get lost

D Lower parallelism

Too many partitions increase overhead for brokers and consumers, making management harder.

Which factor is NOT important when deciding partition count?

A Expected message throughput

B Broker hardware limits

C Number of consumer instances

D Color of the server hardware

The color of server hardware does not affect partition count decisions.

Explain how Kafka uses partitions to improve message processing.

Think about how dividing data helps multiple consumers work at the same time.

Describe the considerations when choosing the number of partitions for a Kafka topic.

Balance performance needs with system complexity.

Practice

1. What does the partition count in a Kafka topic control?

easy

A. The number of parts a topic is divided into for parallel processing

B. The number of messages a topic can hold

C. The size limit of each message in the topic

D. The number of consumers allowed to read the topic

Solution

1. Step 1: Understand Kafka topic structure

   A Kafka topic is divided into partitions to allow parallel processing of messages.

2. Step 2: Identify what partition count controls

   The partition count sets how many parts the topic is split into, enabling multiple consumers to read in parallel.

3. Final Answer:

   The number of parts a topic is divided into for parallel processing -> Option A

4. Quick Check:

   Partition count = number of topic parts [OK]

Hint: Partition count means how many parts a topic has [OK]

Common Mistakes:

• Confusing partition count with message size
• Thinking partition count limits consumer numbers
• Assuming partition count controls message retention

2. Which of the following is the correct way to set the partition count when creating a Kafka topic using the command line?

easy

A. kafka-topics --create --topic myTopic --partitions 3 --bootstrap-server localhost:9092

B. kafka-topics --create --topic myTopic --num-partitions 3 --bootstrap-server localhost:9092

C. kafka-topics --create --topic myTopic --partition-count 3 --bootstrap-server localhost:9092

D. kafka-topics --create --topic myTopic --part 3 --bootstrap-server localhost:9092

Solution

1. Step 1: Recall Kafka topic creation syntax

   The correct flag to set partitions is --partitions.

2. Step 2: Match the correct command option

   kafka-topics --create --topic myTopic --partitions 3 --bootstrap-server localhost:9092 uses --partitions 3, which is the correct syntax.

3. Final Answer:

   kafka-topics --create --topic myTopic --partitions 3 --bootstrap-server localhost:9092 -> Option A

4. Quick Check:

   Partition flag = --partitions [OK]

Hint: Use --partitions to set partition count [OK]

Common Mistakes:

• Using --num-partitions instead of --partitions
• Using --partition-count which is invalid
• Using --part which is invalid

3. Given a Kafka topic with 4 partitions, what will be the partition number for a message with key hash % 4 = 2?

medium

A. Partition 0

B. Partition 1

C. Partition 2

D. Partition 3

Solution

1. Step 1: Understand partition assignment by key hash

   Kafka assigns messages to partitions by taking the key hash modulo the number of partitions.

2. Step 2: Calculate partition number

   Given key hash % 4 = 2, the message goes to partition 2 (zero-based indexing).

3. Final Answer:

   Partition 2 -> Option C

4. Quick Check:

   Key hash % partitions = partition number [OK]

Hint: Partition = key hash modulo partition count [OK]

Common Mistakes:

• Confusing modulo result with partition index
• Assuming partitions start at 1 instead of 0
• Ignoring key hash in partition calculation

4. You tried to create a Kafka topic with 0 partitions using the command: kafka-topics --create --topic test --num-partitions 0 --bootstrap-server localhost:9092. What is the likely issue?

medium

A. Kafka allows 0 partitions, so topic created with no data storage

B. Error because partition count must be at least 1

C. Topic created with default partition count of 1

D. Command ignored the partition count and created topic anyway

Solution

1. Step 1: Understand partition count constraints

   Kafka requires at least one partition per topic; zero is invalid.

2. Step 2: Identify command behavior

   Setting partitions to 0 causes an error because Kafka cannot create a topic without partitions.

3. Final Answer:

   Error because partition count must be at least 1 -> Option B

4. Quick Check:

   Partition count ≥ 1 required [OK]

Hint: Partition count cannot be zero or negative [OK]

Common Mistakes:

• Assuming Kafka defaults to 1 partition silently
• Thinking zero partitions means no data but topic exists
• Ignoring error messages from Kafka CLI

5. You have a Kafka topic with 5 partitions and want to increase parallel processing. Which is the best strategy?

hard

A. Delete the topic and create a new one with 1 partition

B. Decrease the partition count to 3 to reduce complexity

C. Keep partition count at 5 and add more consumers than partitions

D. Increase the partition count to 10 and rebalance consumers

Solution

1. Step 1: Understand effect of partition count on parallelism

   More partitions allow more consumers to read in parallel, increasing throughput.

2. Step 2: Evaluate options for increasing parallel processing

   Increasing partitions to 10 allows better parallelism; rebalance consumers to use new partitions.

3. Final Answer:

   Increase the partition count to 10 and rebalance consumers -> Option D

4. Quick Check:

   More partitions = better parallel processing [OK]

Hint: More partitions mean more parallel consumers [OK]

Common Mistakes:

• Reducing partitions thinking it improves speed
• Adding more consumers than partitions without increasing partitions
• Deleting topic unnecessarily