Partition count strategy in Kafka - Mini Project: Build & Apply

Partition Count Strategy

📖 Scenario: You are setting up a Kafka topic for a small online store. You want to decide how many partitions the topic should have to handle the store's order messages efficiently.

🎯 Goal: Build a simple program that sets the number of partitions for a Kafka topic based on the number of expected orders per second.

📋 What You'll Learn

Create a variable with the expected number of orders per second

Create a variable for the maximum orders per partition per second

Calculate the number of partitions needed using integer division and rounding up

Print the number of partitions

💡 Why This Matters

🌍 Real World

Kafka topics use partitions to spread data load. Knowing how many partitions to create helps keep your system fast and reliable.

💼 Career

Understanding partition count strategy is important for roles like data engineers and backend developers working with Kafka.

1

Set expected orders per second

Create a variable called orders_per_second and set it to 1200.

Kafka

# Create a variable called orders_per_second and set it to 1200
# Your code here

Hint

Think about how many orders your store expects to receive every second.

2

Set max orders per partition

Create a variable called max_orders_per_partition and set it to 300.

Kafka

orders_per_second = 1200
# Create a variable called max_orders_per_partition and set it to 300
# Your code here

Hint

This is the maximum number of orders one partition can handle per second.

3

Calculate number of partitions

Create a variable called partitions that calculates the number of partitions needed by dividing orders_per_second by max_orders_per_partition and rounding up to the nearest whole number.

Kafka

orders_per_second = 1200
max_orders_per_partition = 300
# Calculate partitions needed and assign to partitions variable
# Your code here

Hint

Use integer division and add max_orders_per_partition - 1 before dividing to round up.

4

Print the number of partitions

Write a print statement to display the value of partitions.

Kafka

orders_per_second = 1200
max_orders_per_partition = 300
partitions = (orders_per_second + max_orders_per_partition - 1) // max_orders_per_partition
# Print the number of partitions
# Your code here

Hint

Use print(partitions) to show the result.

Practice

1. What does the partition count in a Kafka topic control?

easy

A. The number of parts a topic is divided into for parallel processing

B. The number of messages a topic can hold

C. The size limit of each message in the topic

D. The number of consumers allowed to read the topic

Solution

1. Step 1: Understand Kafka topic structure

   A Kafka topic is divided into partitions to allow parallel processing of messages.

2. Step 2: Identify what partition count controls

   The partition count sets how many parts the topic is split into, enabling multiple consumers to read in parallel.

3. Final Answer:

   The number of parts a topic is divided into for parallel processing -> Option A

4. Quick Check:

   Partition count = number of topic parts [OK]

Hint: Partition count means how many parts a topic has [OK]

Common Mistakes:

• Confusing partition count with message size
• Thinking partition count limits consumer numbers
• Assuming partition count controls message retention

2. Which of the following is the correct way to set the partition count when creating a Kafka topic using the command line?

easy

A. kafka-topics --create --topic myTopic --partitions 3 --bootstrap-server localhost:9092

B. kafka-topics --create --topic myTopic --num-partitions 3 --bootstrap-server localhost:9092

C. kafka-topics --create --topic myTopic --partition-count 3 --bootstrap-server localhost:9092

D. kafka-topics --create --topic myTopic --part 3 --bootstrap-server localhost:9092

Solution

1. Step 1: Recall Kafka topic creation syntax

   The correct flag to set partitions is --partitions.

2. Step 2: Match the correct command option

   kafka-topics --create --topic myTopic --partitions 3 --bootstrap-server localhost:9092 uses --partitions 3, which is the correct syntax.

3. Final Answer:

   kafka-topics --create --topic myTopic --partitions 3 --bootstrap-server localhost:9092 -> Option A

4. Quick Check:

   Partition flag = --partitions [OK]

Hint: Use --partitions to set partition count [OK]

Common Mistakes:

• Using --num-partitions instead of --partitions
• Using --partition-count which is invalid
• Using --part which is invalid

3. Given a Kafka topic with 4 partitions, what will be the partition number for a message with key hash % 4 = 2?

medium

A. Partition 0

B. Partition 1

C. Partition 2

D. Partition 3

Solution

1. Step 1: Understand partition assignment by key hash

   Kafka assigns messages to partitions by taking the key hash modulo the number of partitions.

2. Step 2: Calculate partition number

   Given key hash % 4 = 2, the message goes to partition 2 (zero-based indexing).

3. Final Answer:

   Partition 2 -> Option C

4. Quick Check:

   Key hash % partitions = partition number [OK]

Hint: Partition = key hash modulo partition count [OK]

Common Mistakes:

• Confusing modulo result with partition index
• Assuming partitions start at 1 instead of 0
• Ignoring key hash in partition calculation

4. You tried to create a Kafka topic with 0 partitions using the command: kafka-topics --create --topic test --num-partitions 0 --bootstrap-server localhost:9092. What is the likely issue?

medium

A. Kafka allows 0 partitions, so topic created with no data storage

B. Error because partition count must be at least 1

C. Topic created with default partition count of 1

D. Command ignored the partition count and created topic anyway

Solution

1. Step 1: Understand partition count constraints

   Kafka requires at least one partition per topic; zero is invalid.

2. Step 2: Identify command behavior

   Setting partitions to 0 causes an error because Kafka cannot create a topic without partitions.

3. Final Answer:

   Error because partition count must be at least 1 -> Option B

4. Quick Check:

   Partition count ≥ 1 required [OK]

Hint: Partition count cannot be zero or negative [OK]

Common Mistakes:

• Assuming Kafka defaults to 1 partition silently
• Thinking zero partitions means no data but topic exists
• Ignoring error messages from Kafka CLI

5. You have a Kafka topic with 5 partitions and want to increase parallel processing. Which is the best strategy?

hard

A. Delete the topic and create a new one with 1 partition

B. Decrease the partition count to 3 to reduce complexity

C. Keep partition count at 5 and add more consumers than partitions

D. Increase the partition count to 10 and rebalance consumers

Solution

1. Step 1: Understand effect of partition count on parallelism

   More partitions allow more consumers to read in parallel, increasing throughput.

2. Step 2: Evaluate options for increasing parallel processing

   Increasing partitions to 10 allows better parallelism; rebalance consumers to use new partitions.

3. Final Answer:

   Increase the partition count to 10 and rebalance consumers -> Option D

4. Quick Check:

   More partitions = better parallel processing [OK]

Hint: More partitions mean more parallel consumers [OK]

Common Mistakes:

• Reducing partitions thinking it improves speed
• Adding more consumers than partitions without increasing partitions
• Deleting topic unnecessarily