Partition key and routing in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the partition chosen for this Kafka message?

Given a Kafka topic with 3 partitions and the following producer code, what partition will the message be sent to?

Kafka

from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers='localhost:9092')

# Key is a string 'user42'
key = b'user42'
value = b'Hello Kafka'

# Send message
future = producer.send('my_topic', key=key, value=value)
record_metadata = future.get(timeout=10)

print(record_metadata.partition)

A Partition 0

B Partition 3

C Partition 2

D Partition 1

💡 Hint

Kafka uses the hash of the key modulo the number of partitions to decide the partition.

Kafka hashes the key 'user42' and then takes modulo 3 (number of partitions). The hash of 'user42' modulo 3 results in partition 1.

🧠 Conceptual

intermediate

Why is using a partition key important in Kafka?

Select the main reason why using a partition key is important when producing messages to Kafka.

A It ensures messages with the same key go to the same partition, preserving order.

B It encrypts the message content for security.

C It compresses the message to save bandwidth.

D It automatically retries sending messages on failure.

💡 Hint

Think about message ordering guarantees in Kafka.

Kafka guarantees order only within a partition. Using a key ensures all messages with that key go to the same partition, preserving order.

🔧 Debug

advanced

Why does this Kafka producer code cause uneven message distribution?

This Kafka producer sends messages with keys that are always None. What is the likely effect on partition distribution?

Kafka

from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers='localhost:9092')

for i in range(10):
    producer.send('my_topic', key=None, value=f'Message {i}'.encode())

producer.flush()

A Messages are evenly distributed across all partitions.

B All messages go to the same partition, causing uneven load.

C Messages cause a runtime error because key cannot be None.

D Messages are dropped silently without sending.

💡 Hint

What happens when no key is provided in Kafka producer?

When key is None, Kafka uses a round-robin or default partitioner which may send all messages to the same partition if the producer is single-threaded or due to batching.

📝 Syntax

advanced

Which Kafka producer code snippet correctly sets a partition key?

Choose the code snippet that correctly sets a string key for Kafka message production.

A producer.send('topic', key='user1', value=b'data')

B producer.send('topic', key=123, value=b'data')

C producer.send('topic', key=b'user1', value=b'data')

D producer.send('topic', key=None, value=b'data')

💡 Hint

Kafka keys must be bytes, not strings or integers.

Kafka producer expects the key as bytes. Passing a string without encoding causes an error. Passing an integer is invalid. None means no key.

🚀 Application

expert

How many partitions will contain messages after this code runs?

A Kafka topic has 4 partitions. The producer sends messages with keys cycling through ['A', 'B', 'C']. How many partitions will have messages after sending 12 messages?

Kafka

keys = [b'A', b'B', b'C']
for i in range(12):
    key = keys[i % 3]
    producer.send('my_topic', key=key, value=b'data')
producer.flush()

A 3 partitions

B 4 partitions

C 1 partition

D 0 partitions

💡 Hint

Messages with the same key always go to the same partition.

There are 3 distinct keys cycling, so messages go to 3 partitions. The 4th partition gets no messages.

Practice

1. What is the main purpose of using a partition key in Kafka?

easy

A. To increase the message size limit

B. To encrypt the message content

C. To ensure related messages go to the same partition

D. To compress the message before sending

Solution

1. Step 1: Understand partition key role

   The partition key is used to decide which partition a message goes to in Kafka.

2. Step 2: Connect key to message ordering

   Using the same key ensures related messages stay in the same partition, preserving order.

3. Final Answer:

   To ensure related messages go to the same partition -> Option C

4. Quick Check:

   Partition key controls message placement = D [OK]

Hint: Partition key groups related messages together [OK]

Common Mistakes:

• Thinking partition key encrypts messages
• Confusing partition key with compression
• Believing partition key increases message size

2. Which of the following is the correct way to specify a partition key when producing a Kafka message in Java?

easy

A. producer.send(new ProducerRecord<>(topic, value));

B. producer.send(new ProducerRecord<>(value, key));

C. producer.send(new ProducerRecord<>(key, topic, value));

D. producer.send(new ProducerRecord<>(topic, key, value));

Solution

1. Step 1: Recall ProducerRecord constructor

   The correct constructor order is (topic, key, value) to specify partition key.

2. Step 2: Match options to constructor

   producer.send(new ProducerRecord<>(topic, key, value)); matches this order exactly, others mix parameters incorrectly.

3. Final Answer:

   producer.send(new ProducerRecord<>(topic, key, value)); -> Option D

4. Quick Check:

   Topic, key, value order = B [OK]

Hint: Remember ProducerRecord(topic, key, value) order [OK]

Common Mistakes:

• Swapping key and value positions
• Omitting the key parameter
• Placing key before topic incorrectly

3. Given a Kafka topic with 3 partitions, what will happen if you send messages with the same partition key "user123" repeatedly?

medium

A. All messages will go to the same partition

B. Messages will be evenly distributed across all partitions

C. Messages will be sent to partition 0 only

D. Messages will be randomly assigned to any partition

Solution

1. Step 1: Understand partition key hashing

   Kafka hashes the partition key to decide the partition number.

2. Step 2: Same key means same hash and partition

   Repeated messages with "user123" key will always hash to the same partition.

3. Final Answer:

   All messages will go to the same partition -> Option A

4. Quick Check:

   Same key -> same partition = C [OK]

Hint: Same key always hashes to same partition [OK]

Common Mistakes:

• Assuming keys distribute evenly without hashing
• Thinking messages go to partition 0 by default
• Believing messages are randomly assigned despite keys

4. You wrote this code to send messages with keys but all messages end up in different partitions. What is the likely mistake?

producer.send(new ProducerRecord<>(topic, null, value));

medium

A. The topic name is missing

B. The key is null, so messages are distributed randomly

C. The value is null, causing errors

D. ProducerRecord constructor is used incorrectly

Solution

1. Step 1: Check the key parameter

   The key is set to null, so Kafka cannot use it for partitioning.

2. Step 2: Understand default partitioning behavior

   Without a key, Kafka distributes messages randomly across partitions.

3. Final Answer:

   The key is null, so messages are distributed randomly -> Option B

4. Quick Check:

   Null key -> random partitions = A [OK]

Hint: Null key means random partition assignment [OK]

Common Mistakes:

• Assuming null key sends to partition 0
• Confusing missing topic with null key
• Thinking null value causes partitioning issues

5. You want to ensure that messages related to the same customer ID always go to the same partition, but your customer IDs sometimes are null or empty strings. How should you handle partition keys to maintain message order for valid IDs and avoid errors?

hard

A. Use the customer ID as key only if it is non-empty; otherwise, assign a default fixed key

B. Always send messages with null keys to let Kafka distribute randomly

C. Convert null or empty IDs to random UUIDs as keys

D. Ignore keys and rely on Kafka's default partitioning

Solution

1. Step 1: Identify the problem with null or empty keys

   Null or empty keys cause Kafka to distribute messages randomly, breaking order.

2. Step 2: Use a default fixed key for missing IDs

   Assigning a fixed key for missing IDs groups those messages in one partition, preserving order for them.

3. Step 3: Keep valid customer IDs as keys

   Valid IDs remain keys to ensure their messages go to the same partition.

4. Final Answer:

   Use the customer ID as key only if it is non-empty; otherwise, assign a default fixed key -> Option A

5. Quick Check:

   Non-empty key or default fixed key = A [OK]

Hint: Use default key for missing IDs to keep order [OK]

Common Mistakes:

• Sending null keys causing random partitioning
• Using random UUIDs breaking message grouping
• Ignoring keys and losing message order