Partition concept in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Kafka partition assignment code?

Consider a Kafka consumer group with 3 consumers and a topic with 6 partitions. The partitions are assigned evenly. What partitions will consumer 2 receive?

Kafka

consumers = ['consumer1', 'consumer2', 'consumer3']
partitions = list(range(6))
assignments = {c: [] for c in consumers}
for i, p in enumerate(partitions):
    assignments[consumers[i % len(consumers)]].append(p)
print(assignments['consumer2'])

A [0, 3]

B [2, 5]

C [1, 3, 5]

D [1, 4]

💡 Hint

Partitions are assigned in a round-robin fashion based on the consumer index.

Partitions 0, 3 go to consumer1; 1, 4 go to consumer2; 2, 5 go to consumer3. So consumer2 gets partitions 1 and 4.

🧠 Conceptual

intermediate

How does Kafka decide which partition a message goes to when a key is provided?

In Kafka, when producing a message with a key, how is the partition chosen?

A Kafka hashes the key and assigns the message to the partition based on the hash modulo the number of partitions.

B Kafka always sends messages with keys to partition 0.

C Kafka randomly assigns the message to any partition regardless of the key.

D Kafka assigns messages with keys to partitions in a round-robin manner.

💡 Hint

Think about how consistent partitioning is achieved for messages with the same key.

Kafka uses a hash function on the key and then takes modulo with the number of partitions to ensure messages with the same key go to the same partition.

❓ Predict Output

advanced

What error does this Kafka producer code raise?

This code tries to send a message to a non-existent partition. What error will it raise?

Kafka

from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('my_topic', key=b'key1', value=b'value1', partition=10)
producer.flush()

A kafka.errors.UnknownTopicOrPartitionError

B kafka.errors.KafkaTimeoutError

C ValueError: Partition index out of range

D No error, message sent successfully

💡 Hint

Check what happens if you specify a partition index that does not exist in the topic.

Kafka raises UnknownTopicOrPartitionError when the specified partition does not exist for the topic.

🚀 Application

advanced

How many partitions will be assigned to each consumer in this group?

A Kafka topic has 7 partitions. There are 4 consumers in the same group consuming this topic. How many partitions will each consumer get?

A Consumers 1 to 3 get 1 partition each, consumer 4 gets 4 partitions

B Consumers 1 to 3 get 2 partitions each, consumer 4 gets 1 partition

C Each consumer gets 2 partitions, and 1 partition is assigned to a random consumer

D Each consumer gets exactly 1 partition, and 3 partitions remain unassigned

💡 Hint

Partitions are distributed as evenly as possible among consumers.

7 partitions divided among 4 consumers means 3 consumers get 2 partitions, and 1 consumer gets 1 partition.

🧠 Conceptual

expert

What is the impact of increasing the number of partitions on message ordering in Kafka?

If you increase the number of partitions for a Kafka topic, what happens to the ordering guarantees of messages with the same key?

A Ordering is preserved globally across all partitions regardless of key.

B Ordering is lost because messages with the same key can be sent to different partitions after increasing partitions.

C Ordering is preserved per key because messages with the same key always go to the same partition.

D Ordering is preserved only if the number of partitions is a power of two.

💡 Hint

Think about how Kafka uses keys to assign partitions and maintain order.

Kafka guarantees ordering per partition. Since keys map to partitions via hashing, messages with the same key stay in the same partition even if partitions increase.

Practice

1.

What is the main purpose of partitions in a Kafka topic?

easy

A. To split the topic into smaller parts for better speed and order

B. To encrypt messages for security

C. To store messages permanently on disk

D. To compress messages to save space

Solution

1. Step 1: Understand the role of partitions in Kafka

   Partitions divide a topic into smaller parts to allow parallel processing and maintain message order within each partition.

2. Step 2: Identify the main benefit of partitions

   Partitions improve speed by enabling parallelism and keep order for messages with the same key.

3. Final Answer:

   To split the topic into smaller parts for better speed and order -> Option A

4. Quick Check:

   Partitions = speed and order [OK]

Hint: Partitions split topics to speed up and keep order [OK]

Common Mistakes:

• Confusing partitions with encryption
• Thinking partitions store messages permanently
• Assuming partitions compress data

2.

Which of the following is the correct way to specify the number of partitions when creating a Kafka topic using the command line?

kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092 --partitions ?

easy

A. --num-partitions 3

B. --partition-count 3

C. --partitions 3

D. --partition 3

Solution

1. Step 1: Recall Kafka topic creation syntax

   The correct option to specify partitions is '--partitions' followed by the number.

2. Step 2: Match the correct option

   Among the options, '--partitions 3' is the valid syntax for setting 3 partitions.

3. Final Answer:

   --partitions 3 -> Option C

4. Quick Check:

   Partitions flag = --partitions [OK]

Hint: Use --partitions to set number when creating topic [OK]

Common Mistakes:

• Using --partition-count instead of --partitions
• Using --num-partitions which is invalid
• Using singular --partition flag

3.

Given a Kafka topic with 3 partitions, if messages with keys "key1", "key2", and "key1" are sent in that order, which partition will the second "key1" message go to?

medium

A. Partition 1

B. Same partition as the first "key1" message

C. Partition 0

D. A random partition different from the first "key1" message

Solution

1. Step 1: Understand key-based partitioning

   Kafka uses the message key to decide the partition. Messages with the same key always go to the same partition.

2. Step 2: Apply to the given keys

   Since both messages have key "key1", the second message will go to the same partition as the first "key1" message.

3. Final Answer:

   Same partition as the first "key1" message -> Option B

4. Quick Check:

   Same key = same partition [OK]

Hint: Same keys always go to the same partition [OK]

Common Mistakes:

• Assuming keys distribute randomly each time
• Thinking second message goes to a different partition
• Ignoring key-based partitioning

4.

Consider this Kafka producer code snippet in Java:

ProducerRecord record = new ProducerRecord<>("my-topic", null, "message");
producer.send(record);

What is the issue related to partitioning in this code?

medium

A. The key is null, so messages may be sent to random partitions

B. The topic name is missing

C. The message value is null

D. Producer is not closed after sending

Solution

1. Step 1: Analyze the ProducerRecord parameters

   The key parameter is null, which means Kafka cannot use key-based partitioning.

2. Step 2: Understand partition assignment without a key

   Without a key, Kafka assigns messages to partitions in a round-robin or default way, which can be random.

3. Final Answer:

   The key is null, so messages may be sent to random partitions -> Option A

4. Quick Check:

   Null key = random partition [OK]

Hint: Null key means no fixed partition, messages spread randomly [OK]

Common Mistakes:

• Thinking topic name is missing (it is present)
• Assuming message value is null (it's "message")
• Ignoring partitioning effect of null key

5.

You have a Kafka topic with 4 partitions. You want to ensure that all messages with the same customer ID go to the same partition to keep their order. Which approach should you use?

hard

A. Create 4 separate topics, one per customer

B. Send all messages to partition 0 manually

C. Use a random key for each message

D. Use the customer ID as the message key when producing messages

Solution

1. Step 1: Understand how Kafka assigns partitions

   Kafka uses the message key to decide which partition a message goes to, ensuring order for messages with the same key.

2. Step 2: Apply to customer ID ordering requirement

   Using the customer ID as the key ensures all messages for that customer go to the same partition and keep order.

3. Step 3: Evaluate other options

   Sending all to partition 0 limits scalability; random keys break order; separate topics complicate management.

4. Final Answer:

   Use the customer ID as the message key when producing messages -> Option D

5. Quick Check:

   Key = customer ID for order [OK]

Hint: Use customer ID as key to keep order in one partition [OK]

Common Mistakes:

• Sending all messages to one partition reduces scalability
• Using random keys breaks message order
• Creating multiple topics is unnecessary and complex