Resource planning and capacity in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Partition assignment output in Kafka consumer group

What is the output of the following Kafka consumer group partition assignment code snippet?

Kafka

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'topic1',
    group_id='group1',
    bootstrap_servers=['localhost:9092'],
    enable_auto_commit=False
)

partitions = consumer.partitions_for_topic('topic1')
print(sorted(partitions))

A None

B [0, 1, 2]

C set()

D ['partition0', 'partition1', 'partition2']

💡 Hint

Check what partitions_for_topic returns and how to convert it to a list.

The method partitions_for_topic returns a set of partition numbers. Sorting it and printing shows a list of partition numbers like [0, 1, 2].

❓ Predict Output

intermediate

Calculating consumer lag from offsets

Given the following code to calculate consumer lag, what is the value of `lag` after execution?

Kafka

end_offsets = {0: 150, 1: 200}
committed_offsets = {0: 140, 1: 180}
lag = sum(end_offsets[p] - committed_offsets[p] for p in end_offsets)
print(lag)

A 50

B None

C 20

D 30

💡 Hint

Calculate the difference for each partition and sum them.

Partition 0 lag is 150 - 140 = 10, partition 1 lag is 200 - 180 = 20, total lag is 10 + 20 = 30.

🔧 Debug

advanced

Identify the error in Kafka producer throughput calculation

What error will the following code raise when calculating throughput?

Kafka

messages_sent = 1000
elapsed_time = 0
throughput = messages_sent / elapsed_time
print(f'Throughput: {throughput} msgs/sec')

A ZeroDivisionError

B TypeError

C NameError

D No error, prints 'Throughput: 1000 msgs/sec'

💡 Hint

Check the value of elapsed_time before division.

Dividing by zero causes a ZeroDivisionError in Python.

❓ Predict Output

advanced

Result of consumer assignment after rebalance

What will be the output of the following code snippet simulating consumer assignment after rebalance?

Kafka

consumer_assignment = {0: 'consumer1', 1: 'consumer2', 2: 'consumer1'}
from collections import Counter
count = Counter(consumer_assignment.values())
print(count['consumer1'])

A 2

B 1

C 3

D KeyError

💡 Hint

Count how many partitions are assigned to 'consumer1'.

'consumer1' is assigned partitions 0 and 2, so count is 2.

🧠 Conceptual

expert

Maximum throughput calculation with partition and consumer limits

Given a Kafka topic with 12 partitions and 4 consumers in a group, each consumer can process 500 messages per second. What is the maximum total throughput achievable assuming perfect load balancing?

A 6000 messages per second

B 3000 messages per second

C 2000 messages per second

D 4000 messages per second

💡 Hint

Each consumer processes 500 msgs/sec. Total throughput is consumers * per consumer rate.

With 4 consumers each processing 500 msgs/sec, total throughput is 4 * 500 = 2000 msgs/sec. But since there are 12 partitions, and each consumer can only process partitions assigned, the max throughput is limited by consumers. So total is 4 * 500 = 2000 msgs/sec.

Practice

1. What is the main goal of resource planning in Kafka?

easy

A. To reduce the number of topics in Kafka

B. To increase the size of Kafka messages

C. To limit the number of users accessing Kafka

D. To ensure Kafka can handle the expected data load smoothly

Solution

1. Step 1: Understand resource planning purpose

   Resource planning is about preparing Kafka to manage data efficiently without delays or failures.

2. Step 2: Identify the main goal

   The main goal is to handle the expected data load smoothly, avoiding bottlenecks or crashes.

3. Final Answer:

   To ensure Kafka can handle the expected data load smoothly -> Option D

4. Quick Check:

   Resource planning = handle data load smoothly [OK]

Hint: Resource planning means handling data load well [OK]

Common Mistakes:

• Confusing resource planning with limiting users
• Thinking it reduces topics or message sizes
• Ignoring the data load aspect

2. Which of the following is the correct way to increase Kafka capacity?

easy

A. Add more brokers to the Kafka cluster

B. Decrease the number of partitions per topic

C. Reduce the hardware resources of brokers

D. Limit the number of consumers

Solution

1. Step 1: Review capacity factors in Kafka

   Capacity depends on brokers, partitions, and hardware resources.

2. Step 2: Identify correct capacity increase method

   Adding more brokers increases cluster capacity by distributing load.

3. Final Answer:

   Add more brokers to the Kafka cluster -> Option A

4. Quick Check:

   More brokers = more capacity [OK]

Hint: Add brokers to increase capacity [OK]

Common Mistakes:

• Thinking decreasing partitions increases capacity
• Reducing hardware lowers capacity
• Limiting consumers does not increase capacity

3. Given a Kafka cluster with 3 brokers and 6 partitions per topic, what happens if you increase partitions to 12 without adding brokers?

medium

A. The number of consumers must be reduced

B. The cluster automatically adds brokers to handle the load

C. The load per broker increases, possibly causing performance issues

D. Kafka will reject the partition increase request

Solution

1. Step 1: Understand partition increase effect

   Increasing partitions without adding brokers means more load per broker.

2. Step 2: Analyze cluster behavior

   Kafka does not auto-add brokers; load per broker rises, risking performance issues.

3. Final Answer:

   The load per broker increases, possibly causing performance issues -> Option C

4. Quick Check:

   More partitions + same brokers = more load per broker [OK]

Hint: More partitions without brokers = more load per broker [OK]

Common Mistakes:

• Assuming Kafka auto-adds brokers
• Thinking consumers must be reduced
• Believing Kafka rejects partition increase

4. You have a Kafka cluster with 5 brokers but notice high CPU usage on one broker only. What is the likely cause?

medium

A. Uneven partition distribution causing load imbalance

B. All brokers have equal load by design

C. Kafka automatically balances CPU usage evenly

D. The broker hardware is faulty and cannot be fixed

Solution

1. Step 1: Understand load distribution in Kafka

   Partitions are assigned to brokers; uneven distribution causes load imbalance.

2. Step 2: Identify cause of high CPU on one broker

   Uneven partition assignment can overload one broker, causing high CPU usage.

3. Final Answer:

   Uneven partition distribution causing load imbalance -> Option A

4. Quick Check:

   Uneven partitions = uneven CPU load [OK]

Hint: Check partition distribution for load imbalance [OK]

Common Mistakes:

• Assuming Kafka balances CPU automatically
• Blaming hardware without checking partitions
• Thinking all brokers always have equal load

5. You want to plan capacity for a Kafka cluster expected to handle 1 million messages per second. Which combination is best to start with?

hard

A. 5 brokers with 5 partitions each and low memory hardware

B. 10 brokers with 100 partitions each and high-performance SSD storage

C. 2 brokers with 10 partitions each and standard HDD storage

D. 1 broker with 200 partitions and no replication

Solution

1. Step 1: Analyze message load requirements

   1 million messages per second is very high; needs many brokers and partitions.

2. Step 2: Choose hardware and partition setup

   High-performance SSDs and many brokers with many partitions support high throughput and reliability.

3. Final Answer:

   10 brokers with 100 partitions each and high-performance SSD storage -> Option B

4. Quick Check:

   High load needs many brokers, partitions, and fast storage [OK]

Hint: High load needs many brokers, partitions, and SSDs [OK]

Common Mistakes:

• Choosing too few brokers or partitions
• Ignoring storage speed impact
• Not planning for replication and reliability