Auto-scaling strategies in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Kafka consumer group scaling code?

Consider a Kafka consumer group with 3 consumers consuming from 3 partitions. The code below simulates scaling the consumers dynamically. What will be the number of active consumers after running this code?

Kafka

consumers = 3
partitions = 3

# Simulate scaling down by 1 consumer
consumers -= 1

# Simulate scaling up by 2 consumers
consumers += 2

# But max consumers cannot exceed partitions
active_consumers = min(consumers, partitions)
print(active_consumers)

A 3

B 1

C 2

D 4

💡 Hint

Remember that the number of active consumers cannot exceed the number of partitions.

The code first reduces consumers from 3 to 2, then increases to 4. But since partitions are 3, active consumers are capped at 3.

🧠 Conceptual

intermediate

Which auto-scaling metric is best for Kafka consumer groups?

You want to auto-scale Kafka consumers based on workload. Which metric is most appropriate to trigger scaling?

A Number of active partitions

B Consumer lag (messages behind)

C CPU usage of consumer machines

D Network bandwidth usage

💡 Hint

Think about what indicates consumers are falling behind.

Consumer lag shows how many messages are waiting to be processed, so scaling based on lag helps keep up with workload.

🔧 Debug

advanced

Why does this Kafka auto-scaling script fail to scale down?

This Python script tries to scale down Kafka consumers when lag is low, but consumers never decrease. What is the bug?

Kafka

consumers = 5
lag = 0

if lag < 10:
    consumers = consumers - 1
else:
    consumers = consumers + 1

print(consumers)

A The script does not check if consumers go below 1

B The condition should be lag <= 10 instead of lag < 10

C The print statement is outside the if block

D The lag variable is never updated dynamically

💡 Hint

Check if lag changes during the script execution.

Lag is set to 0 and never updated, so the condition always triggers scale down once, but in real use lag should be updated dynamically.

📝 Syntax

advanced

Which option correctly implements a Kafka consumer auto-scaling check in Python?

Select the code snippet that correctly checks if consumer lag exceeds threshold and triggers scale up.

A

if lag > threshold
    scale_up()

B

if (lag > threshold)
    scale_up()

C

if lag > threshold:
    scale_up()

D

if lag > threshold then
    scale_up()

💡 Hint

Remember Python requires a colon after if condition.

Python syntax requires a colon at the end of the if statement line.

🚀 Application

expert

How many consumers will be active after this auto-scaling logic runs?

Given 4 partitions and initial 2 consumers, the code below scales consumers based on lag values. What is the final number of active consumers?

Kafka

partitions = 4
consumers = 2
lag_values = [5, 15, 8, 20]

for lag in lag_values:
    if lag > 10:
        consumers += 1
    elif lag < 5 and consumers > 1:
        consumers -= 1

active_consumers = min(consumers, partitions)
print(active_consumers)

A 4

B 2

C 3

D 5

💡 Hint

Count how many times consumers increase or decrease, then cap by partitions.

Lag values >10 occur twice, increasing consumers from 2 to 4. Lag <5 never occurs, so no decrease. Max consumers capped at 4 partitions.

Practice

1. What is the main purpose of auto-scaling in Kafka clusters?

easy

A. To backup data to external storage

B. To manually add brokers when needed

C. To automatically adjust resources based on workload changes

D. To encrypt messages for security

Solution

1. Step 1: Understand auto-scaling concept

   Auto-scaling means adjusting resources automatically without manual intervention.

2. Step 2: Apply to Kafka context

   Kafka auto-scaling adjusts brokers or partitions based on workload changes.

3. Final Answer:

   To automatically adjust resources based on workload changes -> Option C

4. Quick Check:

   Auto-scaling = automatic resource adjustment [OK]

Hint: Auto-scaling means automatic resource changes, not manual [OK]

Common Mistakes:

• Confusing auto-scaling with manual scaling
• Thinking auto-scaling is for data backup
• Assuming auto-scaling handles security

2. Which of the following is a correct way to trigger auto-scaling in Kafka using metrics?

easy

A. Increase partition count without checking load

B. Manually restart brokers every hour

C. Disable monitoring to save resources

D. Monitor CPU usage and add brokers when usage exceeds threshold

Solution

1. Step 1: Identify correct auto-scaling trigger

   Auto-scaling triggers on metrics like CPU usage crossing a threshold.

2. Step 2: Evaluate options

   Only Monitor CPU usage and add brokers when usage exceeds threshold uses monitoring metrics to trigger scaling automatically.

3. Final Answer:

   Monitor CPU usage and add brokers when usage exceeds threshold -> Option D

4. Quick Check:

   Auto-scale triggers = metric thresholds [OK]

Hint: Auto-scaling triggers on metrics, not manual actions [OK]

Common Mistakes:

• Confusing manual restarts with auto-scaling
• Ignoring monitoring importance
• Scaling without checking load

3. Given this pseudo-code for auto-scaling Kafka brokers:

if (cpu_usage > 80%) {
  add_broker()
} else if (cpu_usage < 30%) {
  remove_broker()
} else {
  do_nothing()
}

What happens if CPU usage is 85%?

medium

A. A broker is added

B. No change in brokers

C. A broker is removed

D. Error occurs due to syntax

Solution

1. Step 1: Check CPU usage condition

   CPU usage is 85%, which is greater than 80%.

2. Step 2: Determine action based on condition

   Since cpu_usage > 80%, the code calls add_broker() to add a broker.

3. Final Answer:

   A broker is added -> Option A

4. Quick Check:

   CPU > 80% triggers add_broker() [OK]

Hint: CPU > 80% means add broker, CPU < 30% means remove [OK]

Common Mistakes:

• Confusing add and remove actions
• Ignoring else conditions
• Assuming syntax error without cause

4. This auto-scaling script snippet has an error:

if cpu_usage > 70%:
  add_broker()
elif cpu_usage < 20%:
  remove_broker()

What is the error?

medium

A. Incorrect percentage symbol usage in condition

B. add_broker() function is undefined

C. Missing colon after if statement

D. No error, code is correct

Solution

1. Step 1: Analyze condition syntax

   Using '%' directly in conditions like 'cpu_usage > 70%' is invalid syntax in most languages.

2. Step 2: Identify correct syntax

   Percent values should be numeric (e.g., 0.7 or 70) without '%' symbol in code.

3. Final Answer:

   Incorrect percentage symbol usage in condition -> Option A

4. Quick Check:

   Conditions must use numbers, not '%' symbol [OK]

Hint: Don't use % symbol in code conditions, use numbers only [OK]

Common Mistakes:

• Assuming missing colon is the error
• Thinking functions are undefined without context
• Believing code is correct as is

5. You want to auto-scale Kafka partitions based on message lag. Which strategy best fits this goal?

hard

A. Add brokers randomly without monitoring lag

B. Increase partitions when lag exceeds threshold; decrease when lag is low

C. Scale partitions only based on CPU usage

D. Manually change partitions daily regardless of lag

Solution

1. Step 1: Understand message lag as scaling metric

   Message lag indicates how many messages are waiting to be processed, a key metric for scaling partitions.

2. Step 2: Choose strategy matching lag metric

   Increasing partitions when lag is high and decreasing when low optimizes processing capacity.

3. Final Answer:

   Increase partitions when lag exceeds threshold; decrease when lag is low -> Option B

4. Quick Check:

   Lag-based scaling = adjust partitions by lag [OK]

Hint: Scale partitions based on lag, not random or CPU only [OK]

Common Mistakes:

• Ignoring lag metric importance
• Scaling without monitoring
• Manual scaling instead of auto