Auto-scaling strategies in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is auto-scaling in the context of Kafka?

Auto-scaling in Kafka means automatically adjusting the number of Kafka brokers or partitions based on workload to maintain performance and resource efficiency.

beginner

Name two common metrics used to trigger Kafka auto-scaling.

Common metrics include CPU usage of brokers and consumer lag (the delay in processing messages). These help decide when to add or remove resources.

intermediate

Explain horizontal scaling vs vertical scaling in Kafka auto-scaling.

Horizontal scaling means adding or removing Kafka brokers (machines). Vertical scaling means increasing or decreasing resources (CPU, memory) on existing brokers.

advanced

What is a common challenge when auto-scaling Kafka partitions?

Rebalancing partitions can cause temporary performance drops and data movement, so scaling partitions must be done carefully to avoid disruption.

intermediate

How does Kubernetes help with Kafka auto-scaling?

Kubernetes can manage Kafka brokers as pods and use Horizontal Pod Autoscaler to add or remove broker pods based on resource usage or custom metrics.

Which metric is commonly used to decide when to auto-scale Kafka brokers?

A Message content

B Disk size

C CPU usage

D Number of topics

CPU usage is a key metric to determine if brokers are overloaded and need scaling.

What does horizontal scaling mean in Kafka auto-scaling?

A Adding more CPU to existing brokers

B Reducing number of partitions

C Increasing message size

D Adding more brokers to the cluster

Horizontal scaling means adding or removing brokers (machines) in the Kafka cluster.

What is a risk when increasing Kafka partitions during auto-scaling?

A Data loss

B Temporary performance drop due to rebalancing

C Messages get duplicated

D Topics get deleted

Rebalancing partitions can cause temporary performance issues as data moves between brokers.

Which tool can help automate Kafka broker scaling in container environments?

A Kubernetes Horizontal Pod Autoscaler

B Docker Compose

C Ansible

D Terraform

Kubernetes Horizontal Pod Autoscaler can automatically scale Kafka broker pods based on metrics.

Consumer lag in Kafka is used to measure:

A Delay in consumers processing messages

B How fast producers send messages

C Number of brokers in the cluster

D Size of messages

Consumer lag shows how far behind consumers are in processing messages, useful for scaling decisions.

Describe how auto-scaling works in Kafka and what metrics are important to monitor.

Think about how Kafka adjusts resources automatically based on workload.

Explain the difference between horizontal and vertical scaling in Kafka auto-scaling and why each might be used.

Consider adding machines vs upgrading existing machines.

Practice

1. What is the main purpose of auto-scaling in Kafka clusters?

easy

A. To backup data to external storage

B. To manually add brokers when needed

C. To automatically adjust resources based on workload changes

D. To encrypt messages for security

Solution

1. Step 1: Understand auto-scaling concept

   Auto-scaling means adjusting resources automatically without manual intervention.

2. Step 2: Apply to Kafka context

   Kafka auto-scaling adjusts brokers or partitions based on workload changes.

3. Final Answer:

   To automatically adjust resources based on workload changes -> Option C

4. Quick Check:

   Auto-scaling = automatic resource adjustment [OK]

Hint: Auto-scaling means automatic resource changes, not manual [OK]

Common Mistakes:

• Confusing auto-scaling with manual scaling
• Thinking auto-scaling is for data backup
• Assuming auto-scaling handles security

2. Which of the following is a correct way to trigger auto-scaling in Kafka using metrics?

easy

A. Increase partition count without checking load

B. Manually restart brokers every hour

C. Disable monitoring to save resources

D. Monitor CPU usage and add brokers when usage exceeds threshold

Solution

1. Step 1: Identify correct auto-scaling trigger

   Auto-scaling triggers on metrics like CPU usage crossing a threshold.

2. Step 2: Evaluate options

   Only Monitor CPU usage and add brokers when usage exceeds threshold uses monitoring metrics to trigger scaling automatically.

3. Final Answer:

   Monitor CPU usage and add brokers when usage exceeds threshold -> Option D

4. Quick Check:

   Auto-scale triggers = metric thresholds [OK]

Hint: Auto-scaling triggers on metrics, not manual actions [OK]

Common Mistakes:

• Confusing manual restarts with auto-scaling
• Ignoring monitoring importance
• Scaling without checking load

3. Given this pseudo-code for auto-scaling Kafka brokers:

if (cpu_usage > 80%) {
  add_broker()
} else if (cpu_usage < 30%) {
  remove_broker()
} else {
  do_nothing()
}

What happens if CPU usage is 85%?

medium

A. A broker is added

B. No change in brokers

C. A broker is removed

D. Error occurs due to syntax

Solution

1. Step 1: Check CPU usage condition

   CPU usage is 85%, which is greater than 80%.

2. Step 2: Determine action based on condition

   Since cpu_usage > 80%, the code calls add_broker() to add a broker.

3. Final Answer:

   A broker is added -> Option A

4. Quick Check:

   CPU > 80% triggers add_broker() [OK]

Hint: CPU > 80% means add broker, CPU < 30% means remove [OK]

Common Mistakes:

• Confusing add and remove actions
• Ignoring else conditions
• Assuming syntax error without cause

4. This auto-scaling script snippet has an error:

if cpu_usage > 70%:
  add_broker()
elif cpu_usage < 20%:
  remove_broker()

What is the error?

medium

A. Incorrect percentage symbol usage in condition

B. add_broker() function is undefined

C. Missing colon after if statement

D. No error, code is correct

Solution

1. Step 1: Analyze condition syntax

   Using '%' directly in conditions like 'cpu_usage > 70%' is invalid syntax in most languages.

2. Step 2: Identify correct syntax

   Percent values should be numeric (e.g., 0.7 or 70) without '%' symbol in code.

3. Final Answer:

   Incorrect percentage symbol usage in condition -> Option A

4. Quick Check:

   Conditions must use numbers, not '%' symbol [OK]

Hint: Don't use % symbol in code conditions, use numbers only [OK]

Common Mistakes:

• Assuming missing colon is the error
• Thinking functions are undefined without context
• Believing code is correct as is

5. You want to auto-scale Kafka partitions based on message lag. Which strategy best fits this goal?

hard

A. Add brokers randomly without monitoring lag

B. Increase partitions when lag exceeds threshold; decrease when lag is low

C. Scale partitions only based on CPU usage

D. Manually change partitions daily regardless of lag

Solution

1. Step 1: Understand message lag as scaling metric

   Message lag indicates how many messages are waiting to be processed, a key metric for scaling partitions.

2. Step 2: Choose strategy matching lag metric

   Increasing partitions when lag is high and decreasing when low optimizes processing capacity.

3. Final Answer:

   Increase partitions when lag exceeds threshold; decrease when lag is low -> Option B

4. Quick Check:

   Lag-based scaling = adjust partitions by lag [OK]

Hint: Scale partitions based on lag, not random or CPU only [OK]

Common Mistakes:

• Ignoring lag metric importance
• Scaling without monitoring
• Manual scaling instead of auto