Auto-scaling strategies in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is auto-scaling in the context of Kafka?

Auto-scaling in Kafka means automatically adjusting the number of Kafka brokers or partitions based on workload to maintain performance and resource efficiency.

beginner

Name two common metrics used to trigger Kafka auto-scaling.

Common metrics include CPU usage of brokers and consumer lag (the delay in processing messages). These help decide when to add or remove resources.

intermediate

Explain horizontal scaling vs vertical scaling in Kafka auto-scaling.

Horizontal scaling means adding or removing Kafka brokers (machines). Vertical scaling means increasing or decreasing resources (CPU, memory) on existing brokers.

advanced

What is a common challenge when auto-scaling Kafka partitions?

Rebalancing partitions can cause temporary performance drops and data movement, so scaling partitions must be done carefully to avoid disruption.

intermediate

How does Kubernetes help with Kafka auto-scaling?

Kubernetes can manage Kafka brokers as pods and use Horizontal Pod Autoscaler to add or remove broker pods based on resource usage or custom metrics.

Which metric is commonly used to decide when to auto-scale Kafka brokers?

A Message content

B Disk size

C CPU usage

D Number of topics

CPU usage is a key metric to determine if brokers are overloaded and need scaling.

What does horizontal scaling mean in Kafka auto-scaling?

A Adding more CPU to existing brokers

B Reducing number of partitions

C Increasing message size

D Adding more brokers to the cluster

Horizontal scaling means adding or removing brokers (machines) in the Kafka cluster.

What is a risk when increasing Kafka partitions during auto-scaling?

A Data loss

B Temporary performance drop due to rebalancing

C Messages get duplicated

D Topics get deleted

Rebalancing partitions can cause temporary performance issues as data moves between brokers.

Which tool can help automate Kafka broker scaling in container environments?

A Kubernetes Horizontal Pod Autoscaler

B Docker Compose

C Ansible

D Terraform

Kubernetes Horizontal Pod Autoscaler can automatically scale Kafka broker pods based on metrics.

Consumer lag in Kafka is used to measure:

A Delay in consumers processing messages

B How fast producers send messages

C Number of brokers in the cluster

D Size of messages

Consumer lag shows how far behind consumers are in processing messages, useful for scaling decisions.

Describe how auto-scaling works in Kafka and what metrics are important to monitor.

Think about how Kafka adjusts resources automatically based on workload.

Explain the difference between horizontal and vertical scaling in Kafka auto-scaling and why each might be used.

Consider adding machines vs upgrading existing machines.