Resource planning and capacity in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is resource planning in Kafka?

Resource planning in Kafka means deciding how many servers, storage, and network capacity you need to handle your data streams smoothly without delays or crashes.

beginner

Why is capacity important in Kafka clusters?

Capacity ensures Kafka can handle the volume of messages and users. If capacity is too low, messages get delayed or lost. Proper capacity keeps Kafka fast and reliable.

intermediate

What role do partitions play in Kafka resource planning?

Partitions split topics into smaller parts. More partitions mean better parallel processing but need more CPU and memory. Planning partitions helps balance load and resource use.

intermediate

How does replication affect Kafka capacity planning?

Replication copies data to multiple brokers for safety. More replicas mean more storage and network use. You must plan capacity to handle this extra load.

intermediate

What is a common method to monitor Kafka cluster capacity?

Using tools like Kafka's JMX metrics or monitoring platforms (e.g., Prometheus) helps track CPU, memory, disk, and network use to adjust resources before problems happen.

What happens if Kafka cluster capacity is too low?

A Kafka runs faster

B Messages may be delayed or lost

C More partitions are created automatically

D Replication increases

If capacity is too low, Kafka cannot process messages quickly, causing delays or loss.

Which Kafka component helps distribute load for better resource use?

A Consumers

B Topics

C Partitions

D Producers

Partitions split topics so multiple servers can handle data in parallel.

How does replication affect resource needs in Kafka?

A It increases storage and network use

B It decreases CPU usage

C It reduces storage needs

D It removes the need for partitions

Replication copies data to multiple brokers, increasing storage and network requirements.

Which tool can help monitor Kafka cluster capacity?

A Git

B Jenkins

C Docker

D Prometheus

Prometheus collects metrics like CPU and memory usage to monitor Kafka.

Why plan the number of partitions in Kafka?

A To balance load and resource use

B To reduce message size

C To increase replication factor

D To avoid using topics

Partitions help spread the workload, so planning their number balances resource use.

Explain how resource planning helps maintain Kafka performance.

Think about what happens if you don't have enough servers or storage.

Describe the impact of replication on Kafka cluster capacity.

Replication is like making backups, but it uses more resources.