Resource planning and capacity in Kafka - Commands & Configuration

Introduction

Resource planning and capacity in Kafka helps you make sure your system can handle the amount of data and users without slowing down or crashing. It involves deciding how many servers, topics, and partitions you need to keep everything running smoothly.

When you expect a sudden increase in users sending messages to Kafka topics.

When you want to avoid message delays or data loss during peak hours.

When you plan to add new applications that will produce or consume large amounts of data.

When you need to balance load across multiple Kafka brokers to prevent overload.

When you want to optimize costs by not over-provisioning servers but still keep good performance.

Config File - server.properties

server.properties

broker.id=1
log.dirs=/var/lib/kafka-logs
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
num.partitions=3
default.replication.factor=2
message.max.bytes=1000012
replica.fetch.max.bytes=1048576

This configuration file sets up the Kafka broker's resource limits and capacity settings.

• num.partitions: Number of partitions per topic, which affects parallelism and throughput.
• default.replication.factor: Number of copies of data for fault tolerance.
• num.network.threads and num.io.threads: Threads for handling network and disk I/O to manage load.
• log.retention.hours: How long messages are kept, affecting storage needs.

Commands

This command creates a new Kafka topic named 'example-topic' with 3 partitions and 2 replicas to balance load and provide fault tolerance.

Terminal

kafka-topics.sh --create --topic example-topic --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

Expected Output

Created topic example-topic.

→

--partitions - Sets how many partitions the topic will have for parallel processing.

→

--replication-factor - Sets how many copies of each partition exist for reliability.

→

--bootstrap-server - Specifies the Kafka server to connect to.

This command shows details about the 'example-topic' topic, including partitions and replicas, to verify capacity settings.

Terminal

kafka-topics.sh --describe --topic example-topic --bootstrap-server localhost:9092

Expected Output

Topic: example-topic	PartitionCount: 3	ReplicationFactor: 2	Configs:

	Topic: example-topic	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2

	Topic: example-topic	Partition: 1	Leader: 2	Replicas: 2,1	Isr: 2,1

	Topic: example-topic	Partition: 2	Leader: 1	Replicas: 1,2	Isr: 1,2

→

--describe - Shows detailed information about the topic.

→

--topic - Specifies which topic to describe.

This command checks the consumer group 'example-group' to see how many messages are lagging, helping to understand if capacity is enough for consumers.

Terminal

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group example-group

Expected Output

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID

example-group   example-topic   0          100             105             5               consumer-1-1234abcd-5678-efgh-ijkl-9012mnop3456 /192.168.1.10   consumer-1

example-group   example-topic   1          200             200             0               consumer-2-1234abcd-5678-efgh-ijkl-9012mnop3456 /192.168.1.11   consumer-2

example-group   example-topic   2          150             155             5               consumer-3-1234abcd-5678-efgh-ijkl-9012mnop3456 /192.168.1.12   consumer-3

→

--describe - Shows detailed information about the consumer group.

→

--group - Specifies the consumer group to check.

Key Concept

If you remember nothing else from this pattern, remember: balancing partitions and replicas properly ensures Kafka can handle load and stay reliable.

Common Mistakes

Creating topics with too few partitions.

This limits parallel processing and can cause slow message handling under load.

Set partitions based on expected message volume and consumer count to allow parallelism.

Setting replication factor to 1 in production.

This risks data loss if a broker fails because there is no backup copy.

Use at least replication factor 2 or 3 for fault tolerance.

Ignoring consumer lag metrics.

Lag means consumers can't keep up, indicating capacity or configuration issues.

Regularly monitor consumer groups and increase partitions or consumers if lag grows.

Summary

Create Kafka topics with enough partitions and replicas to handle load and provide fault tolerance.

Use commands to describe topics and consumer groups to monitor capacity and performance.

Adjust configuration and resources based on monitoring to prevent message delays and data loss.