Why tuning handles production load in Kafka - Why It Works

Introduction

When many users or systems send data to Kafka at the same time, it can slow down or crash if not set up right. Tuning Kafka means adjusting settings so it can handle lots of data smoothly without delays or errors.

When your Kafka cluster slows down during peak business hours and messages pile up.

When you notice consumers lagging behind producers and data is delayed.

When you want to avoid losing messages during sudden spikes in traffic.

When your Kafka brokers use too much CPU or memory under load.

When you want to improve throughput and reduce latency for real-time data processing.

Config File - server.properties

server.properties

broker.id=1
listeners=PLAINTEXT://:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=3
default.replication.factor=2
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
replica.fetch.max.bytes=1048576
message.max.bytes=1000012
fetch.max.bytes=52428800
replica.lag.time.max.ms=10000
replica.lag.max.messages=4000

broker.id: Unique ID for each Kafka broker.

listeners: Network address Kafka listens on.

num.network.threads and num.io.threads: Threads for handling network and disk I/O.

socket.send.buffer.bytes and socket.receive.buffer.bytes: Network buffer sizes to improve data flow.

num.partitions: Number of partitions per topic to allow parallel processing.

default.replication.factor: Number of copies of data for fault tolerance.

log.retention.hours: How long logs are kept before deletion.

replica.fetch.max.bytes: Max data size fetched by replicas to keep them in sync.

These settings help Kafka handle more data efficiently and stay reliable under heavy use.

Commands

Starts the Kafka broker using the tuned configuration file to handle production load.

Terminal

kafka-server-start.sh /usr/local/kafka/config/server.properties

Expected Output

[2024-06-01 10:00:00,000] INFO Kafka version : 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 10:00:00,001] INFO Kafka startTimeMs : 1685604000000 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 10:00:01,500] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)

Creates a topic with multiple partitions and replication to distribute load and increase fault tolerance.

Terminal

kafka-topics.sh --create --topic example-topic --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

Expected Output

Created topic example-topic.

→

--partitions - Sets number of partitions for parallelism

→

--replication-factor - Sets number of copies for reliability

Checks consumer lag to see if consumers keep up with the production load.

Terminal

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group example-group

Expected Output

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST  CLIENT-ID

example-group   example-topic   0          1000            1000            0    consumer-1   /127.0.0.1  consumer-1

example-group   example-topic   1          1000            1000            0    consumer-1   /127.0.0.1  consumer-1

example-group   example-topic   2          1000            1000            0    consumer-1   /127.0.0.1  consumer-1

→

--describe - Shows detailed consumer group info

Key Concept

If you remember nothing else from this pattern, remember: tuning Kafka’s configuration lets it handle many messages smoothly without delays or crashes.

Common Mistakes

Using only one partition for a high-traffic topic

Limits parallel processing and causes bottlenecks under load

Create topics with multiple partitions to spread the load

Setting replication factor to 1 in production

No data copies means risk of data loss if a broker fails

Set replication factor to at least 2 for fault tolerance

Not monitoring consumer lag regularly

Consumers falling behind causes delayed processing and stale data

Use kafka-consumer-groups.sh to check lag and adjust consumers

Summary

Start Kafka broker with tuned server.properties to optimize resource use.

Create topics with multiple partitions and replication for load distribution and reliability.

Monitor consumer lag to ensure consumers keep up with production load.