How to Tune Kafka Performance: Key Settings and Tips

To tune Kafka performance, adjust key broker settings like num.network.threads and log.segment.bytes, optimize producer configurations such as batch.size and linger.ms, and fine-tune consumer settings like fetch.min.bytes. Monitoring and balancing resource usage, partition count, and replication also help improve throughput and reduce latency.

Syntax

Kafka performance tuning involves configuring settings in three main areas: broker, producer, and consumer. Each setting controls how Kafka handles data flow and resource use.

• Broker settings: Control server threads, memory, and disk usage.
• Producer settings: Manage batching, compression, and retries.
• Consumer settings: Adjust fetch size and session timeouts.

properties

broker.properties example:
num.network.threads=3
num.io.threads=8
log.segment.bytes=1073741824

producer.properties example:
batch.size=32768
linger.ms=5
compression.type=snappy

consumer.properties example:
fetch.min.bytes=50000
max.poll.records=500
session.timeout.ms=10000

Example

This example shows how to configure a Kafka producer for better throughput by increasing batch size and adding compression.

properties

bootstrap.servers=localhost:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
batch.size=65536
linger.ms=10
compression.type=gzip
acks=all
retries=3

Output

Producer sends larger batches with compression, reducing network calls and improving throughput.

Common Pitfalls

Common mistakes include setting batch sizes too large, causing latency spikes, or too small, causing inefficient network use. Not tuning linger.ms can lead to either high latency or low throughput. Overloading brokers with too many partitions or insufficient threads can cause slow processing.

Also, ignoring consumer fetch.min.bytes and max.poll.records can reduce consumer efficiency.

properties

Wrong producer config:
batch.size=16384
linger.ms=0

Right producer config:
batch.size=65536
linger.ms=10

Quick Reference

Setting	Purpose	Recommended Value
num.network.threads	Number of network threads on broker	3-5
log.segment.bytes	Size of log segment files	1 GB
batch.size	Producer batch size in bytes	32 KB - 64 KB
linger.ms	Producer wait time to batch messages	5-10 ms
compression.type	Compression codec for producer	snappy or gzip
fetch.min.bytes	Minimum bytes consumer fetches	50 KB
max.poll.records	Max records per consumer poll	500

Key Takeaways

Tune broker thread and log segment sizes to optimize resource use.

Increase producer batch size and use compression to improve throughput.

Adjust consumer fetch size and poll records for efficient consumption.

Avoid too large batch sizes or too low linger.ms to balance latency and throughput.

Monitor Kafka metrics regularly to guide tuning decisions.