Memory and buffer configuration in Kafka - Commands & Configuration

Introduction

Kafka uses memory and buffers to handle data efficiently between producers, brokers, and consumers. Proper configuration of these settings helps avoid slowdowns and data loss by managing how much data Kafka holds in memory before writing to disk or sending to clients.

When you want to improve Kafka broker performance by tuning how much memory it uses for caching data.

When producers need to batch messages efficiently before sending to Kafka to reduce network overhead.

When consumers require tuning of fetch buffers to optimize data retrieval speed.

When you face issues like high latency or frequent disk writes in Kafka and want to adjust buffer sizes.

When running Kafka on machines with limited memory and you want to avoid out-of-memory errors.

Config File - server.properties

server.properties

log.dirs=/tmp/kafka-logs
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=true
log.cleaner.threads=2
log.cleaner.dedupe.buffer.size=134217728
log.cleaner.io.buffer.size=524288
log.cleaner.io.max.bytes.per.second=10485760
log.cleaner.backoff.ms=15000
num.replica.fetchers=1
replica.fetch.max.bytes=1048576
replica.fetch.wait.max.ms=500
replica.fetch.min.bytes=1
replica.fetch.max.wait.ms=500
fetch.purgatory.purge.interval.requests=1000
fetch.purgatory.purge.interval.ms=1000
replica.lag.time.max.ms=10000
replica.lag.max.messages=4000
replica.socket.timeout.ms=30000
replica.socket.receive.buffer.bytes=65536
replica.fetch.response.max.bytes=1048576
message.max.bytes=1000012

This file configures Kafka broker memory and buffer settings:

• socket.send.buffer.bytes and socket.receive.buffer.bytes: Size of TCP send and receive buffers for network communication.
• log.cleaner.dedupe.buffer.size and log.cleaner.io.buffer.size: Buffers used by the log cleaner thread to compact logs efficiently.
• replica.fetch.max.bytes and related replica fetch settings: Control how much data replicas fetch from leaders in one request.
• message.max.bytes: Maximum size of a message that can be sent or received.

Tuning these values helps Kafka handle data flow smoothly and avoid bottlenecks.

Commands

Starts the Kafka broker using the server.properties file which includes memory and buffer configurations.

Terminal

kafka-server-start.sh /opt/kafka/config/server.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka version : 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,001] INFO Kafka commitId : abcdef1234567890 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,500] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Checks if Kafka broker is listening on the default port 9092 after startup.

Terminal

netstat -an | grep 9092

Expected Output

tcp        0      0 0.0.0.0:9092            0.0.0.0:*               LISTEN

Creates a test topic to verify Kafka broker is working with the configured memory and buffer settings.

Terminal

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test-topic --partitions 1 --replication-factor 1

Expected Output

Created topic test-topic.

→

--partitions - Sets number of partitions for the topic

→

--replication-factor - Sets replication factor for fault tolerance

Describes the test topic to confirm it is created and broker is managing topic metadata correctly.

Terminal

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic test-topic

Expected Output

Topic: test-topic	PartitionCount: 1	ReplicationFactor: 1	Configs: 

	Topic: test-topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

Key Concept

If you remember nothing else from this pattern, remember: tuning Kafka's memory and buffer sizes controls how efficiently data moves through the system and prevents slowdowns or crashes.

Common Mistakes

Setting socket buffer sizes too small

This causes network bottlenecks and slows down data transfer between clients and brokers.

Set socket.send.buffer.bytes and socket.receive.buffer.bytes to values like 102400 or higher depending on network speed.

Not adjusting log cleaner buffer sizes when enabling log compaction

Leads to inefficient log cleaning and higher disk usage.

Configure log.cleaner.dedupe.buffer.size and log.cleaner.io.buffer.size to appropriate sizes like 128MB and 512KB respectively.

Using default message.max.bytes when sending large messages

Messages larger than the default limit get rejected causing errors.

Increase message.max.bytes to accommodate your largest expected message size.

Summary

Configure Kafka broker memory and buffer settings in server.properties to optimize data flow.

Start Kafka broker with these settings and verify it listens on port 9092.

Create and describe a topic to confirm Kafka is working with the configured buffers.