Memory and buffer configuration in Kafka - Commands & Configuration

Introduction

Kafka uses memory and buffers to handle data efficiently between producers, brokers, and consumers. Proper configuration of these settings helps avoid slowdowns and data loss by managing how much data Kafka holds in memory before writing to disk or sending to clients.

When you want to improve Kafka broker performance by tuning how much memory it uses for caching data.

When producers need to batch messages efficiently before sending to Kafka to reduce network overhead.

When consumers require tuning of fetch buffers to optimize data retrieval speed.

When you face issues like high latency or frequent disk writes in Kafka and want to adjust buffer sizes.

When running Kafka on machines with limited memory and you want to avoid out-of-memory errors.

Config File - server.properties

server.properties

log.dirs=/tmp/kafka-logs
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=true
log.cleaner.threads=2
log.cleaner.dedupe.buffer.size=134217728
log.cleaner.io.buffer.size=524288
log.cleaner.io.max.bytes.per.second=10485760
log.cleaner.backoff.ms=15000
num.replica.fetchers=1
replica.fetch.max.bytes=1048576
replica.fetch.wait.max.ms=500
replica.fetch.min.bytes=1
replica.fetch.max.wait.ms=500
fetch.purgatory.purge.interval.requests=1000
fetch.purgatory.purge.interval.ms=1000
replica.lag.time.max.ms=10000
replica.lag.max.messages=4000
replica.socket.timeout.ms=30000
replica.socket.receive.buffer.bytes=65536
replica.fetch.response.max.bytes=1048576
message.max.bytes=1000012

This file configures Kafka broker memory and buffer settings:

• socket.send.buffer.bytes and socket.receive.buffer.bytes: Size of TCP send and receive buffers for network communication.
• log.cleaner.dedupe.buffer.size and log.cleaner.io.buffer.size: Buffers used by the log cleaner thread to compact logs efficiently.
• replica.fetch.max.bytes and related replica fetch settings: Control how much data replicas fetch from leaders in one request.
• message.max.bytes: Maximum size of a message that can be sent or received.

Tuning these values helps Kafka handle data flow smoothly and avoid bottlenecks.

Commands

Starts the Kafka broker using the server.properties file which includes memory and buffer configurations.

Terminal

kafka-server-start.sh /opt/kafka/config/server.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka version : 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,001] INFO Kafka commitId : abcdef1234567890 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,500] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Checks if Kafka broker is listening on the default port 9092 after startup.

Terminal

netstat -an | grep 9092

Expected Output

tcp        0      0 0.0.0.0:9092            0.0.0.0:*               LISTEN

Creates a test topic to verify Kafka broker is working with the configured memory and buffer settings.

Terminal

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test-topic --partitions 1 --replication-factor 1

Expected Output

Created topic test-topic.

→

--partitions - Sets number of partitions for the topic

→

--replication-factor - Sets replication factor for fault tolerance

Describes the test topic to confirm it is created and broker is managing topic metadata correctly.

Terminal

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic test-topic

Expected Output

Topic: test-topic	PartitionCount: 1	ReplicationFactor: 1	Configs: 

	Topic: test-topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

Key Concept

If you remember nothing else from this pattern, remember: tuning Kafka's memory and buffer sizes controls how efficiently data moves through the system and prevents slowdowns or crashes.

Common Mistakes

Setting socket buffer sizes too small

This causes network bottlenecks and slows down data transfer between clients and brokers.

Set socket.send.buffer.bytes and socket.receive.buffer.bytes to values like 102400 or higher depending on network speed.

Not adjusting log cleaner buffer sizes when enabling log compaction

Leads to inefficient log cleaning and higher disk usage.

Configure log.cleaner.dedupe.buffer.size and log.cleaner.io.buffer.size to appropriate sizes like 128MB and 512KB respectively.

Using default message.max.bytes when sending large messages

Messages larger than the default limit get rejected causing errors.

Increase message.max.bytes to accommodate your largest expected message size.

Summary

Configure Kafka broker memory and buffer settings in server.properties to optimize data flow.

Start Kafka broker with these settings and verify it listens on port 9092.

Create and describe a topic to confirm Kafka is working with the configured buffers.

Practice

1. Which Kafka configuration setting controls the size of the buffer used to batch data before sending to brokers?

easy

A. fetch.min.bytes

B. batch.size

C. buffer.memory

D. linger.ms

Solution

1. Step 1: Understand buffer settings in Kafka producer

   The batch.size setting controls the maximum size of a batch of records sent to brokers.

2. Step 2: Differentiate from other buffer-related settings

   buffer.memory controls total memory for buffering, fetch.min.bytes is for consumers, and linger.ms controls delay before sending batches.

3. Final Answer:

   batch.size -> Option B

4. Quick Check:

   Batch size controls batch buffer size [OK]

Hint: Batch size sets buffer for sending data batches [OK]

Common Mistakes:

• Confusing total buffer memory with batch size
• Mixing consumer fetch settings with producer buffers
• Thinking linger.ms controls buffer size

2. Which of the following is the correct syntax to set the buffer memory size to 32MB in Kafka producer properties file?

easy

A. buffer.memory=33554432

B. buffer.memory=32

C. buffer.memory=32m

D. buffer.memory=32MB

Solution

1. Step 1: Understand Kafka config value units

   Kafka expects buffer.memory in bytes, not with units like MB or m.

2. Step 2: Convert 32MB to bytes

   32MB = 32 * 1024 * 1024 = 33554432 bytes, so the value must be numeric bytes.

3. Final Answer:

   buffer.memory=33554432 -> Option A

4. Quick Check:

   Buffer memory uses bytes, not MB string [OK]

Hint: Use bytes number, not MB string, for buffer.memory [OK]

Common Mistakes:

• Adding MB or m suffix which Kafka does not parse
• Using too small number without units
• Confusing buffer.memory with batch.size units

3. Given this Kafka producer config snippet:

buffer.memory=1048576
batch.size=16384
linger.ms=10

What is the maximum amount of data (in bytes) that can be buffered before sending?

medium

A. 1048586 bytes

B. 16384 bytes

C. 1048576 bytes

D. 10 bytes

Solution

1. Step 1: Identify buffer.memory meaning

   buffer.memory is the total bytes available to buffer records waiting to be sent.

2. Step 2: Understand batch.size role

   batch.size is max size per batch, but total buffer can hold multiple batches up to buffer.memory.

3. Final Answer:

   1048576 bytes -> Option C

4. Quick Check:

   Total buffer memory = 1048576 bytes [OK]

Hint: buffer.memory is total buffer size, batch.size is per batch [OK]

Common Mistakes:

• Confusing batch.size as total buffer
• Adding linger.ms value to buffer size
• Choosing sum of batch.size and buffer.memory

4. You set buffer.memory=1024 and batch.size=2048 in Kafka producer config. What issue will occur?

medium

A. Producer will throw a configuration error on startup

B. No issue; Kafka adjusts batch.size automatically

C. Producer will silently drop messages exceeding buffer.memory

D. Producer will block or throw an error when batch.size exceeds buffer.memory

Solution

1. Step 1: Compare batch.size and buffer.memory values

   Here, batch.size (2048) is larger than buffer.memory (1024), which is invalid.

2. Step 2: Kafka behavior on invalid buffer sizes

   Kafka producer will block or throw an error because it cannot allocate a batch larger than total buffer memory.

3. Final Answer:

   Producer will block or throw an error when batch.size exceeds buffer.memory -> Option D

4. Quick Check:

   batch.size must be ≤ buffer.memory [OK]

Hint: batch.size cannot be bigger than buffer.memory [OK]

Common Mistakes:

• Assuming Kafka auto-corrects batch.size
• Expecting silent message drops
• Thinking config error happens at startup

5. You want to optimize Kafka producer for low latency and high throughput. Which combination of memory and buffer settings is best?

hard

A. Increase buffer.memory, decrease linger.ms, increase batch.size

B. Decrease buffer.memory, increase linger.ms, decrease batch.size

C. Keep buffer.memory small, set linger.ms high, increase batch.size

D. Set buffer.memory and batch.size low, linger.ms high

Solution

1. Step 1: Understand latency vs throughput tradeoff

   Low latency means sending data quickly, high throughput means sending large batches efficiently.

2. Step 2: Adjust buffer and batch settings accordingly

   Increasing buffer.memory and batch.size allows bigger batches, while decreasing linger.ms reduces wait time before sending.

3. Final Answer:

   Increase buffer.memory, decrease linger.ms, increase batch.size -> Option A

4. Quick Check:

   Big buffer + big batch + low linger = low latency & high throughput [OK]

Hint: Big buffer and batch, low linger for speed and volume [OK]

Common Mistakes:

• Increasing linger.ms reduces latency
• Decreasing buffer.memory improves throughput
• Setting batch.size too low for throughput