Batching and linger configuration in Kafka - Commands & Configuration

Introduction

When sending messages to Kafka, batching groups multiple messages to send together, improving speed. Linger time sets how long to wait before sending a batch, allowing more messages to join the batch and reduce network calls.

When you want to improve Kafka producer throughput by sending messages in groups instead of one by one

When your application sends many small messages rapidly and you want to reduce network overhead

When you want to balance latency and throughput by controlling how long the producer waits before sending messages

When you want to optimize resource use on the Kafka broker by reducing the number of requests

When tuning Kafka producer performance for better efficiency in a high-load environment

Config File - producer.properties

producer.properties

bootstrap.servers=localhost:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
batch.size=16384
linger.ms=10

bootstrap.servers: Kafka broker address to connect to.

key.serializer and value.serializer: Define how keys and values are converted to bytes.

batch.size: Maximum size in bytes for a batch before sending.

linger.ms: Time in milliseconds to wait for more messages before sending the batch.

Commands

Starts a Kafka producer using the configuration file with batching and linger settings to send messages to 'example-topic'.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --producer.config producer.properties

Expected Output

> This is a test message

> Another message

→

--broker-list - Specifies the Kafka broker address to connect to

→

--topic - Specifies the topic to send messages to

→

--producer.config - Loads producer configuration including batching and linger settings

Reads the first two messages from 'example-topic' to verify that messages sent with batching and linger settings are received correctly.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --max-messages 2

Expected Output

This is a test message

Another message

→

--bootstrap-server - Specifies the Kafka broker address to connect to

→

--topic - Specifies the topic to read messages from

→

--from-beginning - Reads messages from the start of the topic

→

--max-messages - Limits the number of messages to read

Key Concept

If you remember nothing else from this pattern, remember: batching groups messages to send together and linger.ms sets how long to wait to form a batch, balancing speed and latency.

Common Mistakes

Setting linger.ms to 0 and expecting batching to improve throughput

With linger.ms at 0, the producer sends messages immediately without waiting to batch, reducing batching benefits.

Set linger.ms to a small positive value like 10 ms to allow messages to accumulate before sending.

Setting batch.size too small causing frequent sends

Small batch size limits how many messages can be grouped, increasing network calls and reducing throughput.

Use a reasonable batch.size like 16384 bytes to allow efficient batching.

Summary

Configure batch.size and linger.ms in the Kafka producer properties to control message batching.

Use kafka-console-producer with the config file to send messages with batching and linger settings.

Verify messages are received correctly using kafka-console-consumer reading from the topic.