Batching and linger configuration in Kafka - Deep Dive

Overview - Batching and linger configuration

What is it?

Batching and linger configuration are settings in Kafka producers that control how messages are grouped before sending to the Kafka broker. Batching means collecting multiple messages into one request to improve efficiency. Linger time is how long the producer waits to fill a batch before sending it, even if the batch is not full.

Why it matters

Without batching and linger settings, Kafka producers would send each message individually, causing more network overhead and slower throughput. Proper configuration improves performance and resource use, making data streaming faster and cheaper.

Where it fits

Learners should first understand Kafka basics like producers, consumers, and topics. After mastering batching and linger, they can explore Kafka performance tuning and advanced producer configurations.

Mental Model

Core Idea

Batching and linger settings let Kafka producers group messages to send fewer, bigger requests, balancing speed and efficiency.

Think of it like...

It's like mailing letters: instead of sending each letter alone, you wait a bit to gather several letters into one envelope to save postage and trips to the mailbox.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Incoming msgs │──────▶│ Batch buffer  │──────▶│ Send to broker│
└───────────────┘       └───────────────┘       └───────────────┘
          ▲                     │
          │                     ▼
       Linger time          Batch size limit

Build-Up - 6 Steps

1

Foundation

What is message batching in Kafka

Concept: Introduce the idea of grouping multiple messages into one batch before sending.

Kafka producers can collect multiple messages into a batch. Instead of sending each message immediately, the producer waits to fill a batch. This reduces the number of network requests and improves throughput.

Result

Messages are sent in groups, reducing network overhead and increasing efficiency.

Understanding batching is key because it directly affects how fast and efficiently data moves through Kafka.

2

Foundation

Understanding linger time setting

Concept: Explain linger.ms, the time the producer waits before sending a batch.

Linger time is a setting that tells the producer how long to wait for more messages before sending the current batch. If the batch fills up before this time, it sends immediately. If not, it waits up to linger.ms milliseconds to add more messages.

Result

Producer may delay sending to create bigger batches, improving throughput but adding slight delay.

Knowing linger time helps balance between latency (speed) and throughput (efficiency).

3

Intermediate

How batch size affects sending behavior

🤔Before reading on: do you think increasing batch size always improves performance or can it sometimes hurt? Commit to your answer.

Concept: Batch size limits how many bytes can be grouped before sending.

The batch.size setting controls the maximum size in bytes of a batch. When the batch reaches this size, it sends immediately, ignoring linger time. Larger batch sizes can improve throughput but use more memory and may increase latency.

Result

Producer sends batches either when batch size is reached or linger time expires.

Understanding batch size helps tune memory use and network efficiency for different workloads.

4

Intermediate

Trade-offs between latency and throughput

🤔Before reading on: do you think setting linger.ms to zero improves latency or throughput? Commit to your answer.

Concept: Explain how linger.ms and batch.size settings affect latency and throughput trade-off.

Setting linger.ms to zero means the producer sends messages immediately without waiting, reducing latency but lowering throughput. Increasing linger.ms allows bigger batches, improving throughput but adding delay. Batch size also affects this balance.

Result

Tuning these settings changes how fast messages arrive versus how efficiently they are sent.

Knowing this trade-off helps choose settings based on whether speed or efficiency matters more.

5

Advanced

Impact of batching on producer memory and CPU

🤔Before reading on: do you think bigger batches always reduce CPU usage? Commit to your answer.

Concept: Explore how batching affects resource use inside the producer client.

Larger batches use more memory to hold messages before sending. They can reduce CPU work by sending fewer requests but may increase CPU if compression is used on big batches. Improper settings can cause memory pressure or CPU spikes.

Result

Batching settings influence producer resource consumption and stability.

Understanding resource impact prevents performance bottlenecks and crashes in production.

6

Expert

How linger and batch size interact with retries and acks

🤔Before reading on: do you think batching affects message delivery guarantees? Commit to your answer.

Concept: Explain how batching settings interact with retries and acknowledgment modes.

When retries happen, batches may be resent, increasing latency. With acks=all, the producer waits for all replicas, so bigger batches can increase wait time. Batching can cause message reordering if retries occur. Proper tuning is needed to balance reliability and performance.

Result

Batching influences delivery guarantees and message order under failure conditions.

Knowing this interaction helps design reliable Kafka producers that meet SLAs.

Under the Hood

Kafka producer maintains an internal buffer where messages are appended. The producer checks batch size and linger time to decide when to send. When batch size limit is reached or linger time expires, the batch is serialized, optionally compressed, and sent over the network to the broker. The producer uses background threads to manage batching and sending asynchronously.

Why designed this way?

Batching and linger were designed to optimize network usage and throughput in high-volume streaming. Sending each message individually would cause excessive network overhead. The design balances latency and throughput by allowing configurable wait times and batch sizes, adapting to different use cases.

┌───────────────┐
│ Message input │
└──────┬────────┘
       │
┌──────▼────────┐
│ Batch buffer  │<─────────────┐
│ (collect msgs)│              │
└──────┬────────┘              │
       │                       │
       │ batch.size reached     │
       │ or linger.ms expired   │
       ▼                       │
┌───────────────┐              │
│ Serialize &   │              │
│ Compress     │              │
└──────┬────────┘              │
       │                       │
       ▼                       │
┌───────────────┐              │
│ Send to Kafka │──────────────┘
│ broker        │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does setting linger.ms to zero mean messages are sent immediately without delay? Commit yes or no.

Common Belief: Setting linger.ms to zero means no delay and best latency.

Reality: Even with linger.ms=0, the producer may still batch messages if batch.size is reached quickly or if internal optimizations apply.

Why it matters: Assuming zero linger means no batching can lead to unexpected latency or throughput behavior.

Quick: Does increasing batch.size always improve performance? Commit yes or no.

Common Belief: Bigger batch sizes always improve Kafka producer performance.

Reality: Too large batch sizes can increase latency, memory use, and risk of message loss on failure.

Why it matters: Blindly increasing batch size can cause resource exhaustion and degrade system stability.

Quick: Does batching guarantee message order is preserved? Commit yes or no.

Common Belief: Batching always preserves the order of messages sent to Kafka.

Reality: Retries and batch resends can cause message reordering despite batching.

Why it matters: Assuming order is guaranteed can cause bugs in systems relying on strict message sequence.

Quick: Does linger.ms affect only latency and not throughput? Commit yes or no.

Common Belief: Linger.ms only impacts message delay, not throughput.

Reality: Linger.ms affects both latency and throughput by controlling batch size and send frequency.

Why it matters: Misunderstanding this leads to poor tuning and unexpected performance results.

Expert Zone

1

Batching behavior can vary depending on compression type; some compressions benefit more from larger batches.

2

Kafka producer's internal buffer memory limits can cause backpressure if batches are too large or linger time too long.

3

Linger.ms is a maximum wait time, but actual send can happen earlier if batch size is reached or buffer is full.

When NOT to use

Batching and linger tuning is not ideal for ultra-low latency use cases like real-time trading where every millisecond counts; instead, use immediate sends with minimal batching. Also, for very low message volumes, batching may add unnecessary delay; consider disabling batching.

Production Patterns

In production, teams often set linger.ms to a small non-zero value (e.g., 5-20 ms) and batch.size to a few hundred KB to balance latency and throughput. They monitor producer metrics to adjust settings dynamically based on workload and use compression to reduce network load.

Connections

TCP Nagle's Algorithm

Similar pattern of delaying small packets to send larger ones efficiently over the network.

Understanding batching in Kafka is easier when you see it like TCP's Nagle algorithm, which also balances latency and throughput by grouping data.

Manufacturing Assembly Lines

Both batch work to improve efficiency by grouping tasks before moving to next stage.

Seeing Kafka batching like an assembly line batch process helps grasp why waiting to group work can save time and resources.

Human Brain Chunking

Batching messages is like how the brain groups information into chunks to process efficiently.

Knowing how chunking helps memory and thinking clarifies why batching improves data processing efficiency in Kafka.

Common Pitfalls

#1 Setting linger.ms too high causing excessive message delay.

Wrong approach:

linger.ms=5000

Correct approach:

linger.ms=20

Root cause:

Misunderstanding linger.ms units and impact leads to large delays hurting latency.

#2 Setting batch.size too small causing many small batches and poor throughput.

Wrong approach:

batch.size=1024

Correct approach:

batch.size=16384

Root cause:

Not knowing batch.size is in bytes and too small values cause inefficient network use.

#3 Disabling batching by setting linger.ms=0 and expecting best performance always.

Wrong approach:

linger.ms=0

Correct approach:

linger.ms=5

Root cause:

Ignoring that some batching improves throughput and reduces CPU/network overhead.

Key Takeaways

Batching groups multiple messages to send fewer, larger requests, improving Kafka producer efficiency.

Linger.ms controls how long the producer waits to fill a batch, balancing latency and throughput.

Batch size limits batch memory and network size, affecting performance and resource use.

Tuning batching and linger settings requires balancing speed, resource use, and reliability needs.

Misconfigurations can cause high latency, resource exhaustion, or message reordering under retries.