Batching and linger configuration in Kafka - Step-by-Step Execution

Process Flow - Batching and linger configuration

Message arrives to producer

↓

Add message to batch buffer

↓

Is batch size reached?

No→Wait linger.ms time

↓

Send batch

↓

Send batch

↓

Clear batch buffer

↓

Wait for next message

Messages are collected in a batch buffer. When the batch size is reached or linger time expires, the batch is sent.

Execution Sample

Kafka

producer_config = {
  'batch.size': 16384,  # bytes
  'linger.ms': 5       # milliseconds
}

# Messages arrive and are batched before sending

This config sets batch size and linger time to control when messages are sent in batches.

Process Table

Step	Message Arrived	Batch Size (bytes)	Batch Full?	Linger Timer (ms)	Action	Output
1	Msg1 (4000 bytes)	4000	No	Start 5ms timer	Add to batch	No send
2	Msg2 (5000 bytes)	9000	No	Timer running	Add to batch	No send
3	Msg3 (8000 bytes)	17000	Yes (17000 > 16384)	Cancel timer	Send batch	Batch sent with Msg1, Msg2, Msg3
4	Msg4 (3000 bytes)	3000	No	Start 5ms timer	Add to batch	No send
5	No new message	3000	No	Timer expires	Send batch	Batch sent with Msg4
6	-	0	-	-	Batch cleared	Ready for new messages

💡 Batch sent when size limit reached or linger timer expires

Status Tracker

Variable	Start	After 1	After 2	After 3	After 4	After 5	Final
Batch Size (bytes)	0	4000	9000	17000	3000	0	0
Linger Timer (ms)	Off	5ms started	5ms running	Cancelled	5ms started	Expired	Off
Batch Content	[]	[Msg1]	[Msg1, Msg2]	[Msg1, Msg2, Msg3]	[Msg4]	[]	[]

Key Moments - 3 Insights

Why does the batch send immediately at step 3 instead of waiting for the linger timer?

Because the batch size exceeded the configured limit (17000 > 16384 bytes), the batch sends immediately, cancelling the linger timer as shown in step 3 of the execution_table.

What happens if no new messages arrive before the linger timer expires?

The linger timer triggers sending the current batch even if it is not full, as shown at step 5 where the timer expires and the batch with Msg4 is sent.

Does the linger timer start again after sending a batch?

No, after sending a batch, the batch buffer clears and the linger timer is off until a new message arrives, starting a new timer as shown at step 6.

Visual Quiz - 3 Questions

Look at the execution_table, what is the batch size after step 2?

A 4000 bytes

B 17000 bytes

C 9000 bytes

D 3000 bytes

💡 Hint

Check the 'Batch Size (bytes)' column at step 2 in the execution_table.

At which step does the linger timer get cancelled due to batch size?

A Step 3

B Step 1

C Step 2

D Step 5

💡 Hint

Look for 'Cancel timer' action in the execution_table.

If the linger.ms was set to 0, what would happen to the batch sending behavior?

A Batches would send immediately when batch size is reached only.

B Batches would send immediately after every message without waiting.

C Batches would never send.

D Batches would send only when linger timer expires.

💡 Hint

Consider how linger.ms controls waiting time before sending batches.

Concept Snapshot

Batching collects messages up to batch.size bytes.
linger.ms sets max wait time before sending batch.
Batch sends when size reached or linger expires.
Helps improve throughput by sending fewer larger requests.
Configurable in Kafka producer settings.

Full Transcript

This visual execution shows how Kafka producer batches messages before sending. Messages add to a batch buffer. If the batch size exceeds the configured batch.size, the batch sends immediately, cancelling any linger timer. If the batch size is not reached, the linger.ms timer starts on the first message and waits that time before sending the batch. This ensures messages are sent timely even if batch size is small. Variables tracked include batch size in bytes, linger timer state, and batch content. Key moments clarify why batches send immediately on size limit and how linger timer triggers sending when no new messages arrive. The quiz tests understanding of batch size changes, timer cancellation, and effects of linger.ms settings. This helps beginners see how batching and linger configuration control message sending in Kafka producers.

Practice

1. What does the linger.ms configuration control in Kafka producer settings?

easy

A. The maximum size of a message batch in bytes

B. The maximum time to wait before sending a batch of messages

C. The number of retries for sending a message

D. The timeout for consumer group rebalancing

Solution

1. Step 1: Understand the role of linger.ms

   The linger.ms setting defines how long the producer waits before sending a batch, allowing more messages to accumulate.

2. Step 2: Differentiate from other settings

   Other options like batch size control bytes, retries control attempts, and consumer timeout are unrelated to linger time.

3. Final Answer:

   The maximum time to wait before sending a batch of messages -> Option B

4. Quick Check:

   linger.ms = wait time before batch send [OK]

Hint: linger.ms sets wait time before sending batch [OK]

Common Mistakes:

• Confusing linger.ms with batch size
• Thinking linger.ms controls retries
• Mixing producer and consumer settings

2. Which of the following is the correct way to set the batch size to 32 KB in Kafka producer configuration?

easy

A. "batch.size": 32768

B. "batch.size": "32KB"

C. "batch.size": 32000

D. "batch.size": "32kb"

Solution

1. Step 1: Understand batch.size unit

   The batch.size is specified in bytes, so 32 KB equals 32768 bytes.

2. Step 2: Check option formats

   Options with strings like "32KB" or "32kb" are invalid because batch.size expects an integer number of bytes.

3. Final Answer:

   "batch.size": 32768 -> Option A

4. Quick Check:

   batch.size = bytes number, 32 KB = 32768 [OK]

Hint: batch.size must be integer bytes, not string [OK]

Common Mistakes:

• Using string values instead of integers
• Confusing KB with bytes count
• Using lowercase 'kb' which is invalid

3. Given the following Kafka producer settings:

{
  "batch.size": 16384,
  "linger.ms": 100
}

If messages arrive every 10 ms, what is the expected behavior?

medium

A. Messages are sent immediately without batching

B. Messages are delayed indefinitely until batch size is reached

C. Messages are sent only when batch size reaches 16384 bytes, ignoring linger.ms

D. Messages are batched and sent every 100 ms or when batch size is reached

Solution

1. Step 1: Analyze linger.ms and batch.size interaction

   The producer waits up to 100 ms (linger.ms) to accumulate messages or sends earlier if batch size (16384 bytes) is reached.

2. Step 2: Consider message arrival rate

   Messages arrive every 10 ms, so batching will likely happen before 100 ms or batch size limit, whichever comes first.

3. Final Answer:

   Messages are batched and sent every 100 ms or when batch size is reached -> Option D

4. Quick Check:

   linger.ms = max wait, batch.size = max batch [OK]

Hint: linger.ms sets max wait, batch.size sets max batch [OK]

Common Mistakes:

• Ignoring linger.ms and assuming immediate send
• Thinking linger.ms is minimum wait time
• Assuming messages delay indefinitely

4. You configured Kafka producer with:

{
  "batch.size": 0,
  "linger.ms": 500
}

What is the problem with this configuration?

medium

A. batch.size cannot be zero; it disables batching causing errors

B. linger.ms must be zero or less to disable delay

C. batch.size zero means messages are sent immediately without batching

D. No problem; this configuration is valid and efficient

Solution

1. Step 1: Understand batch.size = 0 effect

   Setting batch.size to zero disables batching, so each message is sent immediately.

2. Step 2: Check linger.ms impact

   Even with linger.ms = 500 ms, messages won't batch because batch.size = 0 forces immediate send.

3. Final Answer:

   batch.size zero means messages are sent immediately without batching -> Option C

4. Quick Check:

   batch.size=0 disables batching, linger.ms ignored [OK]

Hint: batch.size=0 disables batching, sends immediately [OK]

Common Mistakes:

• Assuming batch.size=0 causes errors
• Thinking linger.ms overrides batch.size=0
• Believing this config improves efficiency

5. You want to optimize Kafka producer for low latency but still batch messages efficiently. Which configuration is best?

hard

A. "batch.size": 65536, "linger.ms": 5

B. "batch.size": 16384, "linger.ms": 500

C. "batch.size": 0, "linger.ms": 0

D. "batch.size": 32768, "linger.ms": 1000

Solution

1. Step 1: Define low latency and efficient batching

   Low latency means sending messages quickly; efficient batching means grouping enough messages to reduce overhead.

2. Step 2: Evaluate options

   "batch.size": 65536, "linger.ms": 5 uses a large batch size (65536 bytes) and a very low linger.ms (5 ms), allowing quick sends with good batching. Other options have too high linger.ms or batch.size=0 disables batching.

3. Final Answer:

   "batch.size": 65536, "linger.ms": 5 -> Option A

4. Quick Check:

   Low linger.ms + large batch = low latency + efficient batching [OK]

Hint: Low linger.ms + large batch size = fast and efficient [OK]

Common Mistakes:

• Choosing linger.ms too high for low latency
• Setting batch.size to zero disables batching
• Ignoring balance between batch size and linger.ms