Batching and linger configuration in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Effect of linger.ms on message batching

Consider a Kafka producer configured with linger.ms=100. The producer sends messages every 50ms. What is the expected behavior regarding message batching?

Kafka

producer_config = {
  'linger.ms': 100
}

# Messages sent every 50ms
# How does linger.ms affect batching here?

A Messages are sent immediately without batching because 50ms < linger.ms

B Messages are delayed indefinitely until batch size is reached

C Messages are batched and sent every 100ms or when batch is full

D Messages are sent every 50ms ignoring linger.ms

💡 Hint

Think about how linger.ms controls the maximum wait time before sending a batch.

The linger.ms setting specifies the maximum time to wait before sending a batch. Even if messages arrive earlier, the producer waits up to 100ms to batch more messages. Since messages come every 50ms, they accumulate and are sent every 100ms or when the batch is full.

❓ Predict Output

intermediate

Batch size effect on Kafka producer throughput

A Kafka producer has batch.size=16384 bytes and linger.ms=0. If the producer sends messages smaller than 1KB each rapidly, what is the expected behavior?

Kafka

producer_config = {
  'batch.size': 16384,  # 16 KB
  'linger.ms': 0
}

# Messages are ~1KB each, sent rapidly

A Messages are dropped if batch size is not reached

B Messages are batched up to 16KB before sending

C Messages are delayed up to linger.ms before sending

D Each message is sent immediately without batching

💡 Hint

What does linger.ms=0 imply about waiting for batch completion?

With linger.ms=0, the producer sends messages immediately without waiting to fill the batch. Even if batch.size is large, messages are sent as soon as they are ready.

🔧 Debug

advanced

Identify the cause of delayed message sending

A Kafka producer is configured with linger.ms=500 and batch.size=32768. However, messages are delayed longer than 500ms before sending. What is the most likely cause?

Kafka

producer_config = {
  'linger.ms': 500,
  'batch.size': 32768
}

# Messages are sent slowly, delays > 500ms observed

A The producer is waiting for acknowledgments before sending next batch

B The batch size is too large, causing delays waiting for batch to fill

C linger.ms is ignored if batch.size is not reached

D The network latency causes linger.ms to be ineffective

💡 Hint

Consider how batch.size and linger.ms interact to trigger sending.

The producer waits up to linger.ms before sending a batch, but if the batch size is large and messages arrive slowly, the batch may not fill quickly. This can cause delays longer than linger.ms if the producer is configured to wait for batch filling.

📝 Syntax

advanced

Correct Kafka producer configuration syntax for batching

Which of the following Kafka producer configuration snippets correctly sets batch.size to 32KB and linger.ms to 100ms in Java?

A

props.put("batch.size", 32 * 1024);
props.put("linger.ms", 100);

B

props.put("batch.size", "32768");
props.put("linger.ms", "100");

C

props.put(batch.size, 32768);
props.put(linger.ms, 100);

D

props.put("batch.size", 32768);
props.put("linger.ms", 100);

💡 Hint

Check for correct key names as strings and valid value types.

In Java Kafka producer configs, keys must be strings. Values can be integers or strings. Option A uses a valid expression for 32KB and correct string keys.

🚀 Application

expert

Optimizing Kafka producer for low latency and high throughput

You want to configure a Kafka producer to achieve low latency but still benefit from batching to improve throughput. Which configuration is best?

A Set linger.ms to a small positive value (e.g., 5ms) and batch.size to a moderate value

B Set linger.ms to 0 and batch.size to a large value

C Set linger.ms to a large value (e.g., 500ms) and batch.size to a small value

D Set linger.ms to a large value and batch.size to a large value

💡 Hint

Think about balancing wait time and batch size for latency and throughput.

Setting linger.ms to a small positive value allows the producer to wait briefly to batch messages, improving throughput without adding much latency. A moderate batch size complements this by not requiring too many messages to send a batch.

Practice

1. What does the linger.ms configuration control in Kafka producer settings?

easy

A. The maximum size of a message batch in bytes

B. The maximum time to wait before sending a batch of messages

C. The number of retries for sending a message

D. The timeout for consumer group rebalancing

Solution

1. Step 1: Understand the role of linger.ms

   The linger.ms setting defines how long the producer waits before sending a batch, allowing more messages to accumulate.

2. Step 2: Differentiate from other settings

   Other options like batch size control bytes, retries control attempts, and consumer timeout are unrelated to linger time.

3. Final Answer:

   The maximum time to wait before sending a batch of messages -> Option B

4. Quick Check:

   linger.ms = wait time before batch send [OK]

Hint: linger.ms sets wait time before sending batch [OK]

Common Mistakes:

• Confusing linger.ms with batch size
• Thinking linger.ms controls retries
• Mixing producer and consumer settings

2. Which of the following is the correct way to set the batch size to 32 KB in Kafka producer configuration?

easy

A. "batch.size": 32768

B. "batch.size": "32KB"

C. "batch.size": 32000

D. "batch.size": "32kb"

Solution

1. Step 1: Understand batch.size unit

   The batch.size is specified in bytes, so 32 KB equals 32768 bytes.

2. Step 2: Check option formats

   Options with strings like "32KB" or "32kb" are invalid because batch.size expects an integer number of bytes.

3. Final Answer:

   "batch.size": 32768 -> Option A

4. Quick Check:

   batch.size = bytes number, 32 KB = 32768 [OK]

Hint: batch.size must be integer bytes, not string [OK]

Common Mistakes:

• Using string values instead of integers
• Confusing KB with bytes count
• Using lowercase 'kb' which is invalid

3. Given the following Kafka producer settings:

{
  "batch.size": 16384,
  "linger.ms": 100
}

If messages arrive every 10 ms, what is the expected behavior?

medium

A. Messages are sent immediately without batching

B. Messages are delayed indefinitely until batch size is reached

C. Messages are sent only when batch size reaches 16384 bytes, ignoring linger.ms

D. Messages are batched and sent every 100 ms or when batch size is reached

Solution

1. Step 1: Analyze linger.ms and batch.size interaction

   The producer waits up to 100 ms (linger.ms) to accumulate messages or sends earlier if batch size (16384 bytes) is reached.

2. Step 2: Consider message arrival rate

   Messages arrive every 10 ms, so batching will likely happen before 100 ms or batch size limit, whichever comes first.

3. Final Answer:

   Messages are batched and sent every 100 ms or when batch size is reached -> Option D

4. Quick Check:

   linger.ms = max wait, batch.size = max batch [OK]

Hint: linger.ms sets max wait, batch.size sets max batch [OK]

Common Mistakes:

• Ignoring linger.ms and assuming immediate send
• Thinking linger.ms is minimum wait time
• Assuming messages delay indefinitely

4. You configured Kafka producer with:

{
  "batch.size": 0,
  "linger.ms": 500
}

What is the problem with this configuration?

medium

A. batch.size cannot be zero; it disables batching causing errors

B. linger.ms must be zero or less to disable delay

C. batch.size zero means messages are sent immediately without batching

D. No problem; this configuration is valid and efficient

Solution

1. Step 1: Understand batch.size = 0 effect

   Setting batch.size to zero disables batching, so each message is sent immediately.

2. Step 2: Check linger.ms impact

   Even with linger.ms = 500 ms, messages won't batch because batch.size = 0 forces immediate send.

3. Final Answer:

   batch.size zero means messages are sent immediately without batching -> Option C

4. Quick Check:

   batch.size=0 disables batching, linger.ms ignored [OK]

Hint: batch.size=0 disables batching, sends immediately [OK]

Common Mistakes:

• Assuming batch.size=0 causes errors
• Thinking linger.ms overrides batch.size=0
• Believing this config improves efficiency

5. You want to optimize Kafka producer for low latency but still batch messages efficiently. Which configuration is best?

hard

A. "batch.size": 65536, "linger.ms": 5

B. "batch.size": 16384, "linger.ms": 500

C. "batch.size": 0, "linger.ms": 0

D. "batch.size": 32768, "linger.ms": 1000

Solution

1. Step 1: Define low latency and efficient batching

   Low latency means sending messages quickly; efficient batching means grouping enough messages to reduce overhead.

2. Step 2: Evaluate options

   "batch.size": 65536, "linger.ms": 5 uses a large batch size (65536 bytes) and a very low linger.ms (5 ms), allowing quick sends with good batching. Other options have too high linger.ms or batch.size=0 disables batching.

3. Final Answer:

   "batch.size": 65536, "linger.ms": 5 -> Option A

4. Quick Check:

   Low linger.ms + large batch = low latency + efficient batching [OK]

Hint: Low linger.ms + large batch size = fast and efficient [OK]

Common Mistakes:

• Choosing linger.ms too high for low latency
• Setting batch.size to zero disables batching
• Ignoring balance between batch size and linger.ms