Batching and linger configuration in Kafka - Mini Project: Build & Apply

Batching and Linger Configuration in Kafka Producer

📖 Scenario: You are building a Kafka producer that sends messages to a topic. To improve performance, you want to configure batching and linger time so that messages are sent in groups instead of one by one.

🎯 Goal: Configure a Kafka producer with specific batching and linger settings, send a few messages, and observe the batching effect.

📋 What You'll Learn

Create a Kafka producer configuration dictionary with bootstrap servers

Add batching configuration with batch.size set to 16384 bytes

Add linger configuration with linger.ms set to 10 milliseconds

Send three messages to the topic test-topic

Print confirmation after sending messages

💡 Why This Matters

🌍 Real World

Batching and linger settings help reduce network overhead and improve throughput when sending many messages to Kafka in real applications like logging, metrics, or event streaming.

💼 Career

Understanding Kafka producer configuration is important for roles in data engineering, backend development, and real-time data processing where efficient message delivery is critical.

1

Create Kafka producer configuration

Create a dictionary called producer_config with the key 'bootstrap.servers' set to 'localhost:9092'.

Kafka

# Create the producer_config dictionary with bootstrap.servers
# Your code here

Hint

The configuration dictionary must have the key 'bootstrap.servers' with value 'localhost:9092'.

2

Add batching and linger configuration

Add two entries to the producer_config dictionary: 'batch.size' set to 16384 and 'linger.ms' set to 10.

Kafka

producer_config = {'bootstrap.servers': 'localhost:9092'}
# Add 'batch.size' = 16384 and 'linger.ms' = 10 to producer_config
# Your code here

Hint

Use dictionary key assignment to add 'batch.size' and 'linger.ms' to producer_config.

3

Send messages using the configured producer

Create a Kafka producer using producer_config. Send three messages with values 'message 1', 'message 2', and 'message 3' to the topic 'test-topic'. Use the variable name producer for the Kafka producer.

Kafka

producer_config = {'bootstrap.servers': 'localhost:9092'}
producer_config['batch.size'] = 16384
producer_config['linger.ms'] = 10
# Create Kafka producer and send three messages to 'test-topic'
# Your code here

Hint

Use Producer(producer_config) to create the producer. Use produce method to send messages. Call flush() to ensure all messages are sent.

4

Print confirmation of sent messages

Print the exact text "Sent 3 messages to test-topic with batching and linger settings.".

Kafka

from confluent_kafka import Producer

producer_config = {'bootstrap.servers': 'localhost:9092'}
producer_config['batch.size'] = 16384
producer_config['linger.ms'] = 10

producer = Producer(producer_config)

producer.produce('test-topic', value='message 1')
producer.produce('test-topic', value='message 2')
producer.produce('test-topic', value='message 3')

producer.flush()
# Print confirmation message
# Your code here

Hint

Use print() to display the confirmation message exactly as shown.

Practice

1. What does the linger.ms configuration control in Kafka producer settings?

easy

A. The maximum size of a message batch in bytes

B. The maximum time to wait before sending a batch of messages

C. The number of retries for sending a message

D. The timeout for consumer group rebalancing

Solution

1. Step 1: Understand the role of linger.ms

   The linger.ms setting defines how long the producer waits before sending a batch, allowing more messages to accumulate.

2. Step 2: Differentiate from other settings

   Other options like batch size control bytes, retries control attempts, and consumer timeout are unrelated to linger time.

3. Final Answer:

   The maximum time to wait before sending a batch of messages -> Option B

4. Quick Check:

   linger.ms = wait time before batch send [OK]

Hint: linger.ms sets wait time before sending batch [OK]

Common Mistakes:

• Confusing linger.ms with batch size
• Thinking linger.ms controls retries
• Mixing producer and consumer settings

2. Which of the following is the correct way to set the batch size to 32 KB in Kafka producer configuration?

easy

A. "batch.size": 32768

B. "batch.size": "32KB"

C. "batch.size": 32000

D. "batch.size": "32kb"

Solution

1. Step 1: Understand batch.size unit

   The batch.size is specified in bytes, so 32 KB equals 32768 bytes.

2. Step 2: Check option formats

   Options with strings like "32KB" or "32kb" are invalid because batch.size expects an integer number of bytes.

3. Final Answer:

   "batch.size": 32768 -> Option A

4. Quick Check:

   batch.size = bytes number, 32 KB = 32768 [OK]

Hint: batch.size must be integer bytes, not string [OK]

Common Mistakes:

• Using string values instead of integers
• Confusing KB with bytes count
• Using lowercase 'kb' which is invalid

3. Given the following Kafka producer settings:

{
  "batch.size": 16384,
  "linger.ms": 100
}

If messages arrive every 10 ms, what is the expected behavior?

medium

A. Messages are sent immediately without batching

B. Messages are delayed indefinitely until batch size is reached

C. Messages are sent only when batch size reaches 16384 bytes, ignoring linger.ms

D. Messages are batched and sent every 100 ms or when batch size is reached

Solution

1. Step 1: Analyze linger.ms and batch.size interaction

   The producer waits up to 100 ms (linger.ms) to accumulate messages or sends earlier if batch size (16384 bytes) is reached.

2. Step 2: Consider message arrival rate

   Messages arrive every 10 ms, so batching will likely happen before 100 ms or batch size limit, whichever comes first.

3. Final Answer:

   Messages are batched and sent every 100 ms or when batch size is reached -> Option D

4. Quick Check:

   linger.ms = max wait, batch.size = max batch [OK]

Hint: linger.ms sets max wait, batch.size sets max batch [OK]

Common Mistakes:

• Ignoring linger.ms and assuming immediate send
• Thinking linger.ms is minimum wait time
• Assuming messages delay indefinitely

4. You configured Kafka producer with:

{
  "batch.size": 0,
  "linger.ms": 500
}

What is the problem with this configuration?

medium

A. batch.size cannot be zero; it disables batching causing errors

B. linger.ms must be zero or less to disable delay

C. batch.size zero means messages are sent immediately without batching

D. No problem; this configuration is valid and efficient

Solution

1. Step 1: Understand batch.size = 0 effect

   Setting batch.size to zero disables batching, so each message is sent immediately.

2. Step 2: Check linger.ms impact

   Even with linger.ms = 500 ms, messages won't batch because batch.size = 0 forces immediate send.

3. Final Answer:

   batch.size zero means messages are sent immediately without batching -> Option C

4. Quick Check:

   batch.size=0 disables batching, linger.ms ignored [OK]

Hint: batch.size=0 disables batching, sends immediately [OK]

Common Mistakes:

• Assuming batch.size=0 causes errors
• Thinking linger.ms overrides batch.size=0
• Believing this config improves efficiency

5. You want to optimize Kafka producer for low latency but still batch messages efficiently. Which configuration is best?

hard

A. "batch.size": 65536, "linger.ms": 5

B. "batch.size": 16384, "linger.ms": 500

C. "batch.size": 0, "linger.ms": 0

D. "batch.size": 32768, "linger.ms": 1000

Solution

1. Step 1: Define low latency and efficient batching

   Low latency means sending messages quickly; efficient batching means grouping enough messages to reduce overhead.

2. Step 2: Evaluate options

   "batch.size": 65536, "linger.ms": 5 uses a large batch size (65536 bytes) and a very low linger.ms (5 ms), allowing quick sends with good batching. Other options have too high linger.ms or batch.size=0 disables batching.

3. Final Answer:

   "batch.size": 65536, "linger.ms": 5 -> Option A

4. Quick Check:

   Low linger.ms + large batch = low latency + efficient batching [OK]

Hint: Low linger.ms + large batch size = fast and efficient [OK]

Common Mistakes:

• Choosing linger.ms too high for low latency
• Setting batch.size to zero disables batching
• Ignoring balance between batch size and linger.ms