Batching and linger configuration in Kafka - Mini Project: Build & Apply

Batching and Linger Configuration in Kafka Producer

📖 Scenario: You are building a Kafka producer that sends messages to a topic. To improve performance, you want to configure batching and linger time so that messages are sent in groups instead of one by one.

🎯 Goal: Configure a Kafka producer with specific batching and linger settings, send a few messages, and observe the batching effect.

📋 What You'll Learn

Create a Kafka producer configuration dictionary with bootstrap servers

Add batching configuration with batch.size set to 16384 bytes

Add linger configuration with linger.ms set to 10 milliseconds

Send three messages to the topic test-topic

Print confirmation after sending messages

💡 Why This Matters

🌍 Real World

Batching and linger settings help reduce network overhead and improve throughput when sending many messages to Kafka in real applications like logging, metrics, or event streaming.

💼 Career

Understanding Kafka producer configuration is important for roles in data engineering, backend development, and real-time data processing where efficient message delivery is critical.

1

Create Kafka producer configuration

Create a dictionary called producer_config with the key 'bootstrap.servers' set to 'localhost:9092'.

Kafka

# Create the producer_config dictionary with bootstrap.servers
# Your code here

Need a hint?

The configuration dictionary must have the key 'bootstrap.servers' with value 'localhost:9092'.

2

Add batching and linger configuration

Add two entries to the producer_config dictionary: 'batch.size' set to 16384 and 'linger.ms' set to 10.

Kafka

producer_config = {'bootstrap.servers': 'localhost:9092'}
# Add 'batch.size' = 16384 and 'linger.ms' = 10 to producer_config
# Your code here

Need a hint?

Use dictionary key assignment to add 'batch.size' and 'linger.ms' to producer_config.

3

Send messages using the configured producer

Create a Kafka producer using producer_config. Send three messages with values 'message 1', 'message 2', and 'message 3' to the topic 'test-topic'. Use the variable name producer for the Kafka producer.

Kafka

producer_config = {'bootstrap.servers': 'localhost:9092'}
producer_config['batch.size'] = 16384
producer_config['linger.ms'] = 10
# Create Kafka producer and send three messages to 'test-topic'
# Your code here

Need a hint?

Use Producer(producer_config) to create the producer. Use produce method to send messages. Call flush() to ensure all messages are sent.

4

Print confirmation of sent messages

Print the exact text "Sent 3 messages to test-topic with batching and linger settings.".

Kafka

from confluent_kafka import Producer

producer_config = {'bootstrap.servers': 'localhost:9092'}
producer_config['batch.size'] = 16384
producer_config['linger.ms'] = 10

producer = Producer(producer_config)

producer.produce('test-topic', value='message 1')
producer.produce('test-topic', value='message 2')
producer.produce('test-topic', value='message 3')

producer.flush()
# Print confirmation message
# Your code here

Need a hint?

Use print() to display the confirmation message exactly as shown.