Batch size and compression tuning in Kafka - Mini Project: Build & Apply

Batch Size and Compression Tuning in Kafka Producer

📖 Scenario: You are working on a Kafka producer application that sends messages to a Kafka topic. To improve performance and reduce network usage, you want to tune the batch size and enable compression.This tuning helps in real-world scenarios where sending many small messages individually is slow and costly. Grouping messages into batches and compressing them makes the process faster and more efficient.

🎯 Goal: Build a Kafka producer configuration that sets a specific batch.size and enables compression.type to compress messages before sending.You will create the initial producer configuration, add tuning parameters, implement the producer sending logic, and finally print the configuration to verify your settings.

📋 What You'll Learn

Create a Kafka producer configuration dictionary with basic settings

Add batch.size set to 16384 bytes

Add compression.type set to 'gzip'

Implement a simple message sending loop using the configuration

Print the final producer configuration dictionary

💡 Why This Matters

🌍 Real World

Tuning batch size and compression in Kafka producers helps reduce network load and improve throughput in real-time data pipelines.

💼 Career

Kafka producer tuning is a common task for data engineers and backend developers working with streaming data systems.

1

Create the initial Kafka producer configuration

Create a dictionary called producer_config with these exact entries: 'bootstrap.servers': 'localhost:9092' and 'key.serializer': 'org.apache.kafka.common.serialization.StringSerializer' and 'value.serializer': 'org.apache.kafka.common.serialization.StringSerializer'.

Kafka

# Create the initial Kafka producer configuration dictionary
# Your code here

Need a hint?

Use a Python dictionary with the exact keys and values as shown.

2

Add batch size and compression settings

Add two entries to the producer_config dictionary: 'batch.size' set to 16384 and 'compression.type' set to 'gzip'.

Kafka

producer_config = {
    'bootstrap.servers': 'localhost:9092',
    'key.serializer': 'org.apache.kafka.common.serialization.StringSerializer',
    'value.serializer': 'org.apache.kafka.common.serialization.StringSerializer'
}
# Add batch.size and compression.type to producer_config
# Your code here

Need a hint?

Add the new keys and values inside the existing dictionary.

3

Implement message sending loop

Write a for loop that iterates over range(3) and inside the loop, create a variable message with the value f'Message {i}' where i is the loop variable. (This simulates sending messages.)

Kafka

producer_config = {
    'bootstrap.servers': 'localhost:9092',
    'key.serializer': 'org.apache.kafka.common.serialization.StringSerializer',
    'value.serializer': 'org.apache.kafka.common.serialization.StringSerializer',
    'batch.size': 16384,
    'compression.type': 'gzip'
}
# Write a for loop to simulate sending 3 messages
# Your code here

Need a hint?

Use a for loop with variable i and create the message string inside it.

4

Print the final producer configuration

Write print(producer_config) to display the final producer configuration dictionary.

Kafka

producer_config = {
    'bootstrap.servers': 'localhost:9092',
    'key.serializer': 'org.apache.kafka.common.serialization.StringSerializer',
    'value.serializer': 'org.apache.kafka.common.serialization.StringSerializer',
    'batch.size': 16384,
    'compression.type': 'gzip'
}

for i in range(3):
    message = f'Message {i}'

# Print the producer_config dictionary
# Your code here

Need a hint?

Use print() to show the dictionary exactly.