Memory and buffer configuration in Kafka - Mini Project: Build & Apply

Memory and Buffer Configuration in Kafka Producer

📖 Scenario: You are setting up a Kafka producer to send messages efficiently. To do this, you need to configure memory and buffer settings properly.

🎯 Goal: Configure the Kafka producer's memory and buffer settings step-by-step to optimize message sending.

📋 What You'll Learn

Create a Kafka producer configuration dictionary with basic settings

Add a buffer memory size configuration

Add a batch size configuration

Print the final configuration dictionary

💡 Why This Matters

🌍 Real World

Kafka producers need memory and buffer settings to control how messages are batched and sent efficiently.

💼 Career

Configuring Kafka producer memory and buffers is a common task for backend developers and data engineers working with real-time data pipelines.

1

Create basic Kafka producer configuration

Create a dictionary called producer_config with these exact entries: 'bootstrap.servers': 'localhost:9092' and 'key.serializer': 'org.apache.kafka.common.serialization.StringSerializer' and 'value.serializer': 'org.apache.kafka.common.serialization.StringSerializer'.

Kafka

# Create the producer_config dictionary with bootstrap.servers, key.serializer, and value.serializer
# Your code here

Need a hint?

Use a Python dictionary with the exact keys and values given.

2

Add buffer memory size configuration

Add a new entry to the producer_config dictionary with key 'buffer.memory' and value 33554432 (32 MB).

Kafka

producer_config = {
    'bootstrap.servers': 'localhost:9092',
    'key.serializer': 'org.apache.kafka.common.serialization.StringSerializer',
    'value.serializer': 'org.apache.kafka.common.serialization.StringSerializer'
}
# Add buffer.memory configuration below
# Your code here

Need a hint?

Add the key 'buffer.memory' with the integer value 33554432 to the dictionary.

3

Add batch size configuration

Add a new entry to the producer_config dictionary with key 'batch.size' and value 16384 (16 KB).

Kafka

producer_config = {
    'bootstrap.servers': 'localhost:9092',
    'key.serializer': 'org.apache.kafka.common.serialization.StringSerializer',
    'value.serializer': 'org.apache.kafka.common.serialization.StringSerializer',
    'buffer.memory': 33554432
}
# Add batch.size configuration below
# Your code here

Need a hint?

Add the key 'batch.size' with the integer value 16384 to the dictionary.

4

Print the final Kafka producer configuration

Write a print statement to display the producer_config dictionary.

Kafka

producer_config = {
    'bootstrap.servers': 'localhost:9092',
    'key.serializer': 'org.apache.kafka.common.serialization.StringSerializer',
    'value.serializer': 'org.apache.kafka.common.serialization.StringSerializer',
    'buffer.memory': 33554432,
    'batch.size': 16384
}
# Print the producer_config dictionary below
# Your code here

Need a hint?

Use print(producer_config) to show the dictionary.