Producer throughput optimization in Kafka - Mini Project: Build & Apply

Producer throughput optimization

📖 Scenario: You are working with Apache Kafka to send messages from a producer to a topic. To improve how fast your producer sends messages, you want to optimize the throughput by adjusting some settings.

🎯 Goal: Build a Kafka producer configuration that optimizes throughput by setting batch size and linger time, then send a batch of messages and print a confirmation.

📋 What You'll Learn

Create a Kafka producer configuration dictionary with specific settings

Add a batch size setting to control message batching

Add a linger.ms setting to delay sending to increase batch size

Send multiple messages using the producer

Print a confirmation message after sending

💡 Why This Matters

🌍 Real World

Optimizing Kafka producer throughput helps in real-time data pipelines where sending messages quickly and efficiently is important.

💼 Career

Kafka producer configuration and tuning is a common task for data engineers and backend developers working with streaming data.

1

Create Kafka producer configuration

Create a dictionary called producer_config with these exact entries: bootstrap.servers set to "localhost:9092" and acks set to "all".

Kafka

# Create the producer_config dictionary with bootstrap.servers and acks
# Your code here

Need a hint?

Use a Python dictionary with keys as strings and values as strings.

2

Add batch size setting

Add a new entry to producer_config with key batch.size and value 16384 (an integer).

Kafka

producer_config = {
    "bootstrap.servers": "localhost:9092",
    "acks": "all"
}
# Add batch.size to producer_config
# Your code here

Need a hint?

Remember to add a comma after the previous entry.

3

Add linger.ms setting

Add a new entry to producer_config with key linger.ms and value 5 (an integer).

Kafka

producer_config = {
    "bootstrap.servers": "localhost:9092",
    "acks": "all",
    "batch.size": 16384
}
# Add linger.ms to producer_config
# Your code here

Need a hint?

Adding linger.ms helps the producer wait a bit to batch more messages.

4

Send messages and print confirmation

Using the producer_config, create a Kafka producer object called producer. Then send three messages with keys "key1", "key2", "key3" and values "message1", "message2", "message3" to the topic "test-topic". Finally, print "Messages sent successfully".

Kafka

producer_config = {
    "bootstrap.servers": "localhost:9092",
    "acks": "all",
    "batch.size": 16384,
    "linger.ms": 5
}
# Create producer and send messages
# Your code here

Need a hint?

Use Producer from confluent_kafka, call produce for each message, then flush() to send all.