Configuration best practices in Kafka - Mini Project: Build & Apply

Kafka Configuration Best Practices

📖 Scenario: You are setting up a Kafka producer to send messages to a topic. Proper configuration is important to ensure reliable message delivery and good performance.

🎯 Goal: You will create a Kafka producer configuration dictionary, add a configuration setting for retries, apply the configuration to create a producer, and print the configuration to verify it.

📋 What You'll Learn

Create a dictionary called producer_config with specific Kafka settings

Add a configuration setting called retries with value 5

Create a Kafka producer using KafkaProducer with producer_config

Print the producer_config dictionary

💡 Why This Matters

🌍 Real World

Kafka producers need proper configuration to connect to Kafka brokers and send messages reliably in real-world applications like logging, monitoring, and event streaming.

💼 Career

Understanding Kafka configuration is important for roles in data engineering, backend development, and system architecture where message streaming is used.

1

Create initial Kafka producer configuration

Create a dictionary called producer_config with these exact entries: 'bootstrap_servers': 'localhost:9092', 'key_serializer': str.encode, and 'value_serializer': str.encode.

Kafka

# Create the producer_config dictionary with bootstrap_servers, key_serializer, and value_serializer
# Your code here

Need a hint?

Use curly braces {} to create a dictionary. Use str.encode for serializers.

2

Add retries configuration

Add a new entry to the producer_config dictionary with key 'retries' and value 5.

Kafka

producer_config = {
    'bootstrap_servers': 'localhost:9092',
    'key_serializer': str.encode,
    'value_serializer': str.encode
}
# Add 'retries' with value 5 to producer_config
# Your code here

Need a hint?

Use square brackets [] to add a new key-value pair to the dictionary.

3

Create Kafka producer with configuration

Import KafkaProducer from kafka and create a variable called producer by passing producer_config as keyword arguments using **producer_config.

Kafka

producer_config = {
    'bootstrap_servers': 'localhost:9092',
    'key_serializer': str.encode,
    'value_serializer': str.encode
}
producer_config['retries'] = 5
# Import KafkaProducer and create producer using producer_config
# Your code here

Need a hint?

Use from kafka import KafkaProducer to import. Use **producer_config to unpack the dictionary.

4

Print the producer configuration

Write a print statement to display the producer_config dictionary.

Kafka

from kafka import KafkaProducer

producer_config = {
    'bootstrap_servers': 'localhost:9092',
    'key_serializer': str.encode,
    'value_serializer': str.encode
}
producer_config['retries'] = 5

producer = KafkaProducer(**producer_config)
# Print the producer_config dictionary
# Your code here

Need a hint?

Use print(producer_config) to show the dictionary.