Producer retries and idempotency in Kafka - Mini Project: Build & Apply

Producer retries and idempotency

📖 Scenario: You are building a Kafka producer that sends messages to a topic. Sometimes, network issues cause message sending to fail temporarily. To avoid sending duplicate messages when retrying, you want to enable idempotency and configure retries.

🎯 Goal: Create a Kafka producer with retries enabled and idempotency turned on. Send a message and print confirmation that the message was sent successfully.

📋 What You'll Learn

Create a Kafka producer configuration dictionary with bootstrap servers

Add configuration to enable retries with a count of 3

Enable idempotency in the producer configuration

Send a message to a topic named 'test-topic' and print confirmation

💡 Why This Matters

🌍 Real World

In real applications, network issues or broker failures can cause message sending to fail temporarily. Enabling retries and idempotency ensures messages are not lost or duplicated, which is critical for data accuracy.

💼 Career

Kafka producers with retries and idempotency are common in data engineering and backend development roles where reliable data pipelines are essential.

1

Create Kafka producer configuration

Create a dictionary called producer_config with the key 'bootstrap.servers' set to 'localhost:9092'.

Kafka

# Create the producer_config dictionary with bootstrap.servers
# Your code here

Need a hint?

The configuration dictionary must have the key 'bootstrap.servers' with value 'localhost:9092'.

2

Add retries and enable idempotency

Add to producer_config the key 'retries' with value 3 and the key 'enable.idempotence' with value True.

Kafka

producer_config = {'bootstrap.servers': 'localhost:9092'}
# Add 'retries' = 3 and 'enable.idempotence' = True to producer_config
# Your code here

Need a hint?

Use dictionary assignment to add 'retries' and 'enable.idempotence' keys with the correct values.

3

Create producer and send message

Import Producer from confluent_kafka. Create a Producer instance called producer using producer_config. Use producer.produce() to send the message 'Hello, Kafka!' to the topic 'test-topic'. Call producer.flush() to ensure delivery.

Kafka

producer_config = {'bootstrap.servers': 'localhost:9092'}
producer_config['retries'] = 3
producer_config['enable.idempotence'] = True
# Import Producer and create producer instance
# Send 'Hello, Kafka!' to 'test-topic'
# Flush the producer
# Your code here

Need a hint?

Remember to import Producer, create it with the config, produce the message, and flush.

4

Print confirmation message

Write a print statement that outputs exactly 'Message sent successfully.' after the message is flushed.

Kafka

from confluent_kafka import Producer

producer_config = {'bootstrap.servers': 'localhost:9092'}
producer_config['retries'] = 3
producer_config['enable.idempotence'] = True

producer = Producer(producer_config)
producer.produce('test-topic', value='Hello, Kafka!')
producer.flush()
# Print confirmation message
# Your code here

Need a hint?

Use print() to show the confirmation text exactly as given.