Exactly-once semantics (EOS) in Kafka - Mini Project: Build & Apply

Exactly-once semantics (EOS) with Kafka Producer

📖 Scenario: You are building a simple Kafka producer application that sends messages to a Kafka topic. To ensure data integrity, you want to guarantee that each message is delivered exactly once, even if retries happen due to network issues.

🎯 Goal: Create a Kafka producer configuration that enables exactly-once semantics (EOS) and send a few messages to a Kafka topic named eos_topic.

📋 What You'll Learn

Create a Kafka producer configuration with enable.idempotence set to true

Set a transactional.id to enable transactions

Initialize transactions before sending messages

Send three messages to the topic eos_topic within a transaction

Commit the transaction to ensure exactly-once delivery

Print a confirmation message after successful commit

💡 Why This Matters

🌍 Real World

Exactly-once semantics are critical in financial systems, order processing, and any application where duplicate messages can cause errors or inconsistencies.

💼 Career

Kafka developers and data engineers often need to implement EOS to guarantee data integrity in distributed streaming applications.

1

Create Kafka producer configuration with EOS settings

Create a dictionary called producer_config with these exact entries: 'bootstrap.servers': 'localhost:9092', 'enable.idempotence': True, and 'transactional.id': 'my-transactional-id'.

Kafka

# Create the Kafka producer configuration dictionary
# Your code here

Need a hint?

Use a Python dictionary with the exact keys and values as specified.

2

Initialize Kafka producer and transactions

Import Producer from confluent_kafka. Create a Producer instance called producer using producer_config. Then call producer.init_transactions() to initialize transactions.

Kafka

producer_config = {
    'bootstrap.servers': 'localhost:9092',
    'enable.idempotence': True,
    'transactional.id': 'my-transactional-id'
}

# Import Producer and create producer instance
# Initialize transactions
# Your code here

Need a hint?

Remember to import Producer and call init_transactions() on the producer instance.

3

Send messages within a transaction

Use producer.begin_transaction() to start a transaction. Then send three messages with keys 'key1', 'key2', 'key3' and values 'message1', 'message2', 'message3' respectively to the topic 'eos_topic' using producer.produce(). Finally, call producer.commit_transaction() to commit the transaction.

Kafka

from confluent_kafka import Producer

producer_config = {
    'bootstrap.servers': 'localhost:9092',
    'enable.idempotence': True,
    'transactional.id': 'my-transactional-id'
}

producer = Producer(producer_config)
producer.init_transactions()

# Begin transaction
# Produce three messages to 'eos_topic'
# Commit transaction
# Your code here

Need a hint?

Use begin_transaction() before producing messages and commit_transaction() after.

4

Print confirmation after commit

Add a print statement that outputs exactly "Transaction committed successfully!" after the transaction commit.

Kafka

from confluent_kafka import Producer

producer_config = {
    'bootstrap.servers': 'localhost:9092',
    'enable.idempotence': True,
    'transactional.id': 'my-transactional-id'
}

producer = Producer(producer_config)
producer.init_transactions()

producer.begin_transaction()
producer.produce('eos_topic', key='key1', value='message1')
producer.produce('eos_topic', key='key2', value='message2')
producer.produce('eos_topic', key='key3', value='message3')
producer.commit_transaction()

# Print confirmation message
# Your code here

Need a hint?

Use print("Transaction committed successfully!") exactly.