Consumer offset commit strategies in Kafka - Mini Project: Build & Apply

Consumer offset commit strategies

📖 Scenario: You are building a Kafka consumer application that reads messages from a topic. To ensure your application processes messages reliably, you need to manage how and when the consumer commits the offsets of messages it has processed.This project will guide you through setting up a Kafka consumer, configuring offset commit strategies, and printing the committed offsets.

🎯 Goal: Build a Kafka consumer that reads messages from a topic, uses a specific offset commit strategy, and prints the committed offsets to verify the strategy works.

📋 What You'll Learn

Create a Kafka consumer configuration with a specific group id and offset reset policy

Add a configuration variable to control the offset commit strategy (automatic or manual)

Implement the consumer logic to commit offsets according to the chosen strategy

Print the committed offsets after processing messages

💡 Why This Matters

🌍 Real World

Kafka consumers must manage offset commits to avoid processing messages multiple times or missing messages. Choosing the right commit strategy helps build reliable data pipelines.

💼 Career

Understanding consumer offset commit strategies is essential for roles involving real-time data processing, streaming applications, and building fault-tolerant systems using Kafka.

1

Create Kafka consumer configuration

Create a dictionary called consumer_config with these exact entries: 'bootstrap.servers': 'localhost:9092', 'group.id': 'test-group', and 'auto.offset.reset': 'earliest'.

Kafka

# Create the consumer_config dictionary with bootstrap.servers, group.id, and auto.offset.reset
# Your code here

Need a hint?

Use a Python dictionary with the exact keys and values as shown.

2

Add offset commit strategy configuration

Add a variable called commit_strategy and set it to the string 'auto' to represent automatic offset commits.

Kafka

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'test-group',
    'auto.offset.reset': 'earliest'
}
# Set commit_strategy to 'auto' or 'manual'
# Your code here

Need a hint?

Just create a variable named commit_strategy and assign it the string 'auto'.

3

Implement consumer offset commit logic

Write a for loop that iterates over messages. Inside the loop, if commit_strategy is 'manual', call consumer.commit() after processing each message. Assume consumer and messages are predefined.

Kafka

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'test-group',
    'auto.offset.reset': 'earliest'
}
commit_strategy = 'auto'

# Process messages and commit offsets if commit_strategy is 'manual'
# Your code here

Need a hint?

Use a for loop over messages and an if statement to check commit_strategy.

4

Print committed offsets

Write a print statement to display the string 'Committed offsets:' followed by the result of consumer.committed().

Kafka

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'test-group',
    'auto.offset.reset': 'earliest'
}
commit_strategy = 'auto'

# Assume consumer and messages are predefined
for message in messages:
    # Process the message (placeholder)
    if commit_strategy == 'manual':
        consumer.commit()

# Print committed offsets
# Your code here

Need a hint?

Use print('Committed offsets:', consumer.committed()) to show the committed offsets.