Why consumers process messages in Kafka - See It in Action

Why consumers process messages

📖 Scenario: Imagine you have a system where different parts talk to each other by sending messages. These messages are like letters in a mailbox. Consumers are the parts that read these letters and do something with the information.

🎯 Goal: Build a simple Kafka consumer that reads messages from a topic called orders and prints each message to understand why consumers process messages.

📋 What You'll Learn

Create a Kafka consumer that connects to a Kafka server

Subscribe the consumer to the orders topic

Poll messages from the topic

Print each message's value to the screen

💡 Why This Matters

🌍 Real World

Kafka consumers are used in real systems to read messages like orders, logs, or events and process them to update databases, trigger actions, or analyze data.

💼 Career

Understanding how consumers process messages is essential for roles in backend development, data engineering, and system integration where Kafka is used for reliable data streaming.

1

Set up Kafka consumer configuration

Create a dictionary called consumer_config with these exact entries: 'bootstrap.servers': 'localhost:9092', 'group.id': 'order_group', and 'auto.offset.reset': 'earliest'.

Kafka

# Create the consumer_config dictionary with the required settings
# Your code here

Need a hint?

Use a Python dictionary with keys and values as strings.

2

Create the Kafka consumer

Import Consumer from confluent_kafka and create a variable called consumer by passing consumer_config to Consumer().

Kafka

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'order_group',
    'auto.offset.reset': 'earliest'
}

# Import Consumer and create consumer using consumer_config
# Your code here

Need a hint?

Use from confluent_kafka import Consumer and then consumer = Consumer(consumer_config).

3

Subscribe and poll messages

Use consumer.subscribe(['orders']) to subscribe to the orders topic. Then write a for loop that runs 3 times, inside the loop call consumer.poll(1.0) to get a message, and if the message is not None and has no error, print message.value().decode('utf-8').

Kafka

from confluent_kafka import Consumer

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'order_group',
    'auto.offset.reset': 'earliest'
}

consumer = Consumer(consumer_config)

# Subscribe to 'orders' topic and poll 3 messages
# Your code here

Need a hint?

Subscribe first, then poll messages inside a loop, checking for errors before printing.

4

Close the consumer and show output

After the loop, call consumer.close() to close the consumer. Run the program to print the messages from the orders topic.

Kafka

from confluent_kafka import Consumer

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'order_group',
    'auto.offset.reset': 'earliest'
}

consumer = Consumer(consumer_config)

consumer.subscribe(['orders'])

for _ in range(3):
    message = consumer.poll(1.0)
    if message is not None and not message.error():
        print(message.value().decode('utf-8'))

# Close the consumer
# Your code here

Need a hint?

Don't forget to close the consumer to clean up resources.