Python consumer in Kafka - Mini Project: Build & Apply

Python Kafka Consumer

📖 Scenario: You are building a simple Python program that listens to messages from a Kafka topic. Kafka is like a message bus where different programs can send and receive messages. Your program will act as a consumer that reads messages from a specific topic.

🎯 Goal: Create a Python Kafka consumer that connects to a Kafka server, subscribes to a topic named test-topic, and prints out messages it receives.

📋 What You'll Learn

Create a Kafka consumer connected to localhost:9092

Subscribe to the topic test-topic

Poll messages from the topic

Print each received message's value

💡 Why This Matters

🌍 Real World

Kafka consumers are used in real applications to process streams of data like logs, user activity, or sensor data in real time.

💼 Career

Understanding Kafka consumers is important for roles in data engineering, backend development, and real-time data processing.

1

Set up Kafka consumer configuration

Create a dictionary called conf with these exact entries: 'bootstrap.servers': 'localhost:9092', 'group.id': 'mygroup', and 'auto.offset.reset': 'earliest'.

Kafka

# Your code here

Need a hint?

This dictionary holds the settings needed to connect to Kafka.

2

Create the Kafka consumer

Import Consumer from confluent_kafka and create a variable called consumer by passing the conf dictionary to Consumer(conf).

Kafka

conf = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'mygroup',
    'auto.offset.reset': 'earliest'
}

# Your code here

Need a hint?

Use the Consumer class to create a consumer object with the configuration.

3

Subscribe to the topic

Use the consumer.subscribe() method to subscribe to a list containing the single topic 'test-topic'.

Kafka

from confluent_kafka import Consumer

conf = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'mygroup',
    'auto.offset.reset': 'earliest'
}

consumer = Consumer(conf)

# Your code here

Need a hint?

Subscribe to the topic by passing a list with the topic name.

4

Poll and print messages

Write a while True loop that polls messages from consumer.poll(1.0). If a message is received and has no error, print its value decoded as UTF-8. Break the loop after printing one message.

Kafka

from confluent_kafka import Consumer

conf = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'mygroup',
    'auto.offset.reset': 'earliest'
}

consumer = Consumer(conf)

consumer.subscribe(['test-topic'])

# Your code here

Need a hint?

This loop waits for a message, checks for errors, prints the message text, then stops.