Subscribing to topics in Kafka - Mini Project: Build & Apply

Subscribing to Kafka Topics

📖 Scenario: You are building a simple Kafka consumer application that listens to messages from a specific topic. This is common in real-world systems where services communicate by sending messages through Kafka topics.

🎯 Goal: Create a Kafka consumer that subscribes to a topic named test-topic and prints any messages it receives.

📋 What You'll Learn

Create a Kafka consumer configuration dictionary with the required settings

Create a variable called topic with the value 'test-topic'

Subscribe the consumer to the topic

Print the messages received from the topic

💡 Why This Matters

🌍 Real World

Kafka consumers are used in real systems to process streams of data like logs, user activity, or sensor data.

💼 Career

Understanding how to subscribe and consume messages from Kafka topics is essential for backend developers, data engineers, and anyone working with real-time data pipelines.

1

Create Kafka consumer configuration

Create a dictionary called consumer_config with these exact entries: 'bootstrap.servers': 'localhost:9092', 'group.id': 'my-group', and 'auto.offset.reset': 'earliest'.

Kafka

# Your code here

Need a hint?

Use a Python dictionary with the exact keys and values as shown.

2

Set the topic name

Create a variable called topic and set it to the string 'test-topic'.

Kafka

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'my-group',
    'auto.offset.reset': 'earliest'
}
# Your code here

Need a hint?

Assign the string 'test-topic' to the variable named topic.

3

Subscribe the consumer to the topic

Import Consumer from confluent_kafka. Create a Consumer instance called consumer using consumer_config. Use consumer.subscribe([topic]) to subscribe to the topic.

Kafka

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'my-group',
    'auto.offset.reset': 'earliest'
}
topic = 'test-topic'
# Your code here

Need a hint?

Use the Consumer class and call subscribe with a list containing topic.

4

Print messages from the topic

Write a loop that calls consumer.poll(1.0) to get messages. If a message is received and no error, print message.value().decode('utf-8'). Break the loop after printing one message. Finally, call consumer.close().

Kafka

from confluent_kafka import Consumer

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'my-group',
    'auto.offset.reset': 'earliest'
}
topic = 'test-topic'

consumer = Consumer(consumer_config)
consumer.subscribe([topic])
# Your code here

Need a hint?

Use poll to get messages, check for errors, decode the message value, print it, and then close the consumer.