Why consumers process messages in Kafka - See It in Action

Why consumers process messages

📖 Scenario: Imagine you have a system where different parts talk to each other by sending messages. These messages are like letters in a mailbox. Consumers are the parts that read these letters and do something with the information.

🎯 Goal: Build a simple Kafka consumer that reads messages from a topic called orders and prints each message to understand why consumers process messages.

📋 What You'll Learn

Create a Kafka consumer that connects to a Kafka server

Subscribe the consumer to the orders topic

Poll messages from the topic

Print each message's value to the screen

💡 Why This Matters

🌍 Real World

Kafka consumers are used in real systems to read messages like orders, logs, or events and process them to update databases, trigger actions, or analyze data.

💼 Career

Understanding how consumers process messages is essential for roles in backend development, data engineering, and system integration where Kafka is used for reliable data streaming.

1

Set up Kafka consumer configuration

Create a dictionary called consumer_config with these exact entries: 'bootstrap.servers': 'localhost:9092', 'group.id': 'order_group', and 'auto.offset.reset': 'earliest'.

Kafka

# Create the consumer_config dictionary with the required settings
# Your code here

Hint

Use a Python dictionary with keys and values as strings.

2

Create the Kafka consumer

Import Consumer from confluent_kafka and create a variable called consumer by passing consumer_config to Consumer().

Kafka

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'order_group',
    'auto.offset.reset': 'earliest'
}

# Import Consumer and create consumer using consumer_config
# Your code here

Hint

Use from confluent_kafka import Consumer and then consumer = Consumer(consumer_config).

3

Subscribe and poll messages

Use consumer.subscribe(['orders']) to subscribe to the orders topic. Then write a for loop that runs 3 times, inside the loop call consumer.poll(1.0) to get a message, and if the message is not None and has no error, print message.value().decode('utf-8').

Kafka

from confluent_kafka import Consumer

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'order_group',
    'auto.offset.reset': 'earliest'
}

consumer = Consumer(consumer_config)

# Subscribe to 'orders' topic and poll 3 messages
# Your code here

Hint

Subscribe first, then poll messages inside a loop, checking for errors before printing.

4

Close the consumer and show output

After the loop, call consumer.close() to close the consumer. Run the program to print the messages from the orders topic.

Kafka

from confluent_kafka import Consumer

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'order_group',
    'auto.offset.reset': 'earliest'
}

consumer = Consumer(consumer_config)

consumer.subscribe(['orders'])

for _ in range(3):
    message = consumer.poll(1.0)
    if message is not None and not message.error():
        print(message.value().decode('utf-8'))

# Close the consumer
# Your code here

Hint

Don't forget to close the consumer to clean up resources.

Practice

1. Why do Kafka consumers process messages from topics?

easy

A. To create new topics automatically

B. To read and use data sent by producers

C. To delete messages from the topic

D. To send messages to other consumers

Solution

1. Step 1: Understand the role of consumers in Kafka

   Consumers read messages produced by producers to get data for processing.

2. Step 2: Identify the purpose of processing messages

   Processing messages allows applications to react to new information sent by producers.

3. Final Answer:

   To read and use data sent by producers -> Option B

4. Quick Check:

   Consumers read messages = To use data [OK]

Hint: Consumers read messages to get data from producers [OK]

Common Mistakes:

• Thinking consumers send messages
• Confusing consumers with producers
• Believing consumers delete messages

2. Which of the following is the correct way for a Kafka consumer to subscribe to a topic named orders in Java?

easy

A. consumer.listen("orders");

B. consumer.subscribe("orders");

C. consumer.connect("orders");

D. consumer.subscribe(Collections.singletonList("orders"));

Solution

1. Step 1: Recall Kafka consumer subscription syntax in Java

   The consumer.subscribe method expects a collection of topic names, usually a list.

2. Step 2: Check each option's syntax

   consumer.subscribe(Collections.singletonList("orders")); uses Collections.singletonList("orders"), which is correct. Options A, B, and D use incorrect methods or argument types.

3. Final Answer:

   consumer.subscribe(Collections.singletonList("orders")); -> Option D

4. Quick Check:

   subscribe needs a list of topics [OK]

Hint: Use subscribe with a list of topics in Java [OK]

Common Mistakes:

• Passing a string instead of a list
• Using non-existent methods like listen or connect
• Forgetting to import Collections

3. Given the following Kafka consumer code snippet in Python, what will be printed?

from kafka import KafkaConsumer

consumer = KafkaConsumer('test-topic', bootstrap_servers=['localhost:9092'])
for message in consumer:
    print(message.value.decode('utf-8'))
    break

medium

A. The first message's value from 'test-topic' decoded as UTF-8

B. All messages from 'test-topic' printed continuously

C. An error because decode is not a method

D. No output because the loop breaks immediately

Solution

1. Step 1: Understand the consumer loop behavior

   The for loop reads messages from 'test-topic' one by one.

2. Step 2: Analyze the print and break statements

   The first message's value is decoded from bytes to string and printed, then the loop breaks immediately.

3. Final Answer:

   The first message's value from 'test-topic' decoded as UTF-8 -> Option A

4. Quick Check:

   Print first message then break = first message only [OK]

Hint: Loop breaks after first message, so only one prints [OK]

Common Mistakes:

• Thinking all messages print
• Confusing decode method usage
• Assuming no output due to break

4. What is wrong with this Kafka consumer code snippet in Java?

KafkaConsumer consumer = new KafkaConsumer<>(props);
consumer.subscribe("my-topic");
while(true) {
    ConsumerRecords records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord record : records) {
        System.out.println(record.value());
    }
}

medium

A. poll method should not have a timeout argument

B. ConsumerRecord does not have a value() method

C. subscribe method requires a collection, not a single string

D. The while loop should be replaced with a for loop

Solution

1. Step 1: Check the subscribe method usage

   In Kafka Java client, subscribe expects a collection of topic names, not a single string.

2. Step 2: Verify other code parts

   poll with Duration is correct, ConsumerRecord has value(), and while(true) is valid for continuous consumption.

3. Final Answer:

   subscribe method requires a collection, not a single string -> Option C

4. Quick Check:

   subscribe needs list of topics [OK]

Hint: subscribe needs a list, not a single string [OK]

Common Mistakes:

• Passing a string directly to subscribe
• Misunderstanding poll method arguments
• Thinking ConsumerRecord lacks value()

5. You want your Kafka consumer to process messages only from the latest offset to react quickly to new data. Which consumer configuration should you set before subscribing?

hard

A. "auto.offset.reset" to "latest"

B. "enable.auto.commit" to false

C. "group.id" to a unique value each time

D. "max.poll.records" to 1

Solution

1. Step 1: Understand offset reset behavior

   Setting "auto.offset.reset" to "latest" makes the consumer start reading from the newest messages if no committed offset exists.

2. Step 2: Evaluate other options

   "enable.auto.commit" controls offset commits, "group.id" identifies consumer groups, and "max.poll.records" limits batch size but doesn't affect starting offset.

3. Final Answer:

   "auto.offset.reset" to "latest" -> Option A

4. Quick Check:

   Start from latest messages = auto.offset.reset=latest [OK]

Hint: Set auto.offset.reset to latest to read newest messages [OK]

Common Mistakes:

• Confusing offset reset with auto commit
• Changing group.id to control offset start
• Using max.poll.records to control offset