Why consumers process messages in Kafka - Why It Works

Introduction

In Kafka, messages are sent to topics by producers. Consumers read these messages to perform tasks like updating databases, triggering actions, or analyzing data. Processing messages ensures that the information sent by producers is used effectively in real applications.

When you want to update a user profile in your database after receiving new user data

When you need to trigger an email notification after a purchase event is recorded

When you want to analyze streaming data for real-time insights, like monitoring sensor data

When you need to synchronize data between different systems by reading messages from Kafka

When you want to build a scalable system that reacts to events as they happen

Commands

This command starts a Kafka consumer that reads the first 5 messages from the topic 'example-topic' from the beginning. It shows how consumers process messages by reading them from Kafka.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --max-messages 5

Expected Output

Message 1 content

Message 2 content

Message 3 content

Message 4 content

Message 5 content

→

--from-beginning - Reads messages from the start of the topic, not just new messages

→

--max-messages - Limits the number of messages to read before stopping

This command shows the status of the consumer group 'example-group', including which messages have been processed. It helps verify that consumers are processing messages correctly.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group example-group

Expected Output

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST  CLIENT-ID

example-topic   0          10              10              0    consumer-1   /127.0.0.1  consumer-1-client

→

--group - Specifies the consumer group to check

Key Concept

Consumers process messages to turn data sent by producers into useful actions or results in real time.

Common Mistakes

Not committing offsets after processing messages

Kafka will think messages are not processed and may resend them, causing duplicate work

Always commit offsets after successfully processing messages to mark them as done

Reading messages only from the latest offset without from-beginning when testing

You might miss earlier messages needed for testing or replay

Use --from-beginning flag to read all messages from the start when needed

Summary

Use kafka-console-consumer to read and process messages from a topic.

Check consumer group status to verify message processing and lag.

Commit offsets to avoid reprocessing the same messages.