Consumer API basics in Kafka - Commands & Configuration

Introduction

Kafka Consumer API lets you read messages from Kafka topics. It solves the problem of getting data from Kafka so your app can use it.

When you want your app to process messages sent by other apps through Kafka.

When you need to read data streams in real-time for analytics or monitoring.

When you want to build a service that reacts to events like user actions or system logs.

When you want to consume messages from multiple topics in an organized way.

When you need to control how and when messages are acknowledged to avoid losing data.

Commands

This command starts a Kafka consumer that connects to the Kafka server at localhost:9092, reads messages from the 'example-topic' topic, joins the consumer group 'example-group', and reads all messages from the beginning of the topic.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --group example-group --from-beginning

Expected Output

First message from example-topic

Second message from example-topic

Third message from example-topic

→

--bootstrap-server - Specifies the Kafka server address to connect to.

→

--topic - Specifies the topic to consume messages from.

→

--group - Defines the consumer group this consumer belongs to.

→

--from-beginning - Reads all messages from the start of the topic.

This command shows the status of the consumer group 'example-group', including which partitions it is reading and the current offset positions.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group example-group

Expected Output

GROUP           TOPIC          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID                                   HOST            CLIENT-ID

example-group   example-topic  0          15              20              5    consumer-1-12345-67890-abcde                 /127.0.0.1      consumer-1

→

--describe - Shows detailed information about the consumer group.

→

--group - Specifies the consumer group to describe.

Key Concept

If you remember nothing else from this pattern, remember: Kafka consumers read messages from topics in groups to share the work and keep track of what they have processed.

Common Mistakes

Not specifying a consumer group when starting a consumer.

Without a group, the consumer reads all messages independently and does not share load or track offsets properly.

Always use the --group flag to assign your consumer to a group.

Using --from-beginning without understanding it reads all past messages.

This can cause processing of old messages again, which might not be desired in all cases.

Use --from-beginning only when you want to process all messages from the start; otherwise omit it to read only new messages.

Not checking consumer group status with kafka-consumer-groups.

You might not know if your consumer is lagging or stuck, causing delays or data loss.

Regularly run kafka-consumer-groups --describe to monitor consumer health.

Summary

Use kafka-console-consumer with --bootstrap-server, --topic, and --group to start reading messages.

Add --from-beginning to read all messages from the start of the topic.

Use kafka-consumer-groups --describe to check the status and lag of your consumer group.