Consumer configuration in Kafka - Commands & Configuration

Introduction

Kafka consumers read messages from topics to process data. Consumer configuration sets how they connect, read, and manage message delivery to ensure smooth data flow.

When you want to control how fast your app reads messages from Kafka.

When you need to decide if messages are marked as read automatically or manually.

When you want to handle what happens if your consumer crashes and restarts.

When you want to group multiple consumers to share the work of reading messages.

When you want to set limits on how much data your consumer fetches at once.

Config File - consumer.properties

consumer.properties

bootstrap.servers=localhost:9092
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
group.id=my-consumer-group
enable.auto.commit=false
auto.offset.reset=earliest
max.poll.records=500
session.timeout.ms=10000

bootstrap.servers: Kafka server address to connect.

key.deserializer and value.deserializer: How to convert message bytes to strings.

group.id: Consumer group name to share message reading.

enable.auto.commit: Whether to mark messages as read automatically.

auto.offset.reset: Where to start if no previous read position exists.

max.poll.records: Max messages to fetch in one request.

session.timeout.ms: Time to detect consumer failure.

Commands

Starts a Kafka consumer that reads messages from 'my-topic' from the beginning using the settings in consumer.properties.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic my-topic --group my-consumer-group --from-beginning --consumer.config consumer.properties

Expected Output

This will print messages from 'my-topic' starting from the earliest available message.

Example:

Hello World

Kafka message 1

Kafka message 2

→

--bootstrap-server - Specifies the Kafka server address to connect.

→

--topic - Specifies the topic to consume messages from.

→

--group - Sets the consumer group for load balancing and offset tracking.

→

--from-beginning - Starts reading messages from the earliest offset.

→

--consumer.config - Loads consumer configuration from a properties file.

Shows the status of the consumer group 'my-consumer-group', including offsets and lag.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-consumer-group

Expected Output

TOPIC      PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID                                   HOST            CLIENT-ID

my-topic   0          15              20              5    consumer-1-12345-67890                      /127.0.0.1      consumer-1

→

--bootstrap-server - Specifies the Kafka server address.

→

--describe - Shows detailed information about the consumer group.

→

--group - Specifies which consumer group to inspect.

Key Concept

If you remember nothing else from this pattern, remember: consumer configuration controls how your app reads, commits, and manages message delivery from Kafka topics.

Common Mistakes

Not setting 'group.id' in the consumer configuration.

Without a group ID, Kafka treats each consumer as independent, losing benefits of load balancing and offset tracking.

Always set a meaningful 'group.id' to enable consumer group features.

Leaving 'enable.auto.commit' as true when manual control is needed.

Automatic commits can cause message loss or duplicate processing if your app crashes before processing messages.

Set 'enable.auto.commit=false' and commit offsets manually after processing.

Not setting 'auto.offset.reset' when starting a new consumer group.

If no offset exists, the consumer may not read any messages or start from the latest, missing old data.

Set 'auto.offset.reset=earliest' to read from the beginning if no offset is found.

Summary

Create a consumer configuration file to control how your Kafka consumer reads and commits messages.

Use kafka-console-consumer with the --consumer.config flag to apply your settings.

Check consumer group status and lag with kafka-consumer-groups to monitor consumption.