Java consumer client in Kafka - Step-by-Step Execution

Process Flow - Java consumer client

Start Java Consumer

↓

Configure Properties

↓

Create KafkaConsumer

↓

Subscribe to Topic

↓

Poll for Records

↓

Process Records

↓

Commit Offsets

↓

Repeat Poll or Exit

↓

Close Consumer

This flow shows how a Java Kafka consumer starts, subscribes to a topic, polls messages, processes them, commits offsets, and closes.

Execution Sample

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test-group");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props, new StringDeserializer(), new StringDeserializer());
consumer.subscribe(Collections.singletonList("my-topic"));
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));

This code sets up a Kafka consumer, subscribes to 'my-topic', and polls for messages.

Process Table

Step	Action	Evaluation	Result
1	Create Properties object	Properties empty	Properties with bootstrap.servers and group.id set
2	Create KafkaConsumer with properties and deserializers	KafkaConsumer instance created	Consumer ready to subscribe
3	Subscribe to topic 'my-topic'	Subscription successful	Consumer listens to 'my-topic'
4	Poll for records with 100ms timeout	Poll sent to Kafka broker	ConsumerRecords received (may be empty)
5	Process each record	Iterate over records	Records processed (e.g., print key/value)
6	Commit offsets	Offsets committed synchronously	Kafka knows messages processed
7	Repeat poll or exit	Loop continues or stops	Consumer continues polling or closes
8	Close consumer	Consumer closed	Resources freed, connection closed

💡 Consumer closes after processing or on application shutdown

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6	Final
props	empty	{"bootstrap.servers":"localhost:9092","group.id":"test-group"}	{"bootstrap.servers":"localhost:9092","group.id":"test-group"}	{"bootstrap.servers":"localhost:9092","group.id":"test-group"}	{"bootstrap.servers":"localhost:9092","group.id":"test-group"}	{"bootstrap.servers":"localhost:9092","group.id":"test-group"}	{"bootstrap.servers":"localhost:9092","group.id":"test-group"}	unchanged
consumer	null	null	KafkaConsumer instance	KafkaConsumer subscribed	KafkaConsumer polled	KafkaConsumer processed records	KafkaConsumer committed offsets	KafkaConsumer closed
records	null	null	null	ConsumerRecords (may be empty or contain messages)	Processed records	Processed records	Processed records	null

Key Moments - 3 Insights

Why do we need to set 'group.id' in the properties?

The 'group.id' identifies the consumer group this client belongs to, which Kafka uses to manage message delivery and offset commits. See execution_table step 1 where properties are set.

What happens if the poll() call returns no records?

If poll() returns no records, the consumer waits and polls again in the loop. This is normal if no new messages are available. See execution_table step 4.

Why must we commit offsets after processing records?

Committing offsets tells Kafka which messages have been processed, so on restart the consumer doesn't reprocess them. See execution_table step 6.

Visual Quiz - 3 Questions

Look at the execution_table at step 4, what does the poll() method return?

A A ConsumerRecords object containing messages or empty

B A Properties object with configuration

C A KafkaConsumer instance

D A list of topic names

💡 Hint

Refer to execution_table row 4 under Result column

At which step is the consumer subscribed to the topic?

A Step 2

B Step 3

C Step 5

D Step 6

💡 Hint

Check execution_table row 3 Action and Result columns

If we omit committing offsets, what will likely happen on consumer restart?

A Consumer will skip messages already processed

B Consumer will reprocess messages from last committed offset

C Consumer will fail to connect to Kafka

D Consumer will automatically commit offsets

💡 Hint

See key_moments answer about offset commits and execution_table step 6

Concept Snapshot

Java Kafka Consumer Client:
- Configure Properties (bootstrap.servers, group.id)
- Create KafkaConsumer with deserializers
- Subscribe to topic(s)
- Poll for messages in a loop
- Process messages
- Commit offsets to track progress
- Close consumer on shutdown

Full Transcript

This visual execution trace shows how a Java Kafka consumer client works step-by-step. First, properties are configured with the Kafka server address and consumer group ID. Then, a KafkaConsumer instance is created using these properties and deserializers for keys and values. The consumer subscribes to a topic, for example 'my-topic'. It then polls the Kafka broker for new messages with a timeout. If messages are received, the consumer processes each record, such as printing the key and value. After processing, the consumer commits the offsets to Kafka to mark messages as consumed. This polling and processing loop continues until the consumer is closed, freeing resources and closing connections. Key points include the importance of group.id for consumer groups, handling empty polls, and committing offsets to avoid reprocessing messages on restart.