Consumer API basics in Kafka - Commands & Configuration

Introduction

Kafka Consumer API lets you read messages from Kafka topics. It solves the problem of getting data from Kafka so your app can use it.

When you want your app to process messages sent by other apps through Kafka.

When you need to read data streams in real-time for analytics or monitoring.

When you want to build a service that reacts to events like user actions or system logs.

When you want to consume messages from multiple topics in an organized way.

When you need to control how and when messages are acknowledged to avoid losing data.

Commands

This command starts a Kafka consumer that connects to the Kafka server at localhost:9092, reads messages from the 'example-topic' topic, joins the consumer group 'example-group', and reads all messages from the beginning of the topic.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --group example-group --from-beginning

Expected Output

First message from example-topic

Second message from example-topic

Third message from example-topic

→

--bootstrap-server - Specifies the Kafka server address to connect to.

→

--topic - Specifies the topic to consume messages from.

→

--group - Defines the consumer group this consumer belongs to.

→

--from-beginning - Reads all messages from the start of the topic.

This command shows the status of the consumer group 'example-group', including which partitions it is reading and the current offset positions.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group example-group

Expected Output

GROUP           TOPIC          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID                                   HOST            CLIENT-ID

example-group   example-topic  0          15              20              5    consumer-1-12345-67890-abcde                 /127.0.0.1      consumer-1

→

--describe - Shows detailed information about the consumer group.

→

--group - Specifies the consumer group to describe.

Key Concept

If you remember nothing else from this pattern, remember: Kafka consumers read messages from topics in groups to share the work and keep track of what they have processed.

Common Mistakes

Not specifying a consumer group when starting a consumer.

Without a group, the consumer reads all messages independently and does not share load or track offsets properly.

Always use the --group flag to assign your consumer to a group.

Using --from-beginning without understanding it reads all past messages.

This can cause processing of old messages again, which might not be desired in all cases.

Use --from-beginning only when you want to process all messages from the start; otherwise omit it to read only new messages.

Not checking consumer group status with kafka-consumer-groups.

You might not know if your consumer is lagging or stuck, causing delays or data loss.

Regularly run kafka-consumer-groups --describe to monitor consumer health.

Summary

Use kafka-console-consumer with --bootstrap-server, --topic, and --group to start reading messages.

Add --from-beginning to read all messages from the start of the topic.

Use kafka-consumer-groups --describe to check the status and lag of your consumer group.

Practice

1. What is the primary purpose of the Kafka Consumer API?

easy

A. To send messages to Kafka topics

B. To delete Kafka topics

C. To create Kafka topics

D. To read messages from Kafka topics

Solution

1. Step 1: Understand Kafka Consumer API role

   The Consumer API is designed to read or consume messages from Kafka topics.

2. Step 2: Differentiate from Producer API

   The Producer API sends messages, while Consumer API reads them.

3. Final Answer:

   To read messages from Kafka topics -> Option D

4. Quick Check:

   Consumer API = Read messages [OK]

Hint: Consumer API reads messages, Producer API sends messages [OK]

Common Mistakes:

• Confusing Consumer API with Producer API
• Thinking Consumer API creates or deletes topics
• Mixing up reading and writing operations

2. Which of the following is the correct way to create a Kafka consumer in Java?

easy

A. new KafkaProducer<>(properties);

B. new KafkaConsumer<>(properties);

C. KafkaConsumer.create(properties);

D. KafkaConsumer.start(properties);

Solution

1. Step 1: Recall Kafka consumer instantiation syntax

   Kafka consumer is created by calling the constructor: new KafkaConsumer<>(properties);

2. Step 2: Eliminate incorrect options

   Producer is for sending messages, so new KafkaProducer<>(properties); is wrong. Options C and D are not valid KafkaConsumer methods.

3. Final Answer:

   new KafkaConsumer<>(properties); -> Option B

4. Quick Check:

   Constructor new KafkaConsumer<>(props) = Correct [OK]

Hint: Use new KafkaConsumer<>(props) to create consumer [OK]

Common Mistakes:

• Using KafkaProducer instead of KafkaConsumer
• Calling non-existent static methods
• Confusing method names for creation

3. Given the following Java code snippet, what will be the output if the topic has two messages "Hello" and "World"?

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("my-topic"));
ConsumerRecords records = consumer.poll(Duration.ofMillis(1000));
for (ConsumerRecord record : records) {
    System.out.println(record.value());
}

medium

A. Compilation error

B. No output

C. Hello\nWorld

D. Runtime exception

Solution

1. Step 1: Understand consumer subscription and poll

   The consumer subscribes to "my-topic" and polls messages for 1 second.

2. Step 2: Process messages in the topic

   Since the topic has two messages "Hello" and "World", the poll will fetch both, and the loop prints each value.

3. Final Answer:

   Hello\nWorld -> Option C

4. Quick Check:

   poll() fetches messages = Hello and World printed [OK]

Hint: poll() fetches messages; loop prints each value [OK]

Common Mistakes:

• Assuming poll returns no messages immediately
• Confusing subscribe with assign
• Expecting compile or runtime errors without cause

4. Identify the error in this Kafka consumer code snippet:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "group1");
props.put("key.deserializer", "StringDeserializer");
props.put("value.deserializer", "StringDeserializer");
KafkaConsumer consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("topic1"));
ConsumerRecords records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord record : records) {
    System.out.println(record.value());
}

medium

A. Deserializer class names are incomplete and will cause ClassNotFoundException

B. Missing call to consumer.close()

C. Incorrect subscription method used

D. poll() duration is too short and causes timeout

Solution

1. Step 1: Check deserializer configuration

   The deserializer class names must be fully qualified, e.g., "org.apache.kafka.common.serialization.StringDeserializer".

2. Step 2: Identify impact of incorrect class names

   Using "StringDeserializer" alone will cause ClassNotFoundException at runtime.

3. Final Answer:

   Deserializer class names are incomplete and will cause ClassNotFoundException -> Option A

4. Quick Check:

   Deserializer full class name required = Deserializer class names are incomplete and will cause ClassNotFoundException [OK]

Hint: Use full deserializer class names to avoid errors [OK]

Common Mistakes:

• Using short class names for deserializers
• Ignoring missing consumer.close() (not error here)
• Assuming poll duration causes errors

5. You want to process messages from multiple topics and commit offsets manually after processing. Which code snippet correctly sets up the Kafka consumer for this use case?

hard

A. props.put("enable.auto.commit", "false");
consumer.subscribe(Arrays.asList("topicA", "topicB"));
while (true) {
ConsumerRecords records = consumer.poll(Duration.ofMillis(1000));
for (ConsumerRecord record : records) {
process(record.value());
}
consumer.commitSync();
}

B. props.put("enable.auto.commit", "true");
consumer.subscribe(Collections.singletonList("topicA"));
while (true) {
ConsumerRecords records = consumer.poll(Duration.ofMillis(1000));
for (ConsumerRecord record : records) {
process(record.value());
}
}

C. props.put("enable.auto.commit", "false");
consumer.assign(Arrays.asList(new TopicPartition("topicA", 0)));
consumer.subscribe(Arrays.asList("topicB"));
while (true) {
ConsumerRecords records = consumer.poll(Duration.ofMillis(1000));
for (ConsumerRecord record : records) {
process(record.value());
}
consumer.commitAsync();
}

D. props.put("enable.auto.commit", "false");
consumer.subscribe(Arrays.asList("topicA", "topicB"));
while (true) {
ConsumerRecords records = consumer.poll(Duration.ofMillis(1000));
for (ConsumerRecord record : records) {
process(record.value());
}
}

Solution

1. Step 1: Disable auto commit for manual offset control

   Setting "enable.auto.commit" to "false" allows manual offset commits.

2. Step 2: Subscribe to multiple topics correctly

   Use consumer.subscribe(Arrays.asList("topicA", "topicB")) to listen to multiple topics.

3. Step 3: Commit offsets after processing

   Calling consumer.commitSync() after processing ensures offsets are saved manually.

4. Final Answer:

   Code snippet with manual commit, multiple topic subscription, and commitSync() -> Option A

5. Quick Check:

   Manual commit + multiple topics = props.put("enable.auto.commit", "false"); consumer.subscribe(Arrays.asList("topicA", "topicB")); while (true) { ConsumerRecords records = consumer.poll(Duration.ofMillis(1000)); for (ConsumerRecord record : records) { process(record.value()); } consumer.commitSync(); } [OK]

Hint: Disable auto commit and call commitSync() after processing [OK]

Common Mistakes:

• Using auto commit when manual commit is needed
• Mixing assign() and subscribe() incorrectly
• Forgetting to commit offsets after processing