Java consumer client in Kafka - Commands & Configuration

Introduction

When you want to read messages from a Kafka topic in a Java application, you use a Java consumer client. It connects to Kafka, listens for new messages, and processes them one by one.

When you need to process user activity logs sent to a Kafka topic in real time.

When you want to build a service that reacts to events published by other applications.

When you want to read messages from Kafka and store them in a database.

When you want to monitor system metrics sent as Kafka messages.

When you want to consume messages from Kafka for batch processing.

Config File - ConsumerExample.java

ConsumerExample.java

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class ConsumerExample {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("group.id", "example-group");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Collections.singletonList("example-topic"));

        try {
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                for (ConsumerRecord<String, String> record : records) {
                    System.out.printf("Received message: key = %s, value = %s, offset = %d\n", record.key(), record.value(), record.offset());
                }
            }
        } finally {
            consumer.close();
        }
    }
}

This Java file creates a Kafka consumer client.

Properties: Set Kafka server address, consumer group, and deserializers for keys and values.

KafkaConsumer: Creates the consumer with these properties.

subscribe: Tells the consumer which topic to listen to.

poll: Checks for new messages every 100 milliseconds.

Processing loop: Prints each message's key, value, and offset.

close: Cleans up resources when done.

Commands

Compile the Java consumer client code using the Kafka client library version 3.5.1.

Terminal

javac -cp kafka-clients-3.5.1.jar ConsumerExample.java

Expected Output

No output (command runs silently)

Run the compiled Java consumer client to start reading messages from the Kafka topic.

Terminal

java -cp .:kafka-clients-3.5.1.jar ConsumerExample

Expected Output

Received message: key = user1, value = login, offset = 0

Received message: key = user2, value = logout, offset = 1

Key Concept

If you remember nothing else from this pattern, remember: the consumer must subscribe to topics and poll regularly to receive messages.

Common Mistakes

Not setting the correct deserializer classes for key and value.

Kafka cannot convert bytes to strings, so the consumer fails to read messages properly.

Always set key.deserializer and value.deserializer to StringDeserializer for string messages.

Not calling consumer.poll() in a loop.

Without polling, the consumer does not fetch new messages and appears to do nothing.

Use a continuous loop calling poll() to keep receiving messages.

Not closing the consumer after use.

Resources remain open, which can cause memory leaks or connection issues.

Call consumer.close() in a finally block to clean up properly.

Summary

Set up consumer properties including Kafka server, group ID, and deserializers.

Create a KafkaConsumer and subscribe it to the desired topic.

Use a loop to poll for new messages and process them.

Close the consumer to release resources when finished.

Practice

1.

What is the main purpose of a Java consumer client in Kafka?

easy

A. To create Kafka topics

B. To send messages to Kafka topics

C. To read messages from Kafka topics

D. To delete Kafka topics

Solution

1. Step 1: Understand Kafka client roles

   Kafka clients include producers (send messages) and consumers (read messages).

2. Step 2: Identify the consumer's role

   The Java consumer client reads messages from Kafka topics to process or display them.

3. Final Answer:

   To read messages from Kafka topics -> Option C

4. Quick Check:

   Consumer = reads messages [OK]

Hint: Consumer reads, producer writes messages in Kafka [OK]

Common Mistakes:

• Confusing consumer with producer
• Thinking consumer creates topics
• Assuming consumer deletes topics

2.

Which of the following is the correct way to subscribe a Kafka consumer to a topic named my-topic?

consumer._____ (Collections.singletonList("my-topic"));

easy

A. subscribe

B. assign

C. poll

D. commitSync

Solution

1. Step 1: Understand consumer methods

   The subscribe method registers the consumer to listen to topics.

2. Step 2: Match method to code

   Using consumer.subscribe(Collections.singletonList("my-topic")) correctly subscribes to the topic.

3. Final Answer:

   subscribe -> Option A

4. Quick Check:

   Subscribe = listen to topics [OK]

Hint: Use subscribe() to listen to topics, not assign() [OK]

Common Mistakes:

• Using assign() instead of subscribe()
• Calling poll() to subscribe
• Using commitSync() incorrectly

3.

What will the following Java Kafka consumer code print if the topic has one message with value "Hello Kafka"?

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
    System.out.println(record.value());
}

medium

A. Hello Kafka

B. null

C. ConsumerRecords object

D. No output

Solution

1. Step 1: Poll messages from topic

   The poll method fetches available messages; here it gets one message.

2. Step 2: Print each message value

   The loop prints the value() of each record, which is "Hello Kafka".

3. Final Answer:

   Hello Kafka -> Option A

4. Quick Check:

   Print record.value() = Hello Kafka [OK]

Hint: poll() fetches messages; print record.value() to see content [OK]

Common Mistakes:

• Printing the record object instead of value
• Expecting poll() to return null if no messages
• Confusing key and value fields

4.

Identify the error in this Kafka consumer code snippet:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "group1");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("my-topic"));
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records) {
    System.out.println(record.value());
}

medium

A. group.id property is missing

B. subscribe() should use assign()

C. Deserializers are incorrectly set

D. poll() argument should be Duration, not int

Solution

1. Step 1: Check poll() method signature

   In modern Kafka clients, poll requires a Duration argument, not an int.

2. Step 2: Identify incorrect argument type

   The code uses poll(100) (int), should be poll(Duration.ofMillis(100)).

3. Final Answer:

   poll() argument should be Duration, not int -> Option D

4. Quick Check:

   poll() needs Duration argument [OK]

Hint: Use Duration.ofMillis() with poll(), not plain int [OK]

Common Mistakes:

• Passing int to poll() instead of Duration
• Missing group.id property
• Using assign() instead of subscribe()

5.

You want to consume messages from two topics topicA and topicB and print their keys and values. Which code snippet correctly subscribes and processes messages?

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "group1");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer._____ (Arrays.asList("topicA", "topicB"));
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        System.out.println("Key: " + record.key() + ", Value: " + record.value());
    }
}

hard

A. assign

B. subscribe

C. poll

D. commitSync

Solution

1. Step 1: Choose method to listen to multiple topics

   subscribe accepts a list of topics to listen to, while assign assigns specific partitions.

2. Step 2: Confirm correct usage in code

   Using consumer.subscribe(Arrays.asList("topicA", "topicB")) subscribes to both topics properly.

3. Final Answer:

   subscribe -> Option B

4. Quick Check:

   subscribe() listens to multiple topics [OK]

Hint: Use subscribe() with list to consume multiple topics [OK]

Common Mistakes:

• Using assign() instead of subscribe() for topics
• Calling poll() to subscribe
• Forgetting to print keys