What Is Consumer in Kafka: Definition and Usage Explained

A consumer in Kafka is a client application that reads data from Kafka topics. It subscribes to one or more topics and processes the stream of records produced by Kafka producers.

How It Works

Think of Kafka as a big message board where producers post messages (called records) on different topics. A consumer is like a person who reads messages from this board. It connects to Kafka, subscribes to topics it cares about, and reads messages in the order they were posted.

Consumers work in groups called consumer groups. Each group shares the work of reading messages from topics, so messages are divided among group members. This way, consumers can process data in parallel, like a team sharing tasks to finish faster.

Example

This example shows a simple Kafka consumer in Java that reads messages from a topic named my-topic. It prints each message to the console.

java

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;

import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class SimpleConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("group.id", "my-group");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Collections.singletonList("my-topic"));

        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
            for (ConsumerRecord<String, String> record : records) {
                System.out.printf("Received message: key = %s, value = %s, offset = %d\n",
                        record.key(), record.value(), record.offset());
            }
        }
    }
}

Output

Received message: key = null, value = Hello Kafka, offset = 0
Received message: key = null, value = Welcome to Kafka consumer, offset = 1

When to Use

Use a Kafka consumer when you want to read and process streams of data in real time. For example, you might use consumers to:

• Process user activity logs for analytics
• Update search indexes as new data arrives
• Trigger alerts based on incoming events
• Integrate data between different systems asynchronously

Consumers help build scalable, fault-tolerant data pipelines by distributing work across multiple instances.

Key Points

• A consumer reads messages from Kafka topics.
• Consumers work in groups to share message processing.
• They keep track of which messages they have read using offsets.
• Consumers enable real-time data processing and integration.

Key Takeaways

A Kafka consumer reads and processes messages from Kafka topics.

Consumers in a group share the workload for scalability and fault tolerance.

They track message positions using offsets to avoid reprocessing.

Use consumers to build real-time data processing and integration pipelines.