How Consumer Group Works in Kafka: Explained Simply

In Kafka, a consumer group is a set of consumers that share the work of reading messages from topics. Each message is delivered to only one consumer in the group, enabling parallel processing and fault tolerance.

Syntax

A Kafka consumer group is identified by a group.id configuration. Consumers with the same group.id form a group and coordinate to consume topic partitions.

Key parts:

• group.id: Unique name for the consumer group.
• topic: The Kafka topic to consume messages from.
• consumer: The client instance that reads messages.

java

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "my-consumer-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("my-topic"));

Example

This example shows two consumers in the same group consuming from a topic with two partitions. Each consumer reads from one partition, so messages are split between them.

java

import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class ConsumerExample {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("group.id", "example-group");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Collections.singletonList("example-topic"));

        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
            for (ConsumerRecord<String, String> record : records) {
                System.out.printf("Consumer %s read message: key = %s, value = %s, partition = %d, offset = %d\n",
                    Thread.currentThread().getName(), record.key(), record.value(), record.partition(), record.offset());
            }
        }
    }
}

Output

Consumer Thread-1 read message: key = key1, value = value1, partition = 0, offset = 0
Consumer Thread-2 read message: key = key2, value = value2, partition = 1, offset = 0

Common Pitfalls

• Same group.id for unrelated consumers: Causes consumers to share partitions unexpectedly.
• Too many consumers: More consumers than partitions means some consumers stay idle.
• Not committing offsets: Can cause message reprocessing or data loss.

java

/* Wrong: Different group IDs, so consumers read all messages independently */
props.put("group.id", "group1");
// Another consumer uses "group2" - no load sharing

/* Right: Same group ID for load sharing */
props.put("group.id", "shared-group");

Quick Reference

Concept	Description
group.id	Unique identifier for the consumer group
Partition assignment	Each partition is assigned to only one consumer in the group
Offset commit	Tracks which messages have been processed
Load balancing	Consumers share partitions to balance message processing
Fault tolerance	If a consumer fails, others take over its partitions

Key Takeaways

Kafka consumer groups allow multiple consumers to share message processing by dividing topic partitions.

Each message is delivered to only one consumer in the group, enabling parallelism and fault tolerance.

Consumers must share the same group.id to form a group and coordinate partition assignment.

Having more consumers than partitions causes some consumers to be idle.

Proper offset management is essential to avoid message loss or duplication.