What is group.id in Kafka Consumer and How It Works

group.id in a Kafka consumer is a unique identifier for a group of consumers that work together to read data from Kafka topics. It helps Kafka coordinate which consumer reads which part of the data, enabling load balancing and fault tolerance.

How It Works

Imagine a team of friends sharing a big pizza. Each friend takes a slice so that everyone gets a part without overlap. In Kafka, group.id works like the team name. All consumers with the same group.id form a group that shares the work of reading messages from topics.

Kafka divides the topic's data into parts called partitions. Each consumer in the group reads from different partitions, so no two consumers read the same partition at the same time. If one consumer stops, Kafka moves its partitions to others in the group, keeping the data flowing smoothly.

Example

This example shows how to set group.id in a Kafka consumer configuration using Java. It creates a consumer that joins the group named my-consumer-group.

java

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import java.util.Properties;
import java.util.Collections;

Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-consumer-group");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("my-topic"));

Output

Consumer joins the group 'my-consumer-group' and starts reading assigned partitions from 'my-topic'.

When to Use

Use group.id when you want multiple consumers to share the work of reading messages from Kafka topics. This is useful for scaling your application to handle more data or for ensuring that if one consumer fails, others can continue processing without losing messages.

For example, in a web application that processes user activity logs, you can have several consumers in the same group to speed up processing. If one consumer crashes, Kafka automatically redistributes its workload to the remaining consumers in the group.

Key Points

• group.id identifies a consumer group in Kafka.
• Consumers in the same group share partitions to balance load.
• Kafka reassigns partitions if a consumer leaves or fails.
• Each group gets its own view of the topic data, allowing multiple independent processing streams.

Key Takeaways

group.id groups consumers to share topic partitions and balance load.

Consumers in the same group do not read the same partition simultaneously.

Kafka automatically handles partition reassignment if a consumer fails.

group.id enables scalable and fault-tolerant message processing.

Different consumer groups can read the same topic independently.