Consumer group concept in Kafka - Commands & Configuration

Introduction

When multiple applications read messages from the same Kafka topic, they need a way to share the work without duplicating messages. Consumer groups let several consumers work together to read messages evenly and keep track of what has been read.

When you want to scale message processing by adding more consumers to share the load.

When you need to ensure each message is processed only once by one consumer in a group.

When you want to balance message consumption across multiple servers or instances.

When you want to track which messages have been processed to avoid duplicates.

When you want to handle failover so if one consumer stops, others continue processing.

Commands

Starts a consumer that reads messages from 'example-topic' as part of the 'my-consumer-group'. It reads all messages from the beginning of the topic.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --group my-consumer-group --from-beginning

Expected Output

This will print messages from the topic as they arrive, for example:

Message 1

Message 2

Message 3

→

--group - Specifies the consumer group this consumer belongs to.

→

--from-beginning - Reads all messages from the start of the topic.

Shows details about the consumer group 'my-consumer-group', including which partitions each consumer is reading and the lag (messages not yet processed).

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-consumer-group

Expected Output

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID                                    HOST            CLIENT-ID

example-topic   0          15              20              5    consumer-1-12345                               /127.0.0.1      consumer-1

example-topic   1          20              20              0    consumer-2-67890                               /127.0.0.1      consumer-2

→

--describe - Shows detailed information about the consumer group.

→

--group - Specifies which consumer group to describe.

Key Concept

If you remember nothing else from this pattern, remember: consumer groups let multiple consumers share the work of reading messages from a topic without duplication.

Common Mistakes

Starting multiple consumers with different group names to read the same topic expecting load sharing.

Different groups each get all messages, so messages are duplicated instead of shared.

Use the same consumer group name for all consumers that should share the work.

Not specifying a consumer group when starting a consumer.

Without a group, the consumer acts alone and does not share or track offsets properly.

Always specify a consumer group with --group to enable coordinated consumption.

Assuming messages are deleted after consumption.

Kafka retains messages based on time or size, not consumption, so messages remain until retention expires.

Understand that consumer groups track offsets to know what is read, but messages stay in the topic.

Summary

Use the --group flag to assign consumers to the same group for shared message processing.

Check consumer group status and lag with kafka-consumer-groups --describe to monitor progress.

Consumer groups ensure each message is processed once by one consumer in the group.

Practice

1. What is the main purpose of a consumer group in Kafka?

easy

A. To allow multiple consumers to share the work of reading messages

B. To store messages permanently in Kafka

C. To produce messages to Kafka topics

D. To encrypt messages during transmission

Solution

1. Step 1: Understand consumer group role

   A consumer group lets multiple consumers work together to read messages from a topic.

2. Step 2: Compare other options

   Storing messages, producing messages, and encrypting are not roles of consumer groups.

3. Final Answer:

   To allow multiple consumers to share the work of reading messages -> Option A

4. Quick Check:

   Consumer group = share reading work [OK]

Hint: Consumer groups split message reading among consumers [OK]

Common Mistakes:

• Confusing consumer groups with producers
• Thinking consumer groups store messages
• Assuming consumer groups encrypt data

2. Which of the following is the correct way to specify a consumer group ID in Kafka consumer configuration?

easy

A. "groupName" : "my-group"

B. "group.id" : "my-group"

C. "consumer.group" : "my-group"

D. "consumerId" : "my-group"

Solution

1. Step 1: Recall Kafka consumer config keys

   The correct key to set the consumer group is "group.id".

2. Step 2: Check other options

   Other keys like "consumer.group", "groupName", or "consumerId" are incorrect or do not exist.

3. Final Answer:

   "group.id" : "my-group" -> Option B

4. Quick Check:

   Consumer group ID key = "group.id" [OK]

Hint: Remember Kafka uses "group.id" for consumer groups [OK]

Common Mistakes:

• Using incorrect config keys
• Confusing consumer ID with group ID
• Misspelling the key name

3. Given a Kafka topic with 4 partitions and a consumer group with 2 consumers, how many partitions will each consumer read from?

medium

A. Each consumer reads from 2 partitions

B. Each consumer reads from 4 partitions

C. Each consumer reads from 1 partition

D. Each consumer reads from 0 partitions

Solution

1. Step 1: Understand partition assignment

   Partitions are divided among consumers in a group. With 4 partitions and 2 consumers, partitions split evenly.

2. Step 2: Calculate partitions per consumer

   4 partitions / 2 consumers = 2 partitions per consumer.

3. Final Answer:

   Each consumer reads from 2 partitions -> Option A

4. Quick Check:

   Partitions per consumer = total partitions / consumers = 2 [OK]

Hint: Partitions divide evenly among consumers in a group [OK]

Common Mistakes:

• Assuming each consumer reads all partitions
• Thinking partitions are not shared
• Ignoring partition count

4. What is wrong with this Kafka consumer group code snippet?

props.put("group.id", "my-group");
props.put("enable.auto.commit", "false");
consumer.subscribe(Arrays.asList("my-topic"));
consumer.poll(Duration.ofMillis(100));
// Missing commit after poll

medium

A. The poll timeout is too long

B. The group.id is incorrectly set

C. The consumer subscribes to the wrong topic

D. The consumer never commits offsets, risking message reprocessing

Solution

1. Step 1: Analyze auto commit setting

   Auto commit is disabled, so offsets must be committed manually.

2. Step 2: Check code for commit call

   Code calls poll but does not commit offsets, so messages may be reprocessed on restart.

3. Final Answer:

   The consumer never commits offsets, risking message reprocessing -> Option D

4. Quick Check:

   Auto commit false needs manual commit [OK]

Hint: Disable auto commit means manual commit needed [OK]

Common Mistakes:

• Forgetting to commit offsets manually
• Assuming auto commit is enabled by default
• Ignoring poll timeout relevance

5. You have a Kafka topic with 3 partitions and a consumer group with 4 consumers. What happens to the extra consumer?

hard

A. The extra consumer causes an error in the group

B. The extra consumer reads from all partitions

C. The extra consumer remains idle and does not receive messages

D. Kafka creates a new partition for the extra consumer

Solution

1. Step 1: Understand partition to consumer assignment

   Each partition can be assigned to only one consumer in the group.

2. Step 2: Analyze consumer count vs partitions

   With 3 partitions and 4 consumers, only 3 consumers get partitions; the 4th consumer stays idle.

3. Final Answer:

   The extra consumer remains idle and does not receive messages -> Option C

4. Quick Check:

   More consumers than partitions means idle consumers [OK]

Hint: Consumers > partitions means some consumers idle [OK]

Common Mistakes:

• Thinking Kafka creates new partitions automatically
• Assuming extra consumers share partitions
• Believing extra consumers cause errors