Consumer group concept in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a consumer group in Kafka?

A consumer group is a set of consumers that work together to read data from Kafka topics. Each consumer in the group reads from a unique set of partitions, allowing parallel processing.

intermediate

How does Kafka ensure that messages are processed only once within a consumer group?

Kafka assigns each partition to only one consumer in the group at a time, so messages from a partition are processed by only one consumer, preventing duplicate processing within the group.

intermediate

What happens if a consumer in a consumer group fails?

Kafka automatically reassigns the partitions of the failed consumer to other consumers in the group, ensuring continuous message processing without data loss.

beginner

Can multiple consumer groups read from the same Kafka topic independently?

Yes, each consumer group reads independently from the topic. This allows multiple applications or services to consume the same data without interfering with each other.

intermediate

What is the role of partition assignment in a consumer group?

Partition assignment decides which consumer reads from which partition. It balances the load and ensures each partition is read by only one consumer in the group at a time.

What does a Kafka consumer group do?

A Creates new topics automatically

B Duplicates messages to all consumers

C Stores messages permanently

D Distributes partitions among consumers for parallel processing

A consumer group divides partitions among its consumers so they can process messages in parallel without overlap.

If a consumer leaves a consumer group, what happens to its partitions?

A They stop producing messages

B They are deleted

C They are reassigned to other consumers in the group

D They remain unassigned forever

Kafka reassigns partitions from the leaving consumer to others to keep processing continuous.

Can two consumers in the same group read the same partition at the same time?

A Only if the topic has one partition

B No, only one consumer reads a partition at a time

C Only if they are on different machines

D Yes, always

Kafka ensures exclusive partition assignment within a consumer group to avoid duplicate processing.

Multiple consumer groups reading the same topic means:

A Each group reads independently without affecting others

B They share the same offsets

C Messages are deleted after one group reads them

D Only one group can read at a time

Each consumer group maintains its own offset and reads independently from the topic.

What is the main benefit of using consumer groups?

A Scalable and fault-tolerant message consumption

B Faster message production

C Automatic topic creation

D Message encryption

Consumer groups allow multiple consumers to share the load and handle failures gracefully.

Explain how Kafka consumer groups help in scaling message consumption.

Think about how work is divided among friends to finish faster.

Describe what happens when a consumer in a consumer group crashes.

Imagine a team member leaves and others pick up their tasks.

Practice

1. What is the main purpose of a consumer group in Kafka?

easy

A. To allow multiple consumers to share the work of reading messages

B. To store messages permanently in Kafka

C. To produce messages to Kafka topics

D. To encrypt messages during transmission

Solution

1. Step 1: Understand consumer group role

   A consumer group lets multiple consumers work together to read messages from a topic.

2. Step 2: Compare other options

   Storing messages, producing messages, and encrypting are not roles of consumer groups.

3. Final Answer:

   To allow multiple consumers to share the work of reading messages -> Option A

4. Quick Check:

   Consumer group = share reading work [OK]

Hint: Consumer groups split message reading among consumers [OK]

Common Mistakes:

• Confusing consumer groups with producers
• Thinking consumer groups store messages
• Assuming consumer groups encrypt data

2. Which of the following is the correct way to specify a consumer group ID in Kafka consumer configuration?

easy

A. "groupName" : "my-group"

B. "group.id" : "my-group"

C. "consumer.group" : "my-group"

D. "consumerId" : "my-group"

Solution

1. Step 1: Recall Kafka consumer config keys

   The correct key to set the consumer group is "group.id".

2. Step 2: Check other options

   Other keys like "consumer.group", "groupName", or "consumerId" are incorrect or do not exist.

3. Final Answer:

   "group.id" : "my-group" -> Option B

4. Quick Check:

   Consumer group ID key = "group.id" [OK]

Hint: Remember Kafka uses "group.id" for consumer groups [OK]

Common Mistakes:

• Using incorrect config keys
• Confusing consumer ID with group ID
• Misspelling the key name

3. Given a Kafka topic with 4 partitions and a consumer group with 2 consumers, how many partitions will each consumer read from?

medium

A. Each consumer reads from 2 partitions

B. Each consumer reads from 4 partitions

C. Each consumer reads from 1 partition

D. Each consumer reads from 0 partitions

Solution

1. Step 1: Understand partition assignment

   Partitions are divided among consumers in a group. With 4 partitions and 2 consumers, partitions split evenly.

2. Step 2: Calculate partitions per consumer

   4 partitions / 2 consumers = 2 partitions per consumer.

3. Final Answer:

   Each consumer reads from 2 partitions -> Option A

4. Quick Check:

   Partitions per consumer = total partitions / consumers = 2 [OK]

Hint: Partitions divide evenly among consumers in a group [OK]

Common Mistakes:

• Assuming each consumer reads all partitions
• Thinking partitions are not shared
• Ignoring partition count

4. What is wrong with this Kafka consumer group code snippet?

props.put("group.id", "my-group");
props.put("enable.auto.commit", "false");
consumer.subscribe(Arrays.asList("my-topic"));
consumer.poll(Duration.ofMillis(100));
// Missing commit after poll

medium

A. The poll timeout is too long

B. The group.id is incorrectly set

C. The consumer subscribes to the wrong topic

D. The consumer never commits offsets, risking message reprocessing

Solution

1. Step 1: Analyze auto commit setting

   Auto commit is disabled, so offsets must be committed manually.

2. Step 2: Check code for commit call

   Code calls poll but does not commit offsets, so messages may be reprocessed on restart.

3. Final Answer:

   The consumer never commits offsets, risking message reprocessing -> Option D

4. Quick Check:

   Auto commit false needs manual commit [OK]

Hint: Disable auto commit means manual commit needed [OK]

Common Mistakes:

• Forgetting to commit offsets manually
• Assuming auto commit is enabled by default
• Ignoring poll timeout relevance

5. You have a Kafka topic with 3 partitions and a consumer group with 4 consumers. What happens to the extra consumer?

hard

A. The extra consumer causes an error in the group

B. The extra consumer reads from all partitions

C. The extra consumer remains idle and does not receive messages

D. Kafka creates a new partition for the extra consumer

Solution

1. Step 1: Understand partition to consumer assignment

   Each partition can be assigned to only one consumer in the group.

2. Step 2: Analyze consumer count vs partitions

   With 3 partitions and 4 consumers, only 3 consumers get partitions; the 4th consumer stays idle.

3. Final Answer:

   The extra consumer remains idle and does not receive messages -> Option C

4. Quick Check:

   More consumers than partitions means idle consumers [OK]

Hint: Consumers > partitions means some consumers idle [OK]

Common Mistakes:

• Thinking Kafka creates new partitions automatically
• Assuming extra consumers share partitions
• Believing extra consumers cause errors