Static group membership in Kafka - Mini Project: Build & Apply

Static Group Membership in Kafka Consumer

📖 Scenario: You are building a Kafka consumer application that needs to join a consumer group with static membership. This helps Kafka remember your consumer instance even if it disconnects temporarily, avoiding unnecessary rebalances.

🎯 Goal: Create a Kafka consumer configuration that uses static group membership by setting the group.instance.id. Then, subscribe to a topic and print the assigned partitions.

📋 What You'll Learn

Create a Kafka consumer configuration dictionary with group.id set to my-static-group

Add group.instance.id set to consumer-1 to enable static membership

Subscribe the consumer to the topic my-topic

Print the assigned partitions after subscription

💡 Why This Matters

🌍 Real World

Static group membership is useful in real-world Kafka consumer applications where you want to avoid unnecessary rebalances when consumers restart or temporarily disconnect.

💼 Career

Understanding static group membership helps you build more stable and efficient Kafka consumer applications, a valuable skill for backend developers and data engineers working with Kafka.

1

Create Kafka consumer configuration

Create a dictionary called consumer_config with these exact entries: 'bootstrap.servers': 'localhost:9092' and 'group.id': 'my-static-group'.

Kafka

# Create the consumer_config dictionary with bootstrap.servers and group.id
# Your code here

Hint

Use a Python dictionary with keys 'bootstrap.servers' and 'group.id'.

2

Add static group membership ID

Add the entry 'group.instance.id': 'consumer-1' to the existing consumer_config dictionary.

Kafka

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'my-static-group'
}
# Add 'group.instance.id' with value 'consumer-1' to consumer_config
# Your code here

Hint

Add the key 'group.instance.id' with the value 'consumer-1' inside the dictionary.

3

Subscribe consumer to topic

Create a Kafka consumer using consumer_config and subscribe it to the topic 'my-topic'. Use variable name consumer for the consumer instance.

Kafka

from confluent_kafka import Consumer

consumer = Consumer(consumer_config)
# Subscribe consumer to 'my-topic'
# Your code here

Hint

Use consumer.subscribe(['my-topic']) to subscribe to the topic.

4

Print assigned partitions

Print the assigned partitions of the consumer by calling consumer.assignment() and printing the result.

Kafka

# Print the assigned partitions
# Your code here

Hint

Use print(consumer.assignment()) to show the assigned partitions. It may print an empty list if no partitions are assigned yet.

Practice

1.

What is the main purpose of using group.instance.id in Kafka consumer groups?

easy

A. To increase the number of partitions in a topic

B. To encrypt messages within the consumer group

C. To automatically commit offsets more frequently

D. To assign a stable identity to a consumer and enable static group membership

Solution

1. Step 1: Understand the role of group.instance.id

   This configuration assigns a unique and stable identity to a Kafka consumer within a group.

2. Step 2: Connect stable identity to static group membership

   With a stable identity, Kafka can avoid unnecessary rebalances when a consumer restarts briefly, improving stability.

3. Final Answer:

   To assign a stable identity to a consumer and enable static group membership -> Option D

4. Quick Check:

   Stable consumer identity = static group membership [OK]

Hint: Static membership means stable consumer ID to avoid rebalances [OK]

Common Mistakes:

• Confusing group.instance.id with partition count
• Thinking it controls offset commit frequency
• Assuming it encrypts messages

2.

Which of the following is the correct way to set group.instance.id in a Kafka consumer configuration?

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
// What is the correct line to set static membership?

easy

A. props.put("group.instance.id", "consumer-1");

B. props.put("group.id.instance", "consumer-1");

C. props.put("static.group.id", "consumer-1");

D. props.put("group.instance", "consumer-1");

Solution

1. Step 1: Recall the exact property name for static membership

   The correct property to set a static consumer ID is group.instance.id.

2. Step 2: Match the correct syntax in Java Properties

   Using props.put("group.instance.id", "consumer-1") correctly sets the static ID.

3. Final Answer:

   props.put("group.instance.id", "consumer-1"); -> Option A

4. Quick Check:

   Correct property name = group.instance.id [OK]

Hint: Remember exact property name: group.instance.id [OK]

Common Mistakes:

• Swapping words in the property name
• Using incomplete or incorrect property keys
• Confusing with group.id

3.

Consider this Kafka consumer group with static membership enabled using group.instance.id. If a consumer with group.instance.id = "consumer-1" disconnects briefly and reconnects, what happens?

medium

A. The consumer is removed permanently from the group

B. The group triggers a full rebalance and reassigns partitions

C. The consumer resumes without triggering a rebalance

D. The consumer must use a new group.instance.id to reconnect

Solution

1. Step 1: Understand static membership behavior on brief disconnect

   Static membership keeps the consumer identity stable, so brief disconnects do not cause rebalances.

2. Step 2: Identify the effect on partition assignment

   The consumer resumes with the same partitions without triggering a rebalance, improving latency and stability.

3. Final Answer:

   The consumer resumes without triggering a rebalance -> Option C

4. Quick Check:

   Static membership avoids rebalance on brief disconnect [OK]

Hint: Static ID lets consumer resume without rebalance [OK]

Common Mistakes:

• Assuming every disconnect triggers rebalance
• Thinking consumer is removed permanently
• Believing new ID is required on reconnect

4.

Given this Kafka consumer config snippet, identify the error preventing static group membership:

props.put("group.id", "my-group");
props.put("group.instance.id", "consumer-1");
props.put("session.timeout.ms", "10000");
// Missing property?

medium

A. Missing group.protocol.type set to consumer

B. Missing enable.auto.commit property

C. Missing partition.assignment.strategy for static membership

D. Missing client.id property

Solution

1. Step 1: Check required properties for static membership

   Static membership requires group.protocol.type to be set to consumer explicitly in some Kafka versions.

2. Step 2: Identify missing property in config

   The snippet lacks group.protocol.type, which can prevent static membership from working properly.

3. Final Answer:

   Missing group.protocol.type set to consumer -> Option A

4. Quick Check:

   Static membership needs group.protocol.type=consumer [OK]

Hint: Set group.protocol.type=consumer for static membership [OK]

Common Mistakes:

• Assuming auto commit affects static membership
• Confusing client.id with group.instance.id
• Ignoring group.protocol.type requirement

5.

You have a Kafka consumer group with 3 consumers using static membership via group.instance.id. One consumer crashes and restarts quickly with the same group.instance.id. What is the expected behavior and why is static membership important here?

hard

A. The group pauses all consumption until manual intervention

B. The group avoids a rebalance, so processing continues smoothly with minimal delay

C. The restarted consumer must use a new group.instance.id or it will be rejected

D. The group triggers a rebalance to reassign partitions to other consumers immediately

Solution

1. Step 1: Analyze static membership effect on consumer restart

   Static membership keeps the consumer identity stable, so a quick restart does not cause a rebalance.

2. Step 2: Understand impact on group processing

   Because no rebalance occurs, partition assignments remain unchanged, allowing smooth and low-latency processing.

3. Final Answer:

   The group avoids a rebalance, so processing continues smoothly with minimal delay -> Option B

4. Quick Check:

   Static membership prevents rebalance on quick restart [OK]

Hint: Static membership keeps group stable during quick restarts [OK]

Common Mistakes:

• Expecting rebalance on every consumer restart
• Thinking new ID is required after crash
• Assuming group pauses automatically