Subscribing to topics in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Kafka Consumer Subscription Behavior

What will be the output of this Kafka consumer code snippet when it subscribes to a topic and polls for records?

Kafka

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'my_topic',
    bootstrap_servers=['localhost:9092'],
    auto_offset_reset='earliest',
    consumer_timeout_ms=1000
)

messages = []
for message in consumer:
    messages.append(message.value.decode('utf-8'))

print(messages)

A An empty list [] because the consumer does not subscribe to any topic

B A list of all messages from the beginning of 'my_topic' if any exist, e.g., ['msg1', 'msg2']

C A runtime error because 'auto_offset_reset' is not a valid parameter

D A list containing only the latest message from 'my_topic'

💡 Hint

Check how 'auto_offset_reset' affects the starting point of message consumption.

The consumer subscribes to 'my_topic' and starts reading from the earliest offset due to 'auto_offset_reset="earliest"'. It collects all messages available during the poll and prints them as a list.

🧠 Conceptual

intermediate

Understanding Kafka Consumer Group Subscription

In Kafka, what happens when multiple consumers in the same group subscribe to the same topic?

A Messages are load balanced so each message is delivered to only one consumer in the group.

B Each consumer receives all messages from the topic independently.

C Only the first consumer receives messages; others remain idle.

D Consumers receive messages only if they subscribe to different topics.

💡 Hint

Think about how Kafka distributes partitions among consumers in a group.

Kafka divides topic partitions among consumers in the same group so each message is processed by only one consumer, enabling parallel processing without duplication.

🔧 Debug

advanced

Why does this Kafka consumer not receive messages?

Given this Kafka consumer code, why does it not receive any messages even though the topic has data?

Kafka

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    bootstrap_servers=['localhost:9092'],
    group_id='group1'
)
consumer.subscribe(['my_topic'])

for message in consumer:
    print(message.value.decode('utf-8'))

A The topic name is incorrect; it should be 'my-topic' with a dash.

B The consumer forgot to call 'poll()' before iterating over messages.

C The consumer must specify 'enable_auto_commit=False' to receive messages.

D The consumer does not specify 'auto_offset_reset', so it starts at the latest offset and misses existing messages.

💡 Hint

Check the default behavior of 'auto_offset_reset' when no committed offset exists.

Without 'auto_offset_reset' set to 'earliest', the consumer starts reading from the latest offset, so it misses all existing messages in the topic.

📝 Syntax

advanced

Identify the syntax error in Kafka consumer subscription code

Which option contains a syntax error in subscribing a Kafka consumer to multiple topics?

Kafka

from kafka import KafkaConsumer

consumer = KafkaConsumer(bootstrap_servers=['localhost:9092'])
consumer.subscribe(topics=['topic1', 'topic2'])

A consumer.subscribe(['topic1', 'topic2'])

B consumer.subscribe(topic=['topic1', 'topic2'])

C consumer.subscribe('topic1', 'topic2')

D consumer.subscribe(topics=['topic1', 'topic2'])

💡 Hint

Check the method signature for 'subscribe' in KafkaConsumer.

'subscribe' expects a single argument: a list of topic names or a pattern. Passing multiple string arguments without a list causes a syntax error.

🚀 Application

expert

Determine the number of partitions assigned to each consumer

You have a Kafka topic with 6 partitions and 3 consumers in the same group subscribing to it. How many partitions will each consumer be assigned?

A Each consumer will be assigned 2 partitions.

B Each consumer will be assigned all 6 partitions.

C One consumer gets 4 partitions, others get 1 each.

D Partitions are assigned randomly, so counts vary each time.

💡 Hint

Kafka tries to balance partitions evenly among consumers in a group.

Kafka assigns partitions evenly among consumers in a group. With 6 partitions and 3 consumers, each gets 2 partitions.

Practice

1. What is the main purpose of the subscribe() method in Kafka consumers?

easy

A. To close the Kafka consumer connection

B. To send messages to Kafka topics

C. To listen to one or more Kafka topics and receive messages automatically

D. To create new Kafka topics

Solution

1. Step 1: Understand the role of subscribe()

   The subscribe() method is used by Kafka consumers to specify which topics they want to listen to.

2. Step 2: Identify what subscribe() enables

   Once subscribed, the consumer automatically receives messages from those topics when calling poll().

3. Final Answer:

   To listen to one or more Kafka topics and receive messages automatically -> Option C

4. Quick Check:

   subscribe() = listen to topics [OK]

Hint: subscribe() means start listening to topics [OK]

Common Mistakes:

• Confusing subscribe() with sending messages
• Thinking subscribe() creates topics
• Assuming subscribe() closes connections

2. Which of the following is the correct way to subscribe a Kafka consumer to multiple topics named "topicA" and "topicB"?

easy

A. consumer.subscribe(["topicA", "topicB"])

B. consumer.subscribe("topicA", "topicB")

C. consumer.subscribe("topicA" + "," + "topicB")

D. consumer.subscribe({"topicA", "topicB"})

Solution

1. Step 1: Recall the subscribe() method signature

   In Kafka, subscribe() takes a list (or collection) of topic names as an argument.

2. Step 2: Identify correct syntax for multiple topics

   Passing a list like ["topicA", "topicB"] is the correct way to subscribe to multiple topics.

3. Final Answer:

   consumer.subscribe(["topicA", "topicB"]) -> Option A

4. Quick Check:

   subscribe() needs a list for multiple topics [OK]

Hint: Use a list to subscribe to multiple topics [OK]

Common Mistakes:

• Passing multiple string arguments instead of a list
• Concatenating topic names into one string
• Using curly braces instead of square brackets

3. Given the following Kafka consumer code snippet, what will be the output if the consumer is subscribed to topic "news" and there are two messages "Hello" and "World" in the topic?

consumer.subscribe(["news"])
records = consumer.poll(timeout_ms=1000)
for record in records:
    print(record.value)

medium

A. ['Hello', 'World']

B. Hello\nWorld

C. HelloWorld

D. No output because poll() is missing a loop

Solution

1. Step 1: Understand subscribe() and poll() behavior

   The consumer subscribes to "news" and calls poll() once, which fetches available messages.

2. Step 2: Analyze the for loop printing each record

   The loop prints each message value on its own line, so "Hello" and "World" appear separately.

3. Final Answer:

   Hello\nWorld -> Option B

4. Quick Check:

   poll() returns messages, loop prints each [OK]

Hint: poll() returns messages; loop prints each on new line [OK]

Common Mistakes:

• Thinking poll() returns a list printed directly
• Assuming messages print concatenated without newline
• Believing poll() needs a loop to work at all

4. What is wrong with the following Kafka consumer code snippet?

consumer.subscribe("topic1")
while True:
    records = consumer.poll(1000)
    for record in records:
        print(record.value)

medium

A. subscribe() expects a list, not a single string

B. poll() must be called outside the loop

C. print() cannot be used inside the loop

D. The loop should be a for loop, not while

Solution

1. Step 1: Check subscribe() argument type

   The subscribe() method requires a list of topics, even if subscribing to one topic.

2. Step 2: Identify the error in argument

   Passing a single string instead of a list causes a type error or unexpected behavior.

3. Final Answer:

   subscribe() expects a list, not a single string -> Option A

4. Quick Check:

   subscribe() needs list argument [OK]

Hint: Always pass a list to subscribe(), even for one topic [OK]

Common Mistakes:

• Passing a string instead of a list to subscribe()
• Thinking poll() can't be inside a loop
• Misunderstanding loop type requirements

5. You want to subscribe a Kafka consumer to all topics starting with "logs-" dynamically, without listing each topic name. Which approach correctly achieves this?

hard

A. Use consumer.assign() with topic partitions matching "logs-"

B. Manually list all topics starting with "logs-" in a list and pass to subscribe()

C. Call subscribe() multiple times, once per topic starting with "logs-"

D. Use consumer.subscribe(Pattern.compile("logs-.*")) to subscribe by regex pattern

Solution

1. Step 1: Understand dynamic topic subscription

   Kafka consumers can subscribe to topics matching a regex pattern using subscribe(Pattern).

2. Step 2: Identify correct method for pattern subscription

   Using consumer.subscribe(Pattern.compile("logs-.*")) subscribes to all topics starting with "logs-" automatically.

3. Final Answer:

   Use consumer.subscribe(Pattern.compile("logs-.*")) to subscribe by regex pattern -> Option D

4. Quick Check:

   subscribe() supports regex patterns [OK]

Hint: Use regex pattern with subscribe() for dynamic topic matching [OK]

Common Mistakes:

• Listing topics manually instead of using regex
• Calling subscribe() multiple times incorrectly
• Using assign() which requires partitions, not topics