Subscribing to topics in Kafka - Deep Dive

Overview - Subscribing to topics

What is it?

Subscribing to topics in Kafka means telling a consumer to listen for messages from one or more named channels called topics. A topic is like a category or feed where messages are published. When a consumer subscribes, it receives new messages as they arrive, allowing applications to react to data in real time.

Why it matters

Without subscribing to topics, consumers would not know where to get data from Kafka. This would make it impossible to build systems that react instantly to events, like updating a dashboard or processing orders. Subscribing organizes how data flows from producers to consumers, enabling scalable and reliable communication.

Where it fits

Before learning to subscribe, you should understand Kafka topics and how producers send messages. After mastering subscription, you can learn about consumer groups, offset management, and message processing patterns to build robust streaming applications.

Mental Model

Core Idea

Subscribing to topics is like tuning a radio to specific stations to receive broadcasts you want to hear.

Think of it like...

Imagine a radio listener choosing which stations to listen to. Each station plays different music or news. By tuning in, you get only the content you want, just like a Kafka consumer subscribes to topics to get relevant messages.

┌─────────────┐       ┌───────────────┐       ┌───────────────┐
│  Producer   │──────▶│    Topic A    │──────▶│  Consumer 1   │
└─────────────┘       └───────────────┘       └───────────────┘
                             │
                             │
                             ▼
                      ┌───────────────┐       ┌───────────────┐
                      │    Topic B    │──────▶│  Consumer 2   │
                      └───────────────┘       └───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Kafka Topics

Concept: Learn what a Kafka topic is and how it organizes messages.

A Kafka topic is a named stream of messages. Producers send messages to topics, and consumers read from them. Topics are divided into partitions for scalability and parallelism.

Result

You know that topics are the channels where messages live in Kafka.

Understanding topics is essential because subscribing means choosing which of these channels to listen to.

2

Foundation

What Is Subscribing to a Topic

Concept: Learn the basic idea of subscribing as a way for consumers to receive messages.

Subscribing means a consumer tells Kafka which topics it wants to receive messages from. Kafka then delivers new messages from those topics to the consumer.

Result

You understand that subscribing connects consumers to topics to get data.

Knowing that subscription is the link between topics and consumers helps you see how data flows in Kafka.

3

Intermediate

Using Kafka Consumer API to Subscribe

🤔Before reading on: do you think subscribing requires specifying partitions or just topic names? Commit to your answer.

Concept: Learn how to use Kafka's consumer API to subscribe to topics by name.

In Kafka's consumer API, you call consumer.subscribe(Collections.singletonList("topic-name")) to subscribe to a topic. This tells Kafka to assign partitions automatically and deliver messages from that topic.

Result

The consumer starts receiving messages from the subscribed topic partitions.

Understanding that subscription by topic name lets Kafka handle partition assignment simplifies consumer code and scaling.

4

Intermediate

Subscribing to Multiple Topics

🤔Before reading on: can a consumer subscribe to multiple topics at once? Commit to yes or no.

Concept: Learn that consumers can subscribe to many topics simultaneously.

You can pass a list of topic names to consumer.subscribe(Arrays.asList("topic1", "topic2")) to listen to multiple topics. Kafka will deliver messages from all these topics to the consumer.

Result

The consumer receives messages from all subscribed topics in a single stream.

Knowing that multiple subscriptions are possible allows flexible data consumption from various sources.

5

Intermediate

Using Pattern-Based Subscription

🤔Before reading on: do you think you can subscribe to topics using name patterns (like wildcards)? Commit to yes or no.

Concept: Learn that Kafka supports subscribing to topics matching a regular expression pattern.

Instead of listing topics, you can use consumer.subscribe(Pattern.compile("topic-prefix.*")) to subscribe to all topics whose names match the pattern. This is useful when topics are created dynamically.

Result

The consumer automatically receives messages from any topic matching the pattern, even if created later.

Pattern subscription enables dynamic and scalable consumption without manual topic list updates.

6

Advanced

Partition Assignment and Rebalancing

🤔Before reading on: do you think Kafka assigns partitions to consumers manually or automatically? Commit to your answer.

Concept: Understand how Kafka assigns topic partitions to consumers in a group and rebalances on changes.

When consumers subscribe, Kafka assigns partitions so each message is consumed by only one consumer in the group. If consumers join or leave, Kafka rebalances partitions among them to maintain load balance.

Result

Consumers get exclusive partitions to read from, ensuring no duplicate processing within the group.

Knowing partition assignment and rebalancing explains how Kafka achieves fault tolerance and scalability.

7

Expert

Handling Subscription Changes and Offset Management

🤔Before reading on: do you think changing subscriptions resets message reading positions automatically? Commit to yes or no.

Concept: Learn how changing subscriptions affects consumer offsets and how to manage them safely.

When a consumer changes its subscription, Kafka may rebalance partitions and reset offsets depending on configuration. Proper offset management ensures consumers resume reading where they left off, avoiding message loss or duplication.

Result

Consumers maintain consistent message processing even when subscriptions change dynamically.

Understanding offset behavior during subscription changes prevents common bugs in production streaming systems.

Under the Hood

Kafka consumers maintain a session with the Kafka broker cluster. When subscribing, the consumer sends its topic list or pattern to the broker coordinator. The coordinator assigns partitions to consumers in the same group to balance load. The consumer fetches messages from assigned partitions, tracking offsets locally or in Kafka. Rebalancing occurs when group membership changes, triggering partition reassignment and offset synchronization.

Why designed this way?

Kafka was designed for high throughput and fault tolerance. Automatic partition assignment and rebalancing simplify scaling and recovery. Using topic names or patterns for subscription allows flexible consumption without manual partition management. Offset tracking ensures exactly-once or at-least-once processing guarantees depending on configuration.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│  Consumer 1   │──────▶│ Partition 0   │       │   Topic A     │
│  Consumer 2   │──────▶│ Partition 1   │──────▶│               │
│  Consumer 3   │──────▶│ Partition 2   │       │               │
└───────────────┘       └───────────────┘       └───────────────┘
          ▲                     ▲                       ▲
          │                     │                       │
          └─────────Coordinator ────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does subscribing to a topic mean you get all past messages automatically? Commit to yes or no.

Common Belief: Subscribing to a topic means the consumer receives every message ever sent to that topic.

Reality: Consumers receive messages starting from their configured offset, which can be the latest, earliest, or a specific position. Past messages before the offset are not automatically sent.

Why it matters: Assuming all past messages arrive can cause missed data or duplicate processing if offsets are mismanaged.

Quick: Can multiple consumers in the same group read the same partition simultaneously? Commit to yes or no.

Common Belief: Multiple consumers in the same group can read the same partition at the same time for faster processing.

Reality: Kafka assigns each partition to only one consumer in a group to avoid duplicate processing. Multiple consumers read different partitions.

Why it matters: Misunderstanding this leads to incorrect assumptions about parallelism and can cause data duplication or processing errors.

Quick: Does subscribing with a pattern subscribe only to existing topics or also future ones? Commit to your answer.

Common Belief: Pattern subscription only applies to topics that exist at the time of subscription.

Reality: Pattern subscription automatically includes new topics created later that match the pattern.

Why it matters: Not knowing this can cause unexpected message consumption or missed data when topics are dynamically created.

Quick: Does changing the subscription list during runtime cause message loss? Commit to yes or no.

Common Belief: Changing subscriptions on the fly is safe and does not affect message processing.

Reality: Changing subscriptions triggers rebalancing, which can cause temporary pauses and requires careful offset management to avoid message loss or duplication.

Why it matters: Ignoring this can cause production outages or inconsistent data processing.

Expert Zone

1

Partition assignment strategies (range, round-robin, sticky) affect load balancing and consumer stability in subtle ways.

2

Offset commits can be automatic or manual; choosing the right mode impacts processing guarantees and performance.

3

Subscription patterns can cause unexpected topic matches if naming conventions are not carefully controlled.

When NOT to use

Subscribing directly to topics is not ideal when you need fine-grained control over partitions or want to implement custom load balancing. In such cases, manual partition assignment or Kafka Streams API might be better alternatives.

Production Patterns

In production, consumers often use consumer groups with pattern subscriptions to handle dynamic topic creation. Offset management is carefully tuned for exactly-once processing. Rebalancing listeners are implemented to handle partition changes gracefully without downtime.

Connections

Publish-Subscribe Messaging Pattern

Subscribing to Kafka topics is a concrete implementation of the publish-subscribe pattern.

Understanding Kafka subscription deepens comprehension of how pub-sub systems decouple message producers and consumers for scalable communication.

Load Balancing in Distributed Systems

Kafka's partition assignment during subscription is a form of load balancing among consumers.

Knowing Kafka's subscription mechanism helps grasp general load balancing strategies in distributed computing.

Radio Broadcasting

Subscribing to topics is like tuning into radio stations to receive broadcasts.

This cross-domain connection clarifies how selective listening enables efficient information flow.

Common Pitfalls

#1 Assuming subscribing to a topic automatically reads all past messages.

Wrong approach:

consumer.subscribe(Collections.singletonList("my-topic")); // then expecting all old messages without setting offset

Correct approach:

consumer.subscribe(Collections.singletonList("my-topic"));
consumer.seekToBeginning(consumer.assignment()); // to read from earliest offset

Root cause:

Misunderstanding that subscription alone does not control offset position.

#2 Multiple consumers in the same group subscribing to the same partition causing duplicate processing.

Wrong approach:

Manually assigning the same partition to multiple consumers in one group.

Correct approach:

Use consumer.subscribe() and let Kafka assign partitions uniquely per consumer in the group.

Root cause:

Ignoring Kafka's partition assignment rules and trying manual assignment without coordination.

#3 Changing subscription topics without handling rebalancing causing message loss.

Wrong approach:

consumer.subscribe(newTopics); // without committing offsets or handling rebalance events

Correct approach:

Commit offsets before changing subscription and implement rebalance listener to manage partition changes safely.

Root cause:

Not accounting for Kafka's rebalancing process triggered by subscription changes.

Key Takeaways

Subscribing to Kafka topics connects consumers to the streams of messages they want to process.

Consumers subscribe by topic names or patterns, and Kafka manages partition assignment automatically.

Partition assignment ensures each partition is read by only one consumer in a group, enabling parallelism without duplication.

Offset management during subscription and rebalancing is critical to avoid message loss or duplication.

Understanding subscription internals helps build scalable, fault-tolerant streaming applications with Kafka.

Practice

1. What is the main purpose of the subscribe() method in Kafka consumers?

easy

A. To close the Kafka consumer connection

B. To send messages to Kafka topics

C. To listen to one or more Kafka topics and receive messages automatically

D. To create new Kafka topics

Solution

1. Step 1: Understand the role of subscribe()

   The subscribe() method is used by Kafka consumers to specify which topics they want to listen to.

2. Step 2: Identify what subscribe() enables

   Once subscribed, the consumer automatically receives messages from those topics when calling poll().

3. Final Answer:

   To listen to one or more Kafka topics and receive messages automatically -> Option C

4. Quick Check:

   subscribe() = listen to topics [OK]

Hint: subscribe() means start listening to topics [OK]

Common Mistakes:

• Confusing subscribe() with sending messages
• Thinking subscribe() creates topics
• Assuming subscribe() closes connections

2. Which of the following is the correct way to subscribe a Kafka consumer to multiple topics named "topicA" and "topicB"?

easy

A. consumer.subscribe(["topicA", "topicB"])

B. consumer.subscribe("topicA", "topicB")

C. consumer.subscribe("topicA" + "," + "topicB")

D. consumer.subscribe({"topicA", "topicB"})

Solution

1. Step 1: Recall the subscribe() method signature

   In Kafka, subscribe() takes a list (or collection) of topic names as an argument.

2. Step 2: Identify correct syntax for multiple topics

   Passing a list like ["topicA", "topicB"] is the correct way to subscribe to multiple topics.

3. Final Answer:

   consumer.subscribe(["topicA", "topicB"]) -> Option A

4. Quick Check:

   subscribe() needs a list for multiple topics [OK]

Hint: Use a list to subscribe to multiple topics [OK]

Common Mistakes:

• Passing multiple string arguments instead of a list
• Concatenating topic names into one string
• Using curly braces instead of square brackets

3. Given the following Kafka consumer code snippet, what will be the output if the consumer is subscribed to topic "news" and there are two messages "Hello" and "World" in the topic?

consumer.subscribe(["news"])
records = consumer.poll(timeout_ms=1000)
for record in records:
    print(record.value)

medium

A. ['Hello', 'World']

B. Hello\nWorld

C. HelloWorld

D. No output because poll() is missing a loop

Solution

1. Step 1: Understand subscribe() and poll() behavior

   The consumer subscribes to "news" and calls poll() once, which fetches available messages.

2. Step 2: Analyze the for loop printing each record

   The loop prints each message value on its own line, so "Hello" and "World" appear separately.

3. Final Answer:

   Hello\nWorld -> Option B

4. Quick Check:

   poll() returns messages, loop prints each [OK]

Hint: poll() returns messages; loop prints each on new line [OK]

Common Mistakes:

• Thinking poll() returns a list printed directly
• Assuming messages print concatenated without newline
• Believing poll() needs a loop to work at all

4. What is wrong with the following Kafka consumer code snippet?

consumer.subscribe("topic1")
while True:
    records = consumer.poll(1000)
    for record in records:
        print(record.value)

medium

A. subscribe() expects a list, not a single string

B. poll() must be called outside the loop

C. print() cannot be used inside the loop

D. The loop should be a for loop, not while

Solution

1. Step 1: Check subscribe() argument type

   The subscribe() method requires a list of topics, even if subscribing to one topic.

2. Step 2: Identify the error in argument

   Passing a single string instead of a list causes a type error or unexpected behavior.

3. Final Answer:

   subscribe() expects a list, not a single string -> Option A

4. Quick Check:

   subscribe() needs list argument [OK]

Hint: Always pass a list to subscribe(), even for one topic [OK]

Common Mistakes:

• Passing a string instead of a list to subscribe()
• Thinking poll() can't be inside a loop
• Misunderstanding loop type requirements

5. You want to subscribe a Kafka consumer to all topics starting with "logs-" dynamically, without listing each topic name. Which approach correctly achieves this?

hard

A. Use consumer.assign() with topic partitions matching "logs-"

B. Manually list all topics starting with "logs-" in a list and pass to subscribe()

C. Call subscribe() multiple times, once per topic starting with "logs-"

D. Use consumer.subscribe(Pattern.compile("logs-.*")) to subscribe by regex pattern

Solution

1. Step 1: Understand dynamic topic subscription

   Kafka consumers can subscribe to topics matching a regex pattern using subscribe(Pattern).

2. Step 2: Identify correct method for pattern subscription

   Using consumer.subscribe(Pattern.compile("logs-.*")) subscribes to all topics starting with "logs-" automatically.

3. Final Answer:

   Use consumer.subscribe(Pattern.compile("logs-.*")) to subscribe by regex pattern -> Option D

4. Quick Check:

   subscribe() supports regex patterns [OK]

Hint: Use regex pattern with subscribe() for dynamic topic matching [OK]

Common Mistakes:

• Listing topics manually instead of using regex
• Calling subscribe() multiple times incorrectly
• Using assign() which requires partitions, not topics