Consumer API basics in Kafka - Time & Space Complexity

Time Complexity: Consumer API basics

O(n)

Understanding Time Complexity

When using Kafka's Consumer API, it's important to know how the time to process messages changes as the number of messages grows.

We want to understand how the consumer's work scales when reading many messages.

Scenario Under Consideration

Analyze the time complexity of the following Kafka consumer code snippet.

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("my-topic"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        System.out.println(record.value());
    }
}

This code connects to Kafka, subscribes to a topic, and continuously polls for new messages, processing each one.

Identify Repeating Operations

Look at what repeats as the consumer runs.

• Primary operation: Polling messages and iterating over each message in the batch.
• How many times: The poll loop runs indefinitely, and inside each poll, the for-loop runs once per message received.

How Execution Grows With Input

As the number of messages increases, the consumer processes more messages each poll.

Input Size (messages per poll)	Approx. Operations (prints)
10	10
100	100
1000	1000

Pattern observation: The work grows directly with the number of messages received each time.

Final Time Complexity

Time Complexity: O(n)

This means the time to process messages grows linearly with the number of messages received.

Common Mistake

[X] Wrong: "Polling once will always take the same time regardless of messages."

[OK] Correct: The poll call returns a batch of messages, and processing each message takes time, so more messages mean more work.

Interview Connect

Understanding how message processing time grows helps you design efficient consumers and shows you can reason about real-world streaming data.

Self-Check

"What if we processed messages in parallel inside the poll loop? How would that affect the time complexity?"

Practice

1. What is the primary purpose of the Kafka Consumer API?

easy

A. To send messages to Kafka topics

B. To delete Kafka topics

C. To create Kafka topics

D. To read messages from Kafka topics

Solution

1. Step 1: Understand Kafka Consumer API role

   The Consumer API is designed to read or consume messages from Kafka topics.

2. Step 2: Differentiate from Producer API

   The Producer API sends messages, while Consumer API reads them.

3. Final Answer:

   To read messages from Kafka topics -> Option D

4. Quick Check:

   Consumer API = Read messages [OK]

Hint: Consumer API reads messages, Producer API sends messages [OK]

Common Mistakes:

• Confusing Consumer API with Producer API
• Thinking Consumer API creates or deletes topics
• Mixing up reading and writing operations

2. Which of the following is the correct way to create a Kafka consumer in Java?

easy

A. new KafkaProducer<>(properties);

B. new KafkaConsumer<>(properties);

C. KafkaConsumer.create(properties);

D. KafkaConsumer.start(properties);

Solution

1. Step 1: Recall Kafka consumer instantiation syntax

   Kafka consumer is created by calling the constructor: new KafkaConsumer<>(properties);

2. Step 2: Eliminate incorrect options

   Producer is for sending messages, so new KafkaProducer<>(properties); is wrong. Options C and D are not valid KafkaConsumer methods.

3. Final Answer:

   new KafkaConsumer<>(properties); -> Option B

4. Quick Check:

   Constructor new KafkaConsumer<>(props) = Correct [OK]

Hint: Use new KafkaConsumer<>(props) to create consumer [OK]

Common Mistakes:

• Using KafkaProducer instead of KafkaConsumer
• Calling non-existent static methods
• Confusing method names for creation

3. Given the following Java code snippet, what will be the output if the topic has two messages "Hello" and "World"?

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("my-topic"));
ConsumerRecords records = consumer.poll(Duration.ofMillis(1000));
for (ConsumerRecord record : records) {
    System.out.println(record.value());
}

medium

A. Compilation error

B. No output

C. Hello\nWorld

D. Runtime exception

Solution

1. Step 1: Understand consumer subscription and poll

   The consumer subscribes to "my-topic" and polls messages for 1 second.

2. Step 2: Process messages in the topic

   Since the topic has two messages "Hello" and "World", the poll will fetch both, and the loop prints each value.

3. Final Answer:

   Hello\nWorld -> Option C

4. Quick Check:

   poll() fetches messages = Hello and World printed [OK]

Hint: poll() fetches messages; loop prints each value [OK]

Common Mistakes:

• Assuming poll returns no messages immediately
• Confusing subscribe with assign
• Expecting compile or runtime errors without cause

4. Identify the error in this Kafka consumer code snippet:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "group1");
props.put("key.deserializer", "StringDeserializer");
props.put("value.deserializer", "StringDeserializer");
KafkaConsumer consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("topic1"));
ConsumerRecords records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord record : records) {
    System.out.println(record.value());
}

medium

A. Deserializer class names are incomplete and will cause ClassNotFoundException

B. Missing call to consumer.close()

C. Incorrect subscription method used

D. poll() duration is too short and causes timeout

Solution

1. Step 1: Check deserializer configuration

   The deserializer class names must be fully qualified, e.g., "org.apache.kafka.common.serialization.StringDeserializer".

2. Step 2: Identify impact of incorrect class names

   Using "StringDeserializer" alone will cause ClassNotFoundException at runtime.

3. Final Answer:

   Deserializer class names are incomplete and will cause ClassNotFoundException -> Option A

4. Quick Check:

   Deserializer full class name required = Deserializer class names are incomplete and will cause ClassNotFoundException [OK]

Hint: Use full deserializer class names to avoid errors [OK]

Common Mistakes:

• Using short class names for deserializers
• Ignoring missing consumer.close() (not error here)
• Assuming poll duration causes errors

5. You want to process messages from multiple topics and commit offsets manually after processing. Which code snippet correctly sets up the Kafka consumer for this use case?

hard

A. props.put("enable.auto.commit", "false");
consumer.subscribe(Arrays.asList("topicA", "topicB"));
while (true) {
ConsumerRecords records = consumer.poll(Duration.ofMillis(1000));
for (ConsumerRecord record : records) {
process(record.value());
}
consumer.commitSync();
}

B. props.put("enable.auto.commit", "true");
consumer.subscribe(Collections.singletonList("topicA"));
while (true) {
ConsumerRecords records = consumer.poll(Duration.ofMillis(1000));
for (ConsumerRecord record : records) {
process(record.value());
}
}

C. props.put("enable.auto.commit", "false");
consumer.assign(Arrays.asList(new TopicPartition("topicA", 0)));
consumer.subscribe(Arrays.asList("topicB"));
while (true) {
ConsumerRecords records = consumer.poll(Duration.ofMillis(1000));
for (ConsumerRecord record : records) {
process(record.value());
}
consumer.commitAsync();
}

D. props.put("enable.auto.commit", "false");
consumer.subscribe(Arrays.asList("topicA", "topicB"));
while (true) {
ConsumerRecords records = consumer.poll(Duration.ofMillis(1000));
for (ConsumerRecord record : records) {
process(record.value());
}
}

Solution

1. Step 1: Disable auto commit for manual offset control

   Setting "enable.auto.commit" to "false" allows manual offset commits.

2. Step 2: Subscribe to multiple topics correctly

   Use consumer.subscribe(Arrays.asList("topicA", "topicB")) to listen to multiple topics.

3. Step 3: Commit offsets after processing

   Calling consumer.commitSync() after processing ensures offsets are saved manually.

4. Final Answer:

   Code snippet with manual commit, multiple topic subscription, and commitSync() -> Option A

5. Quick Check:

   Manual commit + multiple topics = props.put("enable.auto.commit", "false"); consumer.subscribe(Arrays.asList("topicA", "topicB")); while (true) { ConsumerRecords records = consumer.poll(Duration.ofMillis(1000)); for (ConsumerRecord record : records) { process(record.value()); } consumer.commitSync(); } [OK]

Hint: Disable auto commit and call commitSync() after processing [OK]

Common Mistakes:

• Using auto commit when manual commit is needed
• Mixing assign() and subscribe() incorrectly
• Forgetting to commit offsets after processing