Consumer poll loop in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of the consumer poll loop in Kafka?

The consumer poll loop continuously fetches messages from Kafka topics, allowing the consumer to process new data as it arrives.

beginner

In Kafka's consumer poll loop, what does the poll() method do?

The poll() method requests records from Kafka brokers and returns any available messages for the consumer to process.

intermediate

Why should the consumer poll loop run continuously without long blocking operations?

Because Kafka expects consumers to poll regularly to maintain their group membership and avoid rebalancing or session timeouts.

intermediate

What happens if a Kafka consumer does not call poll() within the session timeout?

The consumer is considered dead, triggering a rebalance where partitions are reassigned to other consumers.

intermediate

How can you safely commit offsets in a Kafka consumer poll loop?

You can commit offsets synchronously or asynchronously after processing messages to track progress and avoid duplicate processing.

What is the main role of the poll() method in a Kafka consumer loop?

A Create new Kafka topics

B Send messages to Kafka brokers

C Delete Kafka partitions

D Fetch messages from Kafka brokers

The poll() method fetches messages from Kafka brokers for the consumer to process.

What happens if a Kafka consumer stops calling poll() for longer than the session timeout?

A Consumer automatically commits offsets

B Consumer creates new topics

C Consumer is removed from the group and partitions are reassigned

D Nothing happens

Kafka considers the consumer dead and triggers a rebalance to assign partitions to other consumers.

Which of these is a best practice inside a Kafka consumer poll loop?

A Keep the loop running continuously and call poll() regularly

B Call poll() once and then stop

C Perform long blocking operations before calling poll() again

D Close the consumer inside the loop

The consumer should call poll() regularly to maintain group membership and receive messages.

When should a Kafka consumer commit offsets in the poll loop?

A After processing the messages fetched by poll()

B Before calling poll()

C Only once when the consumer starts

D Offsets do not need to be committed

Committing offsets after processing ensures the consumer does not reprocess messages after restart.

What does a rebalance mean in the context of Kafka consumer poll loops?

A Consumer disconnects permanently

B Partitions are reassigned among consumers in the group

C Consumer creates new partitions

D Kafka deletes all messages

Rebalance redistributes partitions among consumers when group membership changes or a consumer fails.

Explain how the Kafka consumer poll loop works and why it is important.

Think about how the consumer stays connected and gets new data.

Describe what happens if a Kafka consumer does not call poll() often enough.

Consider the impact on the consumer group and message processing.

Practice

1. What is the main purpose of using a poll() method inside a Kafka consumer loop?

easy

A. To send messages to Kafka topics

B. To continuously fetch new messages from Kafka topics

C. To close the consumer connection

D. To create a new Kafka topic

Solution

1. Step 1: Understand the role of poll() in Kafka consumer

   The poll() method is used to fetch messages from Kafka topics.

2. Step 2: Recognize the loop usage

   Using poll() inside a loop allows the consumer to keep listening and fetching new messages continuously.

3. Final Answer:

   To continuously fetch new messages from Kafka topics -> Option B

4. Quick Check:

   poll() fetches messages repeatedly [OK]

Hint: poll() inside loop fetches messages repeatedly [OK]

Common Mistakes:

• Thinking poll() sends messages
• Confusing poll() with closing connection
• Assuming poll() creates topics

2. Which of the following is the correct syntax to start a Kafka consumer poll loop in Java?

easy

A. consumer.poll(Duration(100));

B. for(int i=0; i<10; i++) consumer.poll(100);

C. consumer.poll(100); while(true) {}

D. while(true) { consumer.poll(Duration.ofMillis(100)); }

Solution

1. Step 1: Check correct poll method usage

   The poll method requires a Duration argument like Duration.ofMillis(100) in Java.

2. Step 2: Verify loop structure

   A continuous loop like while(true) is used to keep polling messages repeatedly.

3. Final Answer:

   while(true) { consumer.poll(Duration.ofMillis(100)); } -> Option D

4. Quick Check:

   Correct poll syntax with continuous loop [OK]

Hint: Use while(true) with poll(Duration.ofMillis(x)) [OK]

Common Mistakes:

• Using poll with int instead of Duration
• Not using a loop for continuous polling
• Incorrect Duration constructor syntax

3. What will be the output of the following Kafka consumer poll loop snippet?

while (true) {
  var records = consumer.poll(Duration.ofMillis(100));
  System.out.println(records.count());
  break;
}

medium

A. Prints the number of records fetched in one poll and exits

B. Prints zero and continues infinitely

C. Throws an exception because of missing commit

D. Does not compile due to syntax error

Solution

1. Step 1: Analyze the poll call and record count

   The poll() fetches records and records.count() returns how many were fetched.

2. Step 2: Understand the loop and break

   The loop runs once because of break; after printing count, so it prints the count once and stops.

3. Final Answer:

   Prints the number of records fetched in one poll and exits -> Option A

4. Quick Check:

   poll() fetches records, break stops loop [OK]

Hint: Break after poll prints count once [OK]

Common Mistakes:

• Assuming infinite loop without break
• Thinking commit is mandatory to print count
• Confusing syntax causing compile error

4. Identify the error in this Kafka consumer poll loop code snippet:

while (true) {
  var records = consumer.poll(100);
  for (var record : records) {
    System.out.println(record.value());
  }
}

medium

A. poll() argument should be a Duration, not an int

B. for-each loop syntax is incorrect

C. Missing consumer.subscribe() call before loop

D. System.out.println cannot print record values

Solution

1. Step 1: Check poll method argument type

   The poll() method requires a Duration argument, not a plain integer.

2. Step 2: Confirm other code parts

   The for-each loop and print statement are correct; missing subscribe is not shown as error here.

3. Final Answer:

   poll() argument should be a Duration, not an int -> Option A

4. Quick Check:

   poll() needs Duration argument [OK]

Hint: poll() needs Duration, not int [OK]

Common Mistakes:

• Passing int instead of Duration to poll()
• Confusing for-each syntax
• Assuming subscribe() missing causes compile error

5. You want to process messages from multiple Kafka topics continuously and commit offsets manually after processing. Which code snippet correctly implements this using a consumer poll loop?

hard

A. consumer.poll(Duration.ofMillis(200));
consumer.commitSync();

B. while(true) {
var records = consumer.poll(200);
for(var record : records) {
process(record);
}
consumer.commitAsync();
}

C. while(true) {
var records = consumer.poll(Duration.ofMillis(200));
for(var record : records) {
process(record);
}
consumer.commitSync();
}

D. for(int i=0; i<10; i++) {
var records = consumer.poll(Duration.ofMillis(200));
consumer.commitSync();
}

Solution

1. Step 1: Check poll argument and loop structure

   while(true) { var records = consumer.poll(Duration.ofMillis(200)); for(var record : records) { process(record); } consumer.commitSync(); } uses poll(Duration.ofMillis(200)) inside a continuous while(true) loop, which is correct.

2. Step 2: Verify message processing and manual commit

   It processes each record and calls commitSync() after processing, ensuring offsets are committed manually.

3. Final Answer:

   while(true) { var records = consumer.poll(Duration.ofMillis(200)); for(var record : records) { process(record); } consumer.commitSync(); } -> Option C

4. Quick Check:

   Continuous poll, process, then manual commit [OK]

Hint: Use while(true), poll with Duration, then commitSync() [OK]

Common Mistakes:

• Passing int instead of Duration to poll()
• Using for loop instead of continuous loop
• Not processing records before commit
• Missing loop causing single poll only