Consumer offset commit strategies in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Kafka consumer offset commit code?

Consider a Kafka consumer configured with enable.auto.commit=false. After processing a batch of messages, the consumer commits offsets synchronously using commitSync(). What will be the output of the following code snippet?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test-group");
props.put("enable.auto.commit", "false");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("test-topic"));

try {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        System.out.println("Received: " + record.value());
    }
    consumer.commitSync();
    System.out.println("Offsets committed synchronously.");
} finally {
    consumer.close();
}

A Prints all received messages but throws an exception at commitSync()

B Prints all received messages and then prints 'Offsets committed synchronously.'

C Prints nothing and immediately prints 'Offsets committed synchronously.'

D Prints all received messages and commits offsets asynchronously without confirmation

💡 Hint

Think about what commitSync() does and how enable.auto.commit=false affects offset commits.

With enable.auto.commit=false, offsets are not committed automatically. The commitSync() method commits offsets synchronously, so after processing messages, the offsets are committed and the confirmation message is printed.

❓ Predict Output

intermediate

What happens when using commitAsync() in Kafka consumer?

Given a Kafka consumer with enable.auto.commit=false, what is the expected behavior of the following code snippet that uses commitAsync()?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test-group");
props.put("enable.auto.commit", "false");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("test-topic"));

try {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        System.out.println("Received: " + record.value());
    }
    consumer.commitAsync();
    System.out.println("Offsets committed asynchronously.");
} finally {
    consumer.close();
}

A Prints all received messages and then prints 'Offsets committed asynchronously.' immediately without waiting for commit confirmation

B Prints all received messages and waits for commit confirmation before printing the message

C Throws a runtime exception because commitAsync() requires a callback

D Does not commit offsets and prints only received messages

💡 Hint

Remember that commitAsync() commits offsets in the background and does not block the thread.

The commitAsync() method commits offsets asynchronously, so the program continues immediately after calling it and prints the confirmation message without waiting for the commit to complete.

🧠 Conceptual

advanced

Which commit strategy ensures exactly-once processing in Kafka consumers?

In Kafka consumer offset commit strategies, which approach best supports exactly-once processing semantics?

A Manually committing offsets synchronously after processing each record

B Using <code>enable.auto.commit=true</code> with default auto commit interval

C Committing offsets asynchronously without handling commit failures

D Not committing offsets at all and relying on consumer restart

💡 Hint

Think about when offsets should be committed to avoid message loss or duplication.

Manually committing offsets synchronously after processing each record ensures that offsets are only committed after successful processing, supporting exactly-once semantics.

🔧 Debug

advanced

Identify the error in this Kafka consumer offset commit code

What error will this Kafka consumer code produce when trying to commit offsets?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test-group");
props.put("enable.auto.commit", "true");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("test-topic"));

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
    System.out.println("Received: " + record.value());
}
consumer.commitSync();

A Throws CommitFailedException because commitSync() is called with auto commit enabled

B Throws IllegalStateException because offsets are auto-committed and manual commitSync() is called

C No error; offsets are committed synchronously successfully

D Throws IllegalArgumentException due to missing commit callback

💡 Hint

Check if calling commitSync() is allowed when enable.auto.commit=true.

Even if enable.auto.commit=true, calling commitSync() is allowed and will commit offsets synchronously. No error is thrown.

🚀 Application

expert

How many offsets are committed after this batch processing with manual commit?

A Kafka consumer processes messages from partitions 0 and 1 of a topic. It processes 3 messages from partition 0 with offsets 5, 6, 7 and 2 messages from partition 1 with offsets 10, 11. After processing, it commits offsets manually using commitSync() with the following map:

{new TopicPartition("topic", 0): new OffsetAndMetadata(8), new TopicPartition("topic", 1): new OffsetAndMetadata(12)}

How many offsets are committed in total?

A 3 offsets committed, only for partition 0 messages

B 5 offsets committed, one for each message processed

C 0 offsets committed because offsets are off by one

D 2 offsets committed, one for each partition

💡 Hint

Remember that Kafka commits the offset of the next message to read, not the last processed message.

The commit map contains one offset per partition indicating the next offset to read. So two offsets are committed, one for each partition.

Practice

1. What is the main purpose of committing consumer offsets in Kafka?

easy

A. To delete old messages from the topic

B. To increase the speed of message production

C. To create new topics automatically

D. To track which messages have been processed to avoid duplicates

Solution

1. Step 1: Understand consumer offset role

   Consumer offsets mark the position of the last processed message in a partition.

2. Step 2: Identify the purpose of committing offsets

   Committing offsets helps Kafka know where to resume reading after a restart, preventing duplicate processing.

3. Final Answer:

   To track which messages have been processed to avoid duplicates -> Option D

4. Quick Check:

   Offset commit = track processed messages [OK]

Hint: Offsets mark progress to avoid reprocessing messages [OK]

Common Mistakes:

• Confusing offset commit with message production speed
• Thinking offsets delete messages
• Believing offsets create topics

2. Which Kafka consumer configuration enables automatic offset commits?

easy

A. enable.auto.commit=false

B. auto.offset.reset=earliest

C. enable.auto.commit=true

D. auto.commit.interval.ms=0

Solution

1. Step 1: Identify the setting for automatic commits

   The setting enable.auto.commit controls if Kafka commits offsets automatically.

2. Step 2: Check the correct value to enable auto commit

   Setting it to true enables automatic offset commits.

3. Final Answer:

   enable.auto.commit=true -> Option C

4. Quick Check:

   enable.auto.commit=true means auto commit enabled [OK]

Hint: Look for enable.auto.commit=true to enable auto commits [OK]

Common Mistakes:

• Confusing auto.offset.reset with auto commit
• Setting enable.auto.commit to false expecting auto commit
• Misunderstanding auto.commit.interval.ms controls commit frequency, not enabling

3. Given this code snippet, what will be the output if the consumer commits offsets manually after processing 3 messages?

consumer.subscribe(["topic"])
for i in range(5):
    msg = consumer.poll(1000)
    print(f"Processing message {i}")
    if i == 2:
        consumer.commitSync()

medium

A. Processing message 0
Processing message 1
Processing message 2
Processing message 3
Processing message 4

B. Processing message 0
Processing message 1
Processing message 2

C. Processing message 0
Processing message 1
Processing message 2
Processing message 3

D. No output due to commit error

Solution

1. Step 1: Analyze the loop and commit timing

   The loop runs 5 times, printing "Processing message i" each time from 0 to 4.

2. Step 2: Understand commitSync effect

   Calling commitSync() at i=2 commits offsets but does not stop processing.

3. Final Answer:

   Processing message 0 Processing message 1 Processing message 2 Processing message 3 Processing message 4 -> Option A

4. Quick Check:

   commitSync commits but loop continues [OK]

Hint: commitSync doesn't stop processing; loop runs fully [OK]

Common Mistakes:

• Assuming commitSync stops the loop
• Thinking only messages before commit are processed
• Confusing commitSync with commitAsync behavior

4. Identify the error in this Kafka consumer code snippet related to offset commits:

consumer.subscribe(["topic"])
while True:
    msg = consumer.poll(1000)
    if msg is not None:
        process(msg)
    consumer.commitAsync()

medium

A. poll() should not have a timeout

B. commitAsync() is called outside the message check, causing commits even when no message is received

C. process(msg) should be called after commitAsync()

D. subscribe() must be called inside the loop

Solution

1. Step 1: Review commitAsync placement

   commitAsync() is called every loop iteration, even if no message is received.

2. Step 2: Understand why this is a problem

   Committing offsets without processing messages can commit the same offset repeatedly or cause unexpected behavior.

3. Final Answer:

   commitAsync() is called outside the message check, causing commits even when no message is received -> Option B

4. Quick Check:

   commitAsync outside message check causes wrong commits [OK]

Hint: Call commit only after processing a message [OK]

Common Mistakes:

• Calling commitAsync regardless of message presence
• Moving subscribe inside the loop unnecessarily
• Misordering process and commit calls

5. You want to ensure your Kafka consumer starts reading from the earliest offset if no committed offset exists, but commits offsets manually only after processing batches of 10 messages. Which configuration and strategy combination is correct?

hard

A. Set auto.offset.reset to earliest and disable enable.auto.commit, then call commitSync() after every 10 messages

B. Set auto.offset.reset to latest and enable enable.auto.commit, commit automatically every 10 messages

C. Set auto.offset.reset to earliest and enable enable.auto.commit, commit manually after every message

D. Set auto.offset.reset to none and disable enable.auto.commit, never commit offsets

Solution

1. Step 1: Choose offset reset policy

   Setting auto.offset.reset to earliest ensures reading from the start if no offset exists.

2. Step 2: Select commit strategy

   Disabling enable.auto.commit allows manual control; calling commitSync() after 10 messages commits offsets in batches.

3. Final Answer:

   Set auto.offset.reset to earliest and disable enable.auto.commit, then call commitSync() after every 10 messages -> Option A

4. Quick Check:

   earliest + manual commit after batch = correct strategy [OK]

Hint: earliest + manual commit after batch = safe and controlled [OK]

Common Mistakes:

• Using latest offset reset when earliest is needed
• Enabling auto commit but expecting manual control
• Not committing offsets causing reprocessing