Auto-commit vs manual commit in Kafka - Practice Questions

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output when auto-commit is enabled?

Consider a Kafka consumer configured with enable.auto.commit=true and auto.commit.interval.ms=1000. The consumer reads messages continuously. What happens if the consumer crashes after processing some messages but before the next auto-commit?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test-group");
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("my-topic"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        System.out.printf("offset = %d, key = %s, value = %s\n", record.offset(), record.key(), record.value());
        // Process message
    }
    // No manual commit here
}

A The consumer will throw an exception if it crashes before committing offsets.

B Offsets are committed manually by the user after processing each message.

C All processed messages are guaranteed committed immediately after processing each message.

D Messages processed before the crash may be reprocessed after restart because the offset was not committed yet.

💡 Hint

Think about how auto-commit interval affects when offsets are saved.

With auto-commit enabled, offsets are committed periodically (every 1000 ms here). If the consumer crashes before the next commit, some processed messages may not have their offsets saved, causing reprocessing after restart.

❓ Predict Output

intermediate

What happens when manual commit is used incorrectly?

A Kafka consumer is configured with enable.auto.commit=false. The code processes messages but never calls commitSync() or commitAsync(). What is the effect on message processing?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test-group");
props.put("enable.auto.commit", "false");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("my-topic"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        System.out.printf("offset = %d, key = %s, value = %s\n", record.offset(), record.key(), record.value());
        // Process message
    }
    // No commit called here
}

A Messages will be reprocessed after restart because offsets are never committed.

B Offsets are committed asynchronously by default even if auto-commit is false.

C The consumer will throw a runtime exception due to missing commit calls.

D Messages will be processed exactly once because offsets are committed automatically.

💡 Hint

Consider what happens if offsets are never saved.

When auto-commit is disabled and no manual commit is done, offsets are never saved. After restart, the consumer will re-read messages from the last committed offset, causing duplicates.

🧠 Conceptual

advanced

Why choose manual commit over auto-commit in Kafka consumers?

Which of the following is the best reason to use manual commit instead of auto-commit in Kafka consumers?

A To ensure offsets are committed only after successful processing of messages, avoiding data loss or duplication.

B To reduce network traffic by committing offsets less frequently.

C Because auto-commit is deprecated and no longer supported in Kafka.

D To automatically retry failed message processing without manual intervention.

💡 Hint

Think about reliability and message processing guarantees.

Manual commit allows the consumer to commit offsets only after messages are fully processed, which helps avoid data loss or duplicate processing. Auto-commit commits offsets periodically regardless of processing success.

❓ Predict Output

advanced

What error occurs if commitSync() is called outside poll loop?

Given a Kafka consumer with enable.auto.commit=false, what happens if commitSync() is called before any poll() call?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test-group");
props.put("enable.auto.commit", "false");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

// Calling commitSync before poll
consumer.commitSync();

A Throws TimeoutException because commitSync waits for poll to assign partitions.

B No error; commitSync commits the last committed offsets successfully.

C Throws IllegalStateException because no offsets have been fetched yet.

D Throws NullPointerException due to missing consumer records.

💡 Hint

Consider what commitSync needs before it can commit offsets.

commitSync requires the consumer to have assigned partitions and fetched offsets via poll. Calling it before poll causes IllegalStateException because no offsets are known yet.

🚀 Application

expert

How to implement exactly-once processing with manual commit in Kafka consumer?

You want to process Kafka messages exactly once with manual commit. Which approach below correctly ensures that offsets are committed only after successful processing?

A Process messages, then call <code>commitSync()</code> immediately after processing each message inside the for-loop.

B Process all messages in the batch, then call <code>commitSync()</code> once after the for-loop completes successfully.

C Call <code>commitSync()</code> before processing messages to reserve offsets.

D Enable auto-commit and rely on Kafka to commit offsets after processing.

💡 Hint

Think about committing offsets only after all messages are processed successfully.

Committing offsets once after processing the entire batch ensures that no message is marked as processed before it actually is. Committing inside the loop risks committing offsets for unprocessed messages if a failure occurs.

Practice

1. What is the main difference between auto-commit and manual commit in Kafka consumer offset management?

easy

A. Auto-commit requires manual calls to commit offsets, manual commit commits automatically.

B. Auto-commit disables offset tracking, manual commit enables it.

C. Auto-commit commits offsets automatically at intervals, manual commit requires explicit commit calls.

D. Auto-commit commits offsets only after consumer shutdown, manual commit commits continuously.

Solution

1. Step 1: Understand auto-commit behavior

   Auto-commit commits offsets automatically at regular intervals without user intervention.

2. Step 2: Understand manual commit behavior

   Manual commit requires the consumer to explicitly call commit() after processing messages.

3. Final Answer:

   Auto-commit commits offsets automatically at intervals, manual commit requires explicit commit calls. -> Option C

4. Quick Check:

   Auto-commit = automatic, Manual commit = explicit call [OK]

Hint: Auto-commit is automatic; manual commit needs your call [OK]

Common Mistakes:

• Confusing which method commits automatically
• Thinking manual commit disables offset tracking
• Believing auto-commit commits only on shutdown

2. Which of the following is the correct way to disable auto-commit in Kafka consumer configuration?

easy

A. "enable.auto.commit" : false

B. "auto.commit.enable" : false

C. "auto.commit" : "disable"

D. "enable.auto.commit" : true

Solution

1. Step 1: Identify the correct config key

   The correct Kafka consumer config key to control auto-commit is "enable.auto.commit".

2. Step 2: Identify the correct value to disable auto-commit

   Setting "enable.auto.commit" to false disables auto-commit.

3. Final Answer:

   "enable.auto.commit" : false -> Option A

4. Quick Check:

   Disable auto-commit = enable.auto.commit = false [OK]

Hint: Use "enable.auto.commit" = false to disable auto-commit [OK]

Common Mistakes:

• Using wrong config key names
• Setting true instead of false to disable
• Using string values instead of boolean

3. Given this Kafka consumer code snippet with manual commit:

consumer.poll(Duration.ofMillis(100)).forEach(record -> {
    process(record);
});
consumer.commitSync();

What happens if commitSync() is called before processing the records?

medium

A. Offsets are committed before processing, risking message loss on failure.

B. Offsets are committed after processing, ensuring safe processing.

C. commitSync() throws an exception if called before processing.

D. Offsets are not committed at all.

Solution

1. Step 1: Understand commitSync timing

   commitSync() commits the current offsets to Kafka immediately when called.

2. Step 2: Analyze calling commitSync before processing

   If commitSync() is called before processing, offsets are saved before messages are handled, risking data loss if processing fails.

3. Final Answer:

   Offsets are committed before processing, risking message loss on failure. -> Option A

4. Quick Check:

   Commit before processing = risk of message loss [OK]

Hint: Commit offsets only after processing messages [OK]

Common Mistakes:

• Assuming commitSync waits for processing
• Thinking commitSync throws error if called early
• Believing offsets commit automatically without call

4. You have disabled auto-commit and use manual commit in your Kafka consumer. You notice some offsets are not committed, causing message reprocessing. What is the likely cause?

medium

A. Kafka broker is down, so commits fail silently.

B. Auto-commit is enabled, causing conflicts.

C. You called commit() before processing messages.

D. You forgot to call commit() after processing messages.

Solution

1. Step 1: Check manual commit requirements

   When auto-commit is disabled, the consumer must call commit() explicitly after processing.

2. Step 2: Identify cause of missing commits

   If offsets are not committed, likely commit() was not called after processing, causing reprocessing.

3. Final Answer:

   You forgot to call commit() after processing messages. -> Option D

4. Quick Check:

   Manual commit needs explicit commit() call [OK]

Hint: Always call commit() after processing when auto-commit is off [OK]

Common Mistakes:

• Assuming auto-commit still works when disabled
• Blaming Kafka broker without checking code
• Calling commit() before processing messages

5. You want to ensure exactly-once processing in a Kafka consumer application. Which commit strategy is safer and why?

hard

A. Auto-commit, because it commits offsets automatically without extra code.

B. Manual commit, because you commit offsets only after successful processing.

C. Auto-commit, because it commits offsets after processing each message.

D. Manual commit, because it commits offsets before processing messages.

Solution

1. Step 1: Understand exactly-once processing needs

   Exactly-once means processing each message once and committing offset only after success.

2. Step 2: Compare commit strategies

   Auto-commit commits offsets automatically at intervals, risking committing before processing completes. Manual commit lets you commit only after successful processing.

3. Final Answer:

   Manual commit, because you commit offsets only after successful processing. -> Option B

4. Quick Check:

   Exactly-once needs manual commit after success [OK]

Hint: Commit manually after success for exactly-once processing [OK]

Common Mistakes:

• Assuming auto-commit guarantees exactly-once
• Committing offsets before processing
• Ignoring commit timing importance