Auto-commit vs manual commit in Kafka - Visual Side-by-Side Comparison

Process Flow - Auto-commit vs manual commit

Start consuming messages

↓

Auto-commit enabled?

Yes→Kafka commits offsets automatically

↓

Offsets updated in background

↓

Manual commit required

↓

Process message

↓

Commit offset manually

↓

Continue consuming

This flow shows how Kafka consumers handle offset commits automatically or manually after processing messages.

Execution Sample

Kafka

consumer = KafkaConsumer('topic', enable_auto_commit=True)
for message in consumer:
    print(message.value)

# vs manual commit
consumer = KafkaConsumer('topic', enable_auto_commit=False)
for message in consumer:
    process(message)
    consumer.commit()

This code compares Kafka consumer behavior with auto-commit enabled versus manual commit after processing each message.

Process Table

Step	Auto-commit Enabled	Message Received	Offset Commit Action	Commit Timing
1	True	Message 1	Kafka commits offset automatically	After poll interval
2	True	Message 2	Kafka commits offset automatically	After poll interval
3	False	Message 1	No commit yet	Manual commit needed
4	False	Message 1 processed	Commit called manually	Immediately after processing
5	False	Message 2	No commit yet	Manual commit needed
6	False	Message 2 processed	Commit called manually	Immediately after processing
7	End	-	-	Consumer stops or continues

💡 Execution stops when consumer is closed or no more messages.

Status Tracker

Variable	Start	After Step 1	After Step 3	After Step 4	After Step 5	After Step 6	Final
offset_committed	0	1 (auto)	0 (manual)	1 (manual)	1 (manual)	2 (manual)	2 (manual)

Key Moments - 2 Insights

Why does offset_committed stay the same after receiving a message with manual commit?

Because with manual commit (see steps 3 and 5), Kafka does not update the offset until commit() is called (steps 4 and 6).

When does Kafka commit offsets automatically in auto-commit mode?

Kafka commits offsets automatically after the poll interval, not immediately after each message (see steps 1 and 2).

Visual Quiz - 3 Questions

Look at the execution table, what is the offset_committed after step 2 with auto-commit enabled?

A 0

B 1

C 2

D Not committed yet

💡 Hint

Check the 'Offset Commit Action' and 'Commit Timing' columns at step 2.

At which step does manual commit actually update the offset for Message 1?

A Step 4

B Step 3

C Step 5

D Step 2

💡 Hint

Look at when 'Commit called manually' happens for Message 1 in the table.

If auto-commit is disabled, when must the consumer commit offsets to avoid reprocessing messages?

A Before processing messages

B Offsets commit automatically anyway

C After processing each message manually

D Offsets never need to be committed

💡 Hint

Refer to the 'Commit Timing' column for manual commit steps.

Concept Snapshot

Kafka consumers track message offsets to know what was read.
Auto-commit mode commits offsets automatically after polling.
Manual commit requires calling commit() after processing.
Auto-commit is simpler but less precise.
Manual commit gives control to avoid losing or reprocessing messages.

Full Transcript

This visual execution compares Kafka consumer offset commits in auto-commit and manual commit modes. When auto-commit is enabled, Kafka commits offsets automatically after polling messages, shown in steps 1 and 2. With auto-commit disabled, offsets are not committed until the consumer calls commit() after processing each message, shown in steps 3 to 6. The variable offset_committed tracks the last committed offset, updating immediately after commit() in manual mode, but only after poll intervals in auto-commit mode. This helps avoid reprocessing or losing messages depending on commit strategy.

Practice

1. What is the main difference between auto-commit and manual commit in Kafka consumer offset management?

easy

A. Auto-commit requires manual calls to commit offsets, manual commit commits automatically.

B. Auto-commit disables offset tracking, manual commit enables it.

C. Auto-commit commits offsets automatically at intervals, manual commit requires explicit commit calls.

D. Auto-commit commits offsets only after consumer shutdown, manual commit commits continuously.

Solution

1. Step 1: Understand auto-commit behavior

   Auto-commit commits offsets automatically at regular intervals without user intervention.

2. Step 2: Understand manual commit behavior

   Manual commit requires the consumer to explicitly call commit() after processing messages.

3. Final Answer:

   Auto-commit commits offsets automatically at intervals, manual commit requires explicit commit calls. -> Option C

4. Quick Check:

   Auto-commit = automatic, Manual commit = explicit call [OK]

Hint: Auto-commit is automatic; manual commit needs your call [OK]

Common Mistakes:

• Confusing which method commits automatically
• Thinking manual commit disables offset tracking
• Believing auto-commit commits only on shutdown

2. Which of the following is the correct way to disable auto-commit in Kafka consumer configuration?

easy

A. "enable.auto.commit" : false

B. "auto.commit.enable" : false

C. "auto.commit" : "disable"

D. "enable.auto.commit" : true

Solution

1. Step 1: Identify the correct config key

   The correct Kafka consumer config key to control auto-commit is "enable.auto.commit".

2. Step 2: Identify the correct value to disable auto-commit

   Setting "enable.auto.commit" to false disables auto-commit.

3. Final Answer:

   "enable.auto.commit" : false -> Option A

4. Quick Check:

   Disable auto-commit = enable.auto.commit = false [OK]

Hint: Use "enable.auto.commit" = false to disable auto-commit [OK]

Common Mistakes:

• Using wrong config key names
• Setting true instead of false to disable
• Using string values instead of boolean

3. Given this Kafka consumer code snippet with manual commit:

consumer.poll(Duration.ofMillis(100)).forEach(record -> {
    process(record);
});
consumer.commitSync();

What happens if commitSync() is called before processing the records?

medium

A. Offsets are committed before processing, risking message loss on failure.

B. Offsets are committed after processing, ensuring safe processing.

C. commitSync() throws an exception if called before processing.

D. Offsets are not committed at all.

Solution

1. Step 1: Understand commitSync timing

   commitSync() commits the current offsets to Kafka immediately when called.

2. Step 2: Analyze calling commitSync before processing

   If commitSync() is called before processing, offsets are saved before messages are handled, risking data loss if processing fails.

3. Final Answer:

   Offsets are committed before processing, risking message loss on failure. -> Option A

4. Quick Check:

   Commit before processing = risk of message loss [OK]

Hint: Commit offsets only after processing messages [OK]

Common Mistakes:

• Assuming commitSync waits for processing
• Thinking commitSync throws error if called early
• Believing offsets commit automatically without call

4. You have disabled auto-commit and use manual commit in your Kafka consumer. You notice some offsets are not committed, causing message reprocessing. What is the likely cause?

medium

A. Kafka broker is down, so commits fail silently.

B. Auto-commit is enabled, causing conflicts.

C. You called commit() before processing messages.

D. You forgot to call commit() after processing messages.

Solution

1. Step 1: Check manual commit requirements

   When auto-commit is disabled, the consumer must call commit() explicitly after processing.

2. Step 2: Identify cause of missing commits

   If offsets are not committed, likely commit() was not called after processing, causing reprocessing.

3. Final Answer:

   You forgot to call commit() after processing messages. -> Option D

4. Quick Check:

   Manual commit needs explicit commit() call [OK]

Hint: Always call commit() after processing when auto-commit is off [OK]

Common Mistakes:

• Assuming auto-commit still works when disabled
• Blaming Kafka broker without checking code
• Calling commit() before processing messages

5. You want to ensure exactly-once processing in a Kafka consumer application. Which commit strategy is safer and why?

hard

A. Auto-commit, because it commits offsets automatically without extra code.

B. Manual commit, because you commit offsets only after successful processing.

C. Auto-commit, because it commits offsets after processing each message.

D. Manual commit, because it commits offsets before processing messages.

Solution

1. Step 1: Understand exactly-once processing needs

   Exactly-once means processing each message once and committing offset only after success.

2. Step 2: Compare commit strategies

   Auto-commit commits offsets automatically at intervals, risking committing before processing completes. Manual commit lets you commit only after successful processing.

3. Final Answer:

   Manual commit, because you commit offsets only after successful processing. -> Option B

4. Quick Check:

   Exactly-once needs manual commit after success [OK]

Hint: Commit manually after success for exactly-once processing [OK]

Common Mistakes:

• Assuming auto-commit guarantees exactly-once
• Committing offsets before processing
• Ignoring commit timing importance