Consumer offset commit strategies in Kafka - Step-by-Step Execution

Process Flow - Consumer offset commit strategies

Start consuming messages

↓

Process message

↓

Decide commit strategy

↓

Auto commit

↓

Commit offset

↓

Continue consuming

This flow shows how a Kafka consumer processes messages and commits offsets using different strategies: auto, synchronous, or asynchronous commits.

Execution Sample

Kafka

consumer.subscribe(['topic'])
for message in consumer:
    process(message)
    consumer.commit()  # sync commit
    # or consumer.commit_async()  # async commit

A Kafka consumer subscribes to a topic, processes each message, and commits the offset synchronously or asynchronously.

Process Table

Step	Message Offset	Processing	Commit Strategy	Commit Action	Next Offset
1	0	Processed message 0	Auto commit disabled	No commit yet	1
2	1	Processed message 1	Sync commit	Commit offset 1 synchronously	2
3	2	Processed message 2	Async commit	Commit offset 2 asynchronously	3
4	3	Processed message 3	Sync commit	Commit offset 3 synchronously	4
5	-	No more messages	-	Stop consuming	-

💡 No more messages to consume, consumer stops.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
current_offset	None	0	1	2	3	3
commit_status	None	No commit	Committed sync	Committed async	Committed sync	Committed sync

Key Moments - 2 Insights

Why does the offset sometimes commit synchronously and sometimes asynchronously?

The commit strategy depends on the method called: commit() blocks until Kafka confirms, while commit_async() sends the commit request without waiting. See execution_table rows 2 and 3.

What happens if auto commit is disabled?

Offsets are not committed automatically after processing messages. The consumer must call commit() or commit_async() manually to save progress, as shown in step 1 where no commit occurs.

Visual Quiz - 3 Questions

Look at the execution_table, what is the commit action at step 3?

A Commit offset 2 synchronously

B Commit offset 2 asynchronously

C No commit performed

D Commit offset 1 asynchronously

💡 Hint

Check the 'Commit Action' column in execution_table row 3.

At which step does the consumer commit offset 1 synchronously?

A Step 2

B Step 1

C Step 3

D Step 4

💡 Hint

Look at the 'Commit Strategy' and 'Commit Action' columns in execution_table.

If auto commit was enabled, how would the commit actions change in the table?

A Only async commits would be used

B No commits would happen at all

C Commits would happen automatically after each message without explicit calls

D Only sync commits would be used

💡 Hint

Auto commit means Kafka commits offsets automatically after processing messages.

Concept Snapshot

Kafka consumers track message offsets to know what was processed.
Offsets can be committed automatically or manually.
Manual commits can be synchronous (blocking) or asynchronous (non-blocking).
Choosing the right commit strategy balances reliability and performance.
Auto commit is easy but less precise; manual commit gives control.

Full Transcript

This visual execution shows how Kafka consumers handle offset commits. The consumer reads messages one by one, processes them, and commits offsets to Kafka to mark progress. There are three main commit strategies: auto commit, synchronous commit, and asynchronous commit. Auto commit happens automatically at intervals, but here it is disabled, so commits are manual. Synchronous commit waits for Kafka to confirm the offset is saved before continuing, ensuring reliability but slowing down processing. Asynchronous commit sends the commit request without waiting, improving speed but risking lost commits if the consumer crashes. The execution table traces each step: processing a message, choosing commit strategy, committing offset, and moving to the next message. The variable tracker shows how the current offset and commit status change over time. Key moments clarify why commit methods differ and what happens if auto commit is disabled. The quiz tests understanding of commit actions at specific steps and the effect of enabling auto commit. This helps beginners see how offset commits work in practice and why choosing the right strategy matters.

Practice

1. What is the main purpose of committing consumer offsets in Kafka?

easy

A. To delete old messages from the topic

B. To increase the speed of message production

C. To create new topics automatically

D. To track which messages have been processed to avoid duplicates

Solution

1. Step 1: Understand consumer offset role

   Consumer offsets mark the position of the last processed message in a partition.

2. Step 2: Identify the purpose of committing offsets

   Committing offsets helps Kafka know where to resume reading after a restart, preventing duplicate processing.

3. Final Answer:

   To track which messages have been processed to avoid duplicates -> Option D

4. Quick Check:

   Offset commit = track processed messages [OK]

Hint: Offsets mark progress to avoid reprocessing messages [OK]

Common Mistakes:

• Confusing offset commit with message production speed
• Thinking offsets delete messages
• Believing offsets create topics

2. Which Kafka consumer configuration enables automatic offset commits?

easy

A. enable.auto.commit=false

B. auto.offset.reset=earliest

C. enable.auto.commit=true

D. auto.commit.interval.ms=0

Solution

1. Step 1: Identify the setting for automatic commits

   The setting enable.auto.commit controls if Kafka commits offsets automatically.

2. Step 2: Check the correct value to enable auto commit

   Setting it to true enables automatic offset commits.

3. Final Answer:

   enable.auto.commit=true -> Option C

4. Quick Check:

   enable.auto.commit=true means auto commit enabled [OK]

Hint: Look for enable.auto.commit=true to enable auto commits [OK]

Common Mistakes:

• Confusing auto.offset.reset with auto commit
• Setting enable.auto.commit to false expecting auto commit
• Misunderstanding auto.commit.interval.ms controls commit frequency, not enabling

3. Given this code snippet, what will be the output if the consumer commits offsets manually after processing 3 messages?

consumer.subscribe(["topic"])
for i in range(5):
    msg = consumer.poll(1000)
    print(f"Processing message {i}")
    if i == 2:
        consumer.commitSync()

medium

A. Processing message 0
Processing message 1
Processing message 2
Processing message 3
Processing message 4

B. Processing message 0
Processing message 1
Processing message 2

C. Processing message 0
Processing message 1
Processing message 2
Processing message 3

D. No output due to commit error

Solution

1. Step 1: Analyze the loop and commit timing

   The loop runs 5 times, printing "Processing message i" each time from 0 to 4.

2. Step 2: Understand commitSync effect

   Calling commitSync() at i=2 commits offsets but does not stop processing.

3. Final Answer:

   Processing message 0 Processing message 1 Processing message 2 Processing message 3 Processing message 4 -> Option A

4. Quick Check:

   commitSync commits but loop continues [OK]

Hint: commitSync doesn't stop processing; loop runs fully [OK]

Common Mistakes:

• Assuming commitSync stops the loop
• Thinking only messages before commit are processed
• Confusing commitSync with commitAsync behavior

4. Identify the error in this Kafka consumer code snippet related to offset commits:

consumer.subscribe(["topic"])
while True:
    msg = consumer.poll(1000)
    if msg is not None:
        process(msg)
    consumer.commitAsync()

medium

A. poll() should not have a timeout

B. commitAsync() is called outside the message check, causing commits even when no message is received

C. process(msg) should be called after commitAsync()

D. subscribe() must be called inside the loop

Solution

1. Step 1: Review commitAsync placement

   commitAsync() is called every loop iteration, even if no message is received.

2. Step 2: Understand why this is a problem

   Committing offsets without processing messages can commit the same offset repeatedly or cause unexpected behavior.

3. Final Answer:

   commitAsync() is called outside the message check, causing commits even when no message is received -> Option B

4. Quick Check:

   commitAsync outside message check causes wrong commits [OK]

Hint: Call commit only after processing a message [OK]

Common Mistakes:

• Calling commitAsync regardless of message presence
• Moving subscribe inside the loop unnecessarily
• Misordering process and commit calls

5. You want to ensure your Kafka consumer starts reading from the earliest offset if no committed offset exists, but commits offsets manually only after processing batches of 10 messages. Which configuration and strategy combination is correct?

hard

A. Set auto.offset.reset to earliest and disable enable.auto.commit, then call commitSync() after every 10 messages

B. Set auto.offset.reset to latest and enable enable.auto.commit, commit automatically every 10 messages

C. Set auto.offset.reset to earliest and enable enable.auto.commit, commit manually after every message

D. Set auto.offset.reset to none and disable enable.auto.commit, never commit offsets

Solution

1. Step 1: Choose offset reset policy

   Setting auto.offset.reset to earliest ensures reading from the start if no offset exists.

2. Step 2: Select commit strategy

   Disabling enable.auto.commit allows manual control; calling commitSync() after 10 messages commits offsets in batches.

3. Final Answer:

   Set auto.offset.reset to earliest and disable enable.auto.commit, then call commitSync() after every 10 messages -> Option A

4. Quick Check:

   earliest + manual commit after batch = correct strategy [OK]

Hint: earliest + manual commit after batch = safe and controlled [OK]

Common Mistakes:

• Using latest offset reset when earliest is needed
• Enabling auto commit but expecting manual control
• Not committing offsets causing reprocessing