Offset management in Kafka - Step-by-Step Execution

Process Flow - Offset management

Consumer reads message

↓

Process message

↓

Commit offset

↓

Next message read using committed offset

↓

Repeat

This flow shows how a Kafka consumer reads messages, processes them, commits the offset, and then reads the next message starting from the committed offset.

Execution Sample

Kafka

consumer.subscribe(['topic'])
for message in consumer:
    process(message)
    consumer.commit()

This code subscribes to a Kafka topic, processes each message, and commits the offset after processing.

Process Table

Step	Action	Offset Read	Message Processed	Offset Committed	Next Offset to Read
1	Start consuming	0	No	No	0
2	Read message	0	Message at offset 0	No	0
3	Process message	0	Message at offset 0	No	0
4	Commit offset	0	Message at offset 0	1	1
5	Read message	1	Message at offset 1	1	1
6	Process message	1	Message at offset 1	1	1
7	Commit offset	1	Message at offset 1	2	2
8	Read message	2	Message at offset 2	2	2
9	Process message	2	Message at offset 2	2	2
10	Commit offset	2	Message at offset 2	3	3
11	Stop consuming	3	No	3	3

💡 Consumer stops after processing and committing offset 2; next offset 3 is not read.

Status Tracker

Variable	Start	After Step 4	After Step 7	After Step 10	Final
Offset Read	0	1	2	3	3
Offset Committed	No	1	2	3	3
Next Offset to Read	0	1	2	3	3

Key Moments - 3 Insights

Why does the consumer commit the offset after processing the message?

Committing the offset after processing ensures that if the consumer restarts, it will resume from the next unprocessed message, avoiding duplicates or data loss. See steps 3 and 4 in the execution_table.

What happens if the consumer reads a message but does not commit the offset?

If the offset is not committed, on restart the consumer will re-read the same message, causing duplicate processing. This is shown between steps 5 and 6 where offset committed is still 1.

Why is the 'Next Offset to Read' always one more than the 'Offset Committed'?

Because Kafka consumers read messages starting from the last committed offset plus one, ensuring continuous processing without overlap. See the progression in the variable_tracker.

Visual Quiz - 3 Questions

Look at the execution_table, what is the 'Offset Committed' after step 7?

A 0

B 2

C 1

D 3

💡 Hint

Check the 'Offset Committed' column at step 7 in the execution_table.

At which step does the consumer first process the message at offset 2?

A Step 8

B Step 10

C Step 9

D Step 11

💡 Hint

Look at the 'Message Processed' column for offset 2 in the execution_table.

If the consumer never commits offsets, what will happen when it restarts?

A It will re-read messages from the beginning or last committed offset

B It will skip messages already processed

C It will start reading from the last committed offset

D It will stop consuming messages

💡 Hint

Refer to the key_moments explanation about committing offsets and duplicate processing.

Concept Snapshot

Kafka offset management:
- Consumer reads messages from a topic partition.
- After processing, consumer commits the offset.
- Committed offset marks the last processed message.
- On restart, consumer resumes from committed offset + 1.
- Proper offset commit avoids duplicates or data loss.

Full Transcript

This visual execution shows how Kafka consumers manage offsets. The consumer reads a message at a given offset, processes it, then commits that offset. Committing means the consumer tells Kafka it has successfully handled messages up to that offset. The next message read will start from the committed offset plus one. If the consumer stops and restarts, it resumes from the last committed offset, ensuring no messages are missed or processed twice. The execution table traces each step, showing the offset read, message processed, offset committed, and the next offset to read. The variable tracker summarizes how these values change over time. Key moments clarify why committing after processing is important and what happens if offsets are not committed. The quiz tests understanding of these steps and their effects. This helps beginners see the flow of offset management clearly and avoid common mistakes.

Practice

1. What is the main purpose of offset management in Kafka?

easy

A. To track which messages have been read by a consumer

B. To store the actual message data in Kafka topics

C. To manage the number of partitions in a topic

D. To control the replication factor of Kafka brokers

Solution

1. Step 1: Understand offset management role

   Offset management keeps track of the last message a consumer has read in a partition.

2. Step 2: Differentiate from other Kafka features

   Storing messages, partition count, and replication are unrelated to offset tracking.

3. Final Answer:

   To track which messages have been read by a consumer -> Option A

4. Quick Check:

   Offset management = track read messages [OK]

Hint: Offset management tracks read messages, not data storage [OK]

Common Mistakes:

• Confusing offset with message storage
• Mixing offset with partition or replication management
• Thinking offset controls message content

2. Which of the following is the correct way to manually commit offsets in Kafka consumer API?

easy

A. consumer.offsetCommit()

B. consumer.commitOffset()

C. consumer.saveOffset()

D. consumer.commitSync()

Solution

1. Step 1: Recall Kafka consumer API methods

   The official method to commit offsets manually is commitSync() or commitAsync().

2. Step 2: Identify the exact method name

   Only commitSync() matches the correct Kafka API method for synchronous commit.

3. Final Answer:

   consumer.commitSync() -> Option D

4. Quick Check:

   Manual commit method = commitSync() [OK]

Hint: Manual commit uses commitSync() or commitAsync() [OK]

Common Mistakes:

• Using incorrect method names like commitOffset()
• Confusing commitSync() with offsetCommit()
• Assuming saveOffset() exists

3. Given the following code snippet, what will be the output if the consumer seeks to offset 5 and reads one message?

consumer.assign(Collections.singleton(topicPartition));
consumer.seek(topicPartition, 5);
ConsumerRecords records = consumer.poll(Duration.ofMillis(1000));
for (ConsumerRecord record : records) {
    System.out.println(record.offset());
    break;
}

medium

A. 5

B. 0

C. 1

D. Exception thrown

Solution

1. Step 1: Understand seek behavior

   Calling seek sets the consumer's position to offset 5 in the assigned partition.

2. Step 2: Poll and print offset

   The poll fetches messages starting at offset 5; printing record.offset() outputs 5.

3. Final Answer:

   5 -> Option A

4. Quick Check:

   seek to 5 then poll = offset 5 [OK]

Hint: seek sets start offset; first polled message offset matches [OK]

Common Mistakes:

• Assuming poll starts from offset 0
• Expecting offset 1 instead of 5
• Thinking seek causes an exception

4. What is wrong with the following code snippet for committing offsets manually?

consumer.poll(Duration.ofMillis(1000));
consumer.commitAsync();
consumer.close();

medium

A. consumer.close() should be called before commitAsync()

B. commitAsync() should be called before poll()

C. commitAsync() is called without processing polled records

D. poll() duration is too short for commitAsync()

Solution

1. Step 1: Check offset commit timing

   Offsets should be committed after processing records, not just after poll().

2. Step 2: Identify missing processing step

   The code commits offsets without handling any messages, so it commits offsets that may not be processed.

3. Final Answer:

   commitAsync() is called without processing polled records -> Option C

4. Quick Check:

   Commit after processing, not just after poll [OK]

Hint: Always process records before committing offsets [OK]

Common Mistakes:

• Calling commit before processing messages
• Closing consumer before committing
• Misunderstanding poll duration effect

5. You want to restart a Kafka consumer and continue reading from the last committed offset, but also have the option to start from the beginning if no offset is found. Which approach correctly manages offsets to achieve this?

hard

A. Always call consumer.seekToEnd() before polling messages

B. Use consumer.seekToBeginning() if no committed offset exists, else start from committed offset

C. Manually set offset to 0 every time before polling

D. Disable offset commits and rely on auto.offset.reset = latest

Solution

1. Step 1: Understand offset reset behavior

   If no committed offset exists, consumer can seek to beginning to read all messages.

2. Step 2: Combine offset check and seek

   Check for committed offset; if found, start there; if not, call seekToBeginning() to start fresh.

3. Final Answer:

   Use consumer.seekToBeginning() if no committed offset exists, else start from committed offset -> Option B

4. Quick Check:

   Seek beginning if no offset, else resume from committed [OK]

Hint: Seek beginning only if no committed offset found [OK]

Common Mistakes:

• Always seeking to end loses old messages
• Resetting offset to 0 blindly every time
• Disabling commits causes loss of progress