Consumer offset commit strategies in Kafka - Commands & Configuration

Introduction

When a Kafka consumer reads messages, it needs to remember which messages it has processed. This is done by committing offsets. Different commit strategies help balance between processing speed and data safety.

When you want to ensure no message is processed twice even if the consumer crashes.

When you want faster processing and can tolerate some duplicate message processing.

When you want to commit offsets automatically without manual intervention.

When you want to control exactly when offsets are committed after processing messages.

When you want to avoid losing messages during consumer restarts.

Commands

Starts a Kafka consumer that automatically commits offsets every 5 seconds. This is the automatic commit strategy for convenience and speed.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --group example-group --enable-auto-commit true --auto-commit-interval-ms 5000

Expected Output

This will start printing messages from example-topic as they arrive.

Offsets will be committed automatically every 5 seconds.

→

--enable-auto-commit - Enables automatic offset commits.

→

--auto-commit-interval-ms - Sets how often offsets are committed automatically.

Starts a Kafka consumer with automatic commits disabled. This means you must commit offsets manually to control when messages are marked as processed.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --group example-group --enable-auto-commit false

Expected Output

This will start printing messages from example-topic as they arrive.

Offsets will not be committed automatically.

→

--enable-auto-commit - Disables automatic offset commits.

Shows the current committed offsets and lag for the consumer group. Useful to verify which messages have been processed and committed.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --group example-group --describe

Expected Output

TOPIC          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID                                   HOST            CLIENT-ID

example-topic  0          15              20              5    consumer-1-12345-67890                      /127.0.0.1      consumer-1

→

--group - Specifies the consumer group to inspect.

Key Concept

If you remember nothing else from this pattern, remember: committing offsets tells Kafka which messages you have processed, and choosing when to commit balances speed and safety.

Common Mistakes

Leaving auto-commit enabled but processing messages slowly without handling duplicates.

Offsets may be committed before processing finishes, causing message loss if the consumer crashes.

Disable auto-commit and commit offsets manually after processing each message or batch.

Disabling auto-commit but never committing offsets manually.

Kafka will think messages are unprocessed and resend them, causing duplicates.

Always commit offsets manually at appropriate points in your processing.

Committing offsets before processing messages.

If processing fails after commit, messages are lost and not retried.

Commit offsets only after successful processing.

Summary

Automatic offset commit lets Kafka track processed messages regularly without manual work.

Manual offset commit gives precise control to avoid message loss or duplicates.

Use kafka-consumer-groups command to check committed offsets and consumer lag.