Auto-commit vs manual commit in Kafka - CLI Comparison

Introduction

Kafka consumers read messages from topics. They must tell Kafka which messages they have processed by committing offsets. Auto-commit does this automatically, while manual commit lets you control when to mark messages as processed.

When you want Kafka to track message processing automatically without extra code.

When you need to ensure messages are fully processed before marking them done.

When processing messages might fail and you want to retry before committing.

When you want to control commit timing to optimize performance.

When you want to avoid losing messages or processing duplicates.

Commands

Starts a Kafka consumer that automatically commits offsets after reading messages. This means Kafka tracks progress without manual intervention.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --group example-group --enable-auto-commit true

Expected Output

This will print messages from example-topic as they arrive, committing offsets automatically.

→

--enable-auto-commit - Enables automatic offset commits.

→

--group - Specifies the consumer group for offset tracking.

Starts a Kafka consumer with auto-commit disabled. You must manually commit offsets in your consumer code to mark messages as processed.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --group example-group --enable-auto-commit false

Expected Output

This will print messages from example-topic but will not commit offsets automatically.

→

--enable-auto-commit - Disables automatic offset commits.

→

--group - Specifies the consumer group for offset tracking.

Shows the current offset positions and lag for the consumer group. Useful to verify if offsets are committed.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group example-group

Expected Output

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST  CLIENT-ID

example-topic   0          15              20              5    consumer-1   /127.0.0.1  consumer-1

→

--describe - Shows detailed offset and lag information.

→

--group - Specifies the consumer group to inspect.

Key Concept

If you remember nothing else from this pattern, remember: auto-commit is easy but can lose messages on failure, manual commit gives control but needs extra code.

Common Mistakes

Using auto-commit when message processing can fail.

Offsets get committed before processing finishes, causing message loss if failure happens.

Disable auto-commit and commit offsets manually after successful processing.

Forgetting to commit offsets when auto-commit is disabled.

Kafka will keep delivering the same messages repeatedly, causing duplicates.

Add code to commit offsets after processing each message or batch.

Setting auto-commit interval too high.

Offsets commit too late, increasing risk of reprocessing messages after failure.

Set a reasonable auto-commit interval or use manual commit for precise control.

Summary

Auto-commit lets Kafka track message progress automatically but risks losing messages on failure.

Manual commit requires explicit code to mark messages processed, giving full control and safety.

Use kafka-consumer-groups command to check committed offsets and consumer lag.