What is enable.auto.commit in Kafka: Explanation and Usage

enable.auto.commit in Kafka is a configuration setting that controls whether the consumer automatically saves its position (offset) in the topic after reading messages. When set to true, Kafka commits offsets automatically at regular intervals, helping track which messages have been processed.

How It Works

Imagine you are reading a book and want to remember the last page you read so you can continue later. In Kafka, the "page" is called an offset, which marks the position of the last message a consumer has processed. The enable.auto.commit setting tells Kafka if it should remember this position automatically for you.

If enable.auto.commit is set to true, Kafka will save the offset at regular time intervals without you needing to do anything. This is like a bookmark that updates itself every few seconds. If it is false, you must manually tell Kafka when to save the offset, giving you more control but requiring extra work.

Example

This example shows a simple Kafka consumer in Java with enable.auto.commit set to true. The consumer automatically commits offsets every 1 second (1000 ms).

java

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class AutoCommitConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "test-group");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
        props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Collections.singletonList("my-topic"));

        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
            for (ConsumerRecord<String, String> record : records) {
                System.out.printf("Received message: key = %s, value = %s, offset = %d\n", record.key(), record.value(), record.offset());
            }
        }
    }
}

Output

Received message: key = key1, value = hello, offset = 5
Received message: key = key2, value = world, offset = 6

When to Use

Use enable.auto.commit=true when you want a simple setup where Kafka automatically tracks which messages your consumer has processed. This is good for applications where occasional duplicate processing is acceptable and you want to avoid manual offset management.

Use enable.auto.commit=false when you need precise control over when offsets are saved, such as when processing messages involves multiple steps or external systems. This helps avoid losing messages or processing them twice if your application crashes.

Key Points

• enable.auto.commit controls automatic saving of consumer offsets.
• When true, offsets commit automatically at intervals set by auto.commit.interval.ms.
• Automatic commits simplify offset management but may cause duplicate processing on failure.
• Manual commits (false) give more control and reliability for complex processing.

Key Takeaways

enable.auto.commit=true lets Kafka save consumer offsets automatically at intervals.

Automatic commits simplify offset tracking but can cause duplicates if failures occur.

Set enable.auto.commit=false for manual offset control in critical processing scenarios.

auto.commit.interval.ms controls how often offsets are saved when auto commit is enabled.

Choosing between auto and manual commit depends on your application's reliability needs.