How to Consume Kafka Messages from the Beginning

To consume messages from the beginning of a Kafka topic, set the consumer configuration auto.offset.reset to earliest. This tells Kafka to start reading from the oldest available message if no committed offset exists for the consumer group.

Syntax

The key configuration to consume from the beginning is auto.offset.reset. It accepts values like earliest, latest, or none.

Setting auto.offset.reset=earliest means the consumer will start reading from the oldest message if no previous offset is found.

Other important settings include group.id to identify the consumer group and bootstrap.servers to connect to Kafka brokers.

java

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "my-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("auto.offset.reset", "earliest");

Example

This Java example shows a Kafka consumer configured to read all messages from the beginning of the topic my-topic. It prints each message key and value.

java

import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class ConsumeFromBeginning {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("group.id", "my-group");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("auto.offset.reset", "earliest");

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(Collections.singletonList("my-topic"));

            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                for (ConsumerRecord<String, String> record : records) {
                    System.out.printf("Consumed message: key = %s, value = %s, offset = %d\n", record.key(), record.value(), record.offset());
                }
            }
        }
    }
}

Output

Consumed message: key = key1, value = Hello, offset = 0
Consumed message: key = key2, value = World, offset = 1
...

Common Pitfalls

• Not setting auto.offset.reset to earliest: The default is usually latest, which skips old messages.
• Using the same group.id repeatedly: If offsets are already committed, the consumer will resume from the last committed offset, not the beginning.
• Expecting to read from beginning without resetting offsets: To re-read from the start, you may need to use a new group.id or reset offsets manually.

java

/* Wrong: default offset reset (latest) skips old messages */
props.put("auto.offset.reset", "latest");

/* Right: consume from beginning if no offset */
props.put("auto.offset.reset", "earliest");

Quick Reference

Setting	Description	Example Value
bootstrap.servers	Kafka broker addresses	localhost:9092
group.id	Consumer group identifier	my-group
key.deserializer	Class to deserialize message keys	org.apache.kafka.common.serialization.StringDeserializer
value.deserializer	Class to deserialize message values	org.apache.kafka.common.serialization.StringDeserializer
auto.offset.reset	Where to start if no offset found	earliest

Key Takeaways

Set 'auto.offset.reset' to 'earliest' to consume messages from the beginning if no offset exists.

Use a unique 'group.id' to start fresh consumption from the beginning.

If offsets exist for a group, Kafka resumes from the last committed offset, not the start.

Always configure deserializers and bootstrap servers correctly for your consumer.

To re-consume old messages, consider resetting offsets or using a new consumer group.