What is Commit Log in Kafka: Explanation and Example

In Apache Kafka, a commit log is an append-only sequence of records that stores all messages in the order they arrive. It acts like a durable, ordered journal where producers write data and consumers read it sequentially, ensuring reliable message delivery and replay.

How It Works

Think of Kafka's commit log as a notebook where you write down every event or message in the exact order it happens. This notebook never erases or changes entries; it only adds new ones at the end. This way, anyone reading the notebook can see the full history of events in the same order they were recorded.

In Kafka, producers add messages to this commit log, and consumers read from it at their own pace. Because the log is ordered and persistent, consumers can replay messages anytime, which is useful for recovering from failures or processing data multiple times.

Example

This example shows how to produce and consume messages using Kafka's commit log concept with the Kafka console tools.

bash

kafka-console-producer --broker-list localhost:9092 --topic test-topic
message1
message2

kafka-console-consumer --bootstrap-server localhost:9092 --topic test-topic --from-beginning
message1
message2

Output

message1
message2

When to Use

Use Kafka's commit log when you need a reliable, ordered, and durable way to record and process streams of data. It is ideal for event sourcing, real-time analytics, log aggregation, and building data pipelines where message order and replayability matter.

For example, a bank can use Kafka's commit log to record all transactions in order, ensuring accurate account balances and the ability to audit or replay transactions if needed.

Key Points

• The commit log is an append-only, ordered sequence of messages.
• Producers write messages to the end of the log.
• Consumers read messages in order and can replay them anytime.
• This design ensures durability, fault tolerance, and message ordering.

Key Takeaways

Kafka's commit log stores messages in a durable, ordered sequence.

Producers append messages; consumers read and replay them as needed.

It ensures reliable message delivery and supports fault tolerance.

Ideal for use cases requiring event ordering and replayability.