Log compaction in Kafka - Commands & Configuration

Introduction

Log compaction in Kafka helps keep only the latest value for each key in a topic. This saves space and ensures consumers can always get the most recent update without reading the entire log.

When you want to keep a snapshot of the latest state for each key in a topic.

When you need to recover the current state quickly after a failure.

When you want to reduce storage by removing old records but keep the latest updates.

When you have a topic that stores user profiles or configurations that change over time.

When you want to ensure consumers can rebuild state from a compacted topic efficiently.

Config File - server.properties

server.properties

log.cleaner.enable=true
log.cleaner.threads=2
log.retention.ms=604800000
log.cleanup.policy=compact

log.cleaner.enable=true turns on log compaction in Kafka.

log.cleaner.threads=2 sets how many threads run the cleaner process.

log.retention.ms=604800000 keeps logs for 7 days before deletion.

log.cleanup.policy=compact sets the topic to use log compaction instead of deletion.

Commands

This command creates a Kafka topic named 'user-profiles' with log compaction enabled. It sets 3 partitions and replication factor 1 for simplicity.

Terminal

kafka-topics.sh --create --topic user-profiles --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1 --config cleanup.policy=compact

Expected Output

Created topic user-profiles.

→

--config cleanup.policy=compact - Enables log compaction for this topic

→

--partitions 3 - Sets number of partitions

→

--replication-factor 1 - Sets replication factor

Starts a producer to send messages to the 'user-profiles' topic. You can type key-value pairs to simulate updates.

Terminal

kafka-console-producer.sh --topic user-profiles --bootstrap-server localhost:9092 --property parse.key=true --property key.separator=:

Expected Output

No output (command runs silently)

→

--topic user-profiles - Specifies the topic to send messages to

→

--bootstrap-server localhost:9092 - Connects to the Kafka server

→

--property parse.key=true - Enables parsing of keys from input

→

--property key.separator=: - Sets the separator between key and value

Reads all messages from the beginning of the 'user-profiles' topic to see the current and past data.

Terminal

kafka-console-consumer.sh --topic user-profiles --bootstrap-server localhost:9092 --from-beginning --property print.key=true --property key.separator=:

Expected Output

user1:{"name":"Alice","age":30}

user2:{"name":"Bob","age":25}

user1:{"name":"Alice","age":31}

→

--from-beginning - Reads all messages from the start of the topic

→

--property print.key=true - Prints the key along with the value

→

--property key.separator=: - Sets the separator between key and value in output

Shows details about the 'user-profiles' topic including its cleanup policy to confirm log compaction is enabled.

Terminal

kafka-topics.sh --describe --topic user-profiles --bootstrap-server localhost:9092

Expected Output

Topic: user-profiles	PartitionCount: 3	ReplicationFactor: 1	Configs: cleanup.policy=compact

Key Concept

If you remember nothing else from this pattern, remember: log compaction keeps only the latest record for each key, saving space and enabling fast state recovery.

Common Mistakes

Creating a topic without setting cleanup.policy=compact

The topic will not compact logs and will keep all old messages, wasting storage.

Always set --config cleanup.policy=compact when creating a topic that needs log compaction.

Not enabling log.cleaner.enable=true in the Kafka broker config

Log compaction will not run because the cleaner process is disabled.

Set log.cleaner.enable=true in server.properties and restart Kafka brokers.

Using keys inconsistently in messages sent to the topic

Log compaction works by key; if keys are missing or random, compaction cannot keep the latest state.

Always send messages with consistent keys to enable proper compaction.

Summary

Create a Kafka topic with cleanup.policy=compact to enable log compaction.

Enable log.cleaner.enable=true in Kafka broker configuration to run the compaction process.

Produce messages with keys to the topic so Kafka can keep only the latest record per key.

Use kafka-console-consumer to verify messages and kafka-topics to check topic configuration.