Log compaction in Kafka - Commands & Configuration

Introduction

Log compaction in Kafka helps keep only the latest value for each key in a topic. This saves space and ensures consumers can always get the most recent update without reading the entire log.

When you want to keep a snapshot of the latest state for each key in a topic.

When you need to recover the current state quickly after a failure.

When you want to reduce storage by removing old records but keep the latest updates.

When you have a topic that stores user profiles or configurations that change over time.

When you want to ensure consumers can rebuild state from a compacted topic efficiently.

Config File - server.properties

server.properties

log.cleaner.enable=true
log.cleaner.threads=2
log.retention.ms=604800000
log.cleanup.policy=compact

log.cleaner.enable=true turns on log compaction in Kafka.

log.cleaner.threads=2 sets how many threads run the cleaner process.

log.retention.ms=604800000 keeps logs for 7 days before deletion.

log.cleanup.policy=compact sets the topic to use log compaction instead of deletion.

Commands

This command creates a Kafka topic named 'user-profiles' with log compaction enabled. It sets 3 partitions and replication factor 1 for simplicity.

Terminal

kafka-topics.sh --create --topic user-profiles --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1 --config cleanup.policy=compact

Expected Output

Created topic user-profiles.

→

--config cleanup.policy=compact - Enables log compaction for this topic

→

--partitions 3 - Sets number of partitions

→

--replication-factor 1 - Sets replication factor

Starts a producer to send messages to the 'user-profiles' topic. You can type key-value pairs to simulate updates.

Terminal

kafka-console-producer.sh --topic user-profiles --bootstrap-server localhost:9092 --property parse.key=true --property key.separator=:

Expected Output

No output (command runs silently)

→

--topic user-profiles - Specifies the topic to send messages to

→

--bootstrap-server localhost:9092 - Connects to the Kafka server

→

--property parse.key=true - Enables parsing of keys from input

→

--property key.separator=: - Sets the separator between key and value

Reads all messages from the beginning of the 'user-profiles' topic to see the current and past data.

Terminal

kafka-console-consumer.sh --topic user-profiles --bootstrap-server localhost:9092 --from-beginning --property print.key=true --property key.separator=:

Expected Output

user1:{"name":"Alice","age":30}

user2:{"name":"Bob","age":25}

user1:{"name":"Alice","age":31}

→

--from-beginning - Reads all messages from the start of the topic

→

--property print.key=true - Prints the key along with the value

→

--property key.separator=: - Sets the separator between key and value in output

Shows details about the 'user-profiles' topic including its cleanup policy to confirm log compaction is enabled.

Terminal

kafka-topics.sh --describe --topic user-profiles --bootstrap-server localhost:9092

Expected Output

Topic: user-profiles	PartitionCount: 3	ReplicationFactor: 1	Configs: cleanup.policy=compact

Key Concept

If you remember nothing else from this pattern, remember: log compaction keeps only the latest record for each key, saving space and enabling fast state recovery.

Common Mistakes

Creating a topic without setting cleanup.policy=compact

The topic will not compact logs and will keep all old messages, wasting storage.

Always set --config cleanup.policy=compact when creating a topic that needs log compaction.

Not enabling log.cleaner.enable=true in the Kafka broker config

Log compaction will not run because the cleaner process is disabled.

Set log.cleaner.enable=true in server.properties and restart Kafka brokers.

Using keys inconsistently in messages sent to the topic

Log compaction works by key; if keys are missing or random, compaction cannot keep the latest state.

Always send messages with consistent keys to enable proper compaction.

Summary

Create a Kafka topic with cleanup.policy=compact to enable log compaction.

Enable log.cleaner.enable=true in Kafka broker configuration to run the compaction process.

Produce messages with keys to the topic so Kafka can keep only the latest record per key.

Use kafka-console-consumer to verify messages and kafka-topics to check topic configuration.

Practice

1. What is the main purpose of log compaction in Kafka?

easy

A. To compress messages to save disk space

B. To delete all messages older than a certain time

C. To replicate messages across multiple brokers

D. To keep only the latest message for each key in a topic

Solution

1. Step 1: Understand log compaction concept

   Log compaction keeps the latest message per key, removing older duplicates.

2. Step 2: Compare with other options

   Deleting old messages is retention, compression is different, replication is for fault tolerance.

3. Final Answer:

   To keep only the latest message for each key in a topic -> Option D

4. Quick Check:

   Log compaction = latest message per key [OK]

Hint: Log compaction keeps latest per key, not all old messages [OK]

Common Mistakes:

• Confusing log compaction with time-based retention
• Thinking compression means log compaction
• Mixing replication with compaction

2. Which configuration enables log compaction on a Kafka topic?

easy

A. cleanup.policy=delete

B. cleanup.policy=compact

C. log.compaction.enabled=true

D. retention.ms=0

Solution

1. Step 1: Identify correct cleanup policy

   Log compaction is enabled by setting cleanup.policy=compact on the topic.

2. Step 2: Eliminate incorrect options

   cleanup.policy=delete is default retention, log.compaction.enabled is invalid, retention.ms controls time retention.

3. Final Answer:

   cleanup.policy=compact -> Option B

4. Quick Check:

   Enable compaction with cleanup.policy=compact [OK]

Hint: Remember: compact, not delete, enables log compaction [OK]

Common Mistakes:

• Using cleanup.policy=delete instead of compact
• Assuming a separate 'log.compaction.enabled' setting exists
• Confusing retention.ms with compaction setting

3. Given a compacted Kafka topic with keys and values:
key1: v1, key2: v2, key1: v3, key3: v4
What messages remain after log compaction?

medium

A. key1: v3, key2: v2, key3: v4

B. key1: v1, key2: v2, key3: v4

C. key1: v3, key2: v2

D. key2: v2, key3: v4

Solution

1. Step 1: Identify latest message per key

   For key1, latest is v3; key2 is v2; key3 is v4.

2. Step 2: List messages after compaction

   Only latest per key remain: key1:v3, key2:v2, key3:v4.

3. Final Answer:

   key1: v3, key2: v2, key3: v4 -> Option A

4. Quick Check:

   Latest message per key kept [OK]

Hint: Keep only last message per key after compaction [OK]

Common Mistakes:

• Keeping all messages instead of only latest per key
• Confusing order and keeping first message per key
• Dropping keys incorrectly

4. You set cleanup.policy=compact on a topic but notice old messages are not removed. What is a likely cause?

medium

A. The topic has no keys in messages

B. The retention.ms is set to 0

C. The broker is running in debug mode

D. The topic has too many partitions

Solution

1. Step 1: Understand log compaction requirement

   Log compaction requires messages to have keys to identify duplicates.

2. Step 2: Analyze why old messages remain

   If messages lack keys, compaction cannot remove duplicates, so old messages stay.

3. Final Answer:

   The topic has no keys in messages -> Option A

4. Quick Check:

   Compaction needs keys to remove old messages [OK]

Hint: Compaction needs keys; no keys means no compaction [OK]

Common Mistakes:

• Assuming retention.ms=0 disables compaction
• Blaming debug mode for compaction issues
• Thinking partition count affects compaction directly

5. You want to store the current state of user profiles in Kafka using log compaction. Which approach is best?

hard

A. Use a topic with cleanup.policy=delete and produce messages without keys

B. Use a topic with retention.ms=0 and produce messages with random keys

C. Use a topic with cleanup.policy=compact and produce messages with user ID as key

D. Use a topic with cleanup.policy=compact but produce messages without keys

Solution

1. Step 1: Identify requirements for storing current state

   Current state needs latest update per user, so keys must be user IDs.

2. Step 2: Choose correct topic configuration

   Log compaction enabled with cleanup.policy=compact keeps latest per key.

3. Step 3: Eliminate wrong options

   Delete policy removes old messages by time, no keys prevent compaction, random keys break state tracking.

4. Final Answer:

   Use a topic with cleanup.policy=compact and produce messages with user ID as key -> Option C

5. Quick Check:

   Compaction + keys = current state storage [OK]

Hint: Use compact policy and keys for current state topics [OK]

Common Mistakes:

• Using delete policy for state storage
• Producing messages without keys
• Using random keys that break compaction logic