Log compaction in Kafka - Step-by-Step Execution

Process Flow - Log compaction

Start: Kafka Topic with Log

↓

New Messages Appended

↓

Log grows with keys and values

↓

Log Compaction Triggered

↓

For each key, keep only latest message

↓

Old duplicates removed

↓

Compacted Log Ready for Consumers

Log compaction keeps only the latest message per key in a Kafka topic, removing older duplicates to save space and ensure consumers get the newest data.

Execution Sample

Kafka

key = "user1"
log = [
  ("user1", "data1"),
  ("user2", "data2"),
  ("user1", "data3")
]
compacted_log = {}

This code simulates a Kafka log with messages for keys and prepares for compaction by storing only the latest message per key.

Process Table

Step	Action	Current Key	Current Value	Compacted Log State
1	Read first message	user1	data1	{"user1": "data1"}
2	Read second message	user2	data2	{"user1": "data1", "user2": "data2"}
3	Read third message	user1	data3	{"user1": "data3", "user2": "data2"}
4	Log compaction complete	-	-	{"user1": "data3", "user2": "data2"}

💡 All messages processed; compacted log keeps only latest value per key.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
compacted_log	{}	{"user1": "data1"}	{"user1": "data1", "user2": "data2"}	{"user1": "data3", "user2": "data2"}	{"user1": "data3", "user2": "data2"}

Key Moments - 2 Insights

Why does the value for 'user1' change after step 3?

Because log compaction keeps only the latest message per key, the value for 'user1' is updated from 'data1' to 'data3' as shown in execution_table step 3.

Does log compaction remove keys completely if they appear only once?

No, keys with a single message remain in the compacted log. Only older duplicates are removed, as seen in execution_table where 'user2' stays unchanged.

Visual Quiz - 3 Questions

Look at the execution_table at step 2, what keys are present in the compacted log?

A ["user1"]

B ["user1", "user2"]

C ["user2"]

D []

💡 Hint

Check the 'Compacted Log State' column at step 2 in execution_table.

At which step does the value for 'user1' get updated to 'data3'?

A Step 3

B Step 2

C Step 1

D Step 4

💡 Hint

Look at the 'Current Value' and 'Compacted Log State' columns in execution_table.

If a new message ("user2", "data4") is appended after step 3, what would happen in the compacted log?

A No change to compacted log

B "user2" is removed

C "user2" value updates to "data4"

D "user1" value changes to "data4"

💡 Hint

Log compaction keeps the latest message per key, so new messages update the key's value.

Concept Snapshot

Log compaction in Kafka:
- Keeps only the latest message per key in a topic
- Removes older duplicates to save space
- Ensures consumers read the newest data
- Runs in background without stopping producers
- Useful for changelog or state topics

Full Transcript

Log compaction is a Kafka feature that keeps only the latest message for each key in a topic. As new messages arrive, older messages with the same key are removed during compaction. This process helps save storage and ensures consumers get the most recent data for each key. The example shows a log with messages for 'user1' and 'user2'. After reading all messages, the compacted log keeps only the latest value for 'user1' and the single value for 'user2'. This is done step-by-step by updating the compacted log dictionary. Key moments include understanding why values update and that keys with single messages remain. The quizzes test understanding of the compacted log state at different steps and how new messages affect it.

Practice

1. What is the main purpose of log compaction in Kafka?

easy

A. To compress messages to save disk space

B. To delete all messages older than a certain time

C. To replicate messages across multiple brokers

D. To keep only the latest message for each key in a topic

Solution

1. Step 1: Understand log compaction concept

   Log compaction keeps the latest message per key, removing older duplicates.

2. Step 2: Compare with other options

   Deleting old messages is retention, compression is different, replication is for fault tolerance.

3. Final Answer:

   To keep only the latest message for each key in a topic -> Option D

4. Quick Check:

   Log compaction = latest message per key [OK]

Hint: Log compaction keeps latest per key, not all old messages [OK]

Common Mistakes:

• Confusing log compaction with time-based retention
• Thinking compression means log compaction
• Mixing replication with compaction

2. Which configuration enables log compaction on a Kafka topic?

easy

A. cleanup.policy=delete

B. cleanup.policy=compact

C. log.compaction.enabled=true

D. retention.ms=0

Solution

1. Step 1: Identify correct cleanup policy

   Log compaction is enabled by setting cleanup.policy=compact on the topic.

2. Step 2: Eliminate incorrect options

   cleanup.policy=delete is default retention, log.compaction.enabled is invalid, retention.ms controls time retention.

3. Final Answer:

   cleanup.policy=compact -> Option B

4. Quick Check:

   Enable compaction with cleanup.policy=compact [OK]

Hint: Remember: compact, not delete, enables log compaction [OK]

Common Mistakes:

• Using cleanup.policy=delete instead of compact
• Assuming a separate 'log.compaction.enabled' setting exists
• Confusing retention.ms with compaction setting

3. Given a compacted Kafka topic with keys and values:
key1: v1, key2: v2, key1: v3, key3: v4
What messages remain after log compaction?

medium

A. key1: v3, key2: v2, key3: v4

B. key1: v1, key2: v2, key3: v4

C. key1: v3, key2: v2

D. key2: v2, key3: v4

Solution

1. Step 1: Identify latest message per key

   For key1, latest is v3; key2 is v2; key3 is v4.

2. Step 2: List messages after compaction

   Only latest per key remain: key1:v3, key2:v2, key3:v4.

3. Final Answer:

   key1: v3, key2: v2, key3: v4 -> Option A

4. Quick Check:

   Latest message per key kept [OK]

Hint: Keep only last message per key after compaction [OK]

Common Mistakes:

• Keeping all messages instead of only latest per key
• Confusing order and keeping first message per key
• Dropping keys incorrectly

4. You set cleanup.policy=compact on a topic but notice old messages are not removed. What is a likely cause?

medium

A. The topic has no keys in messages

B. The retention.ms is set to 0

C. The broker is running in debug mode

D. The topic has too many partitions

Solution

1. Step 1: Understand log compaction requirement

   Log compaction requires messages to have keys to identify duplicates.

2. Step 2: Analyze why old messages remain

   If messages lack keys, compaction cannot remove duplicates, so old messages stay.

3. Final Answer:

   The topic has no keys in messages -> Option A

4. Quick Check:

   Compaction needs keys to remove old messages [OK]

Hint: Compaction needs keys; no keys means no compaction [OK]

Common Mistakes:

• Assuming retention.ms=0 disables compaction
• Blaming debug mode for compaction issues
• Thinking partition count affects compaction directly

5. You want to store the current state of user profiles in Kafka using log compaction. Which approach is best?

hard

A. Use a topic with cleanup.policy=delete and produce messages without keys

B. Use a topic with retention.ms=0 and produce messages with random keys

C. Use a topic with cleanup.policy=compact and produce messages with user ID as key

D. Use a topic with cleanup.policy=compact but produce messages without keys

Solution

1. Step 1: Identify requirements for storing current state

   Current state needs latest update per user, so keys must be user IDs.

2. Step 2: Choose correct topic configuration

   Log compaction enabled with cleanup.policy=compact keeps latest per key.

3. Step 3: Eliminate wrong options

   Delete policy removes old messages by time, no keys prevent compaction, random keys break state tracking.

4. Final Answer:

   Use a topic with cleanup.policy=compact and produce messages with user ID as key -> Option C

5. Quick Check:

   Compaction + keys = current state storage [OK]

Hint: Use compact policy and keys for current state topics [OK]

Common Mistakes:

• Using delete policy for state storage
• Producing messages without keys
• Using random keys that break compaction logic