Log compaction in Kafka - Mini Project: Build & Apply

Log Compaction in Kafka

📖 Scenario: You are working with Apache Kafka, a system that helps send messages between different parts of a program. Sometimes, you want to keep only the latest message for each key to save space and keep data fresh. This is called log compaction.

🎯 Goal: You will create a Kafka topic with log compaction enabled, produce messages with keys, and then consume the messages to see how log compaction keeps only the latest message for each key.

📋 What You'll Learn

Create a Kafka topic named user-updates with log compaction enabled

Produce messages with keys user1, user2, and user1 again with updated values

Consume messages from the user-updates topic to observe log compaction effect

💡 Why This Matters

🌍 Real World

Log compaction is used in real systems to keep only the latest state for each key, saving storage and making data processing efficient.

💼 Career

Understanding log compaction is important for roles working with Kafka in data engineering, backend development, and real-time data processing.

1

Create Kafka topic with log compaction

Use the Kafka command line tool to create a topic called user-updates with --config cleanup.policy=compact to enable log compaction.

Kafka

# Run the command to create the topic with log compaction enabled
# Your code here

Hint

Use kafka-topics --create with --config cleanup.policy=compact to enable log compaction.

2

Produce messages with keys to the topic

Use the Kafka console producer to send three messages to user-updates with keys and values exactly as: user1:Alice, user2:Bob, and user1:Alicia.

Kafka

# Use kafka-console-producer with --property to send messages with keys
# Your code here

Hint

Use kafka-console-producer with --property parse.key=true and --property key.separator=: to send keyed messages.

3

Consume messages from the topic

Use the Kafka console consumer to read messages from user-updates from the beginning, showing keys and values.

Kafka

# Use kafka-console-consumer with --property to show keys and values from beginning
# Your code here

Hint

Use kafka-console-consumer with --from-beginning and --property print.key=true to see keys and values.

4

Observe log compaction effect

Run the consumer command and observe that only the latest message for user1 is kept, showing user1:Alicia and user2:Bob.

Kafka

# Run the consumer command and observe output
# Your code here

Hint

Log compaction keeps only the latest message per key. You should see user2:Bob and user1:Alicia in the output.

Practice

1. What is the main purpose of log compaction in Kafka?

easy

A. To compress messages to save disk space

B. To delete all messages older than a certain time

C. To replicate messages across multiple brokers

D. To keep only the latest message for each key in a topic

Solution

1. Step 1: Understand log compaction concept

   Log compaction keeps the latest message per key, removing older duplicates.

2. Step 2: Compare with other options

   Deleting old messages is retention, compression is different, replication is for fault tolerance.

3. Final Answer:

   To keep only the latest message for each key in a topic -> Option D

4. Quick Check:

   Log compaction = latest message per key [OK]

Hint: Log compaction keeps latest per key, not all old messages [OK]

Common Mistakes:

• Confusing log compaction with time-based retention
• Thinking compression means log compaction
• Mixing replication with compaction

2. Which configuration enables log compaction on a Kafka topic?

easy

A. cleanup.policy=delete

B. cleanup.policy=compact

C. log.compaction.enabled=true

D. retention.ms=0

Solution

1. Step 1: Identify correct cleanup policy

   Log compaction is enabled by setting cleanup.policy=compact on the topic.

2. Step 2: Eliminate incorrect options

   cleanup.policy=delete is default retention, log.compaction.enabled is invalid, retention.ms controls time retention.

3. Final Answer:

   cleanup.policy=compact -> Option B

4. Quick Check:

   Enable compaction with cleanup.policy=compact [OK]

Hint: Remember: compact, not delete, enables log compaction [OK]

Common Mistakes:

• Using cleanup.policy=delete instead of compact
• Assuming a separate 'log.compaction.enabled' setting exists
• Confusing retention.ms with compaction setting

3. Given a compacted Kafka topic with keys and values:
key1: v1, key2: v2, key1: v3, key3: v4
What messages remain after log compaction?

medium

A. key1: v3, key2: v2, key3: v4

B. key1: v1, key2: v2, key3: v4

C. key1: v3, key2: v2

D. key2: v2, key3: v4

Solution

1. Step 1: Identify latest message per key

   For key1, latest is v3; key2 is v2; key3 is v4.

2. Step 2: List messages after compaction

   Only latest per key remain: key1:v3, key2:v2, key3:v4.

3. Final Answer:

   key1: v3, key2: v2, key3: v4 -> Option A

4. Quick Check:

   Latest message per key kept [OK]

Hint: Keep only last message per key after compaction [OK]

Common Mistakes:

• Keeping all messages instead of only latest per key
• Confusing order and keeping first message per key
• Dropping keys incorrectly

4. You set cleanup.policy=compact on a topic but notice old messages are not removed. What is a likely cause?

medium

A. The topic has no keys in messages

B. The retention.ms is set to 0

C. The broker is running in debug mode

D. The topic has too many partitions

Solution

1. Step 1: Understand log compaction requirement

   Log compaction requires messages to have keys to identify duplicates.

2. Step 2: Analyze why old messages remain

   If messages lack keys, compaction cannot remove duplicates, so old messages stay.

3. Final Answer:

   The topic has no keys in messages -> Option A

4. Quick Check:

   Compaction needs keys to remove old messages [OK]

Hint: Compaction needs keys; no keys means no compaction [OK]

Common Mistakes:

• Assuming retention.ms=0 disables compaction
• Blaming debug mode for compaction issues
• Thinking partition count affects compaction directly

5. You want to store the current state of user profiles in Kafka using log compaction. Which approach is best?

hard

A. Use a topic with cleanup.policy=delete and produce messages without keys

B. Use a topic with retention.ms=0 and produce messages with random keys

C. Use a topic with cleanup.policy=compact and produce messages with user ID as key

D. Use a topic with cleanup.policy=compact but produce messages without keys

Solution

1. Step 1: Identify requirements for storing current state

   Current state needs latest update per user, so keys must be user IDs.

2. Step 2: Choose correct topic configuration

   Log compaction enabled with cleanup.policy=compact keeps latest per key.

3. Step 3: Eliminate wrong options

   Delete policy removes old messages by time, no keys prevent compaction, random keys break state tracking.

4. Final Answer:

   Use a topic with cleanup.policy=compact and produce messages with user ID as key -> Option C

5. Quick Check:

   Compaction + keys = current state storage [OK]

Hint: Use compact policy and keys for current state topics [OK]

Common Mistakes:

• Using delete policy for state storage
• Producing messages without keys
• Using random keys that break compaction logic