Log compaction in Kafka - Mini Project: Build & Apply

Log Compaction in Kafka

📖 Scenario: You are working with Apache Kafka, a system that helps send messages between different parts of a program. Sometimes, you want to keep only the latest message for each key to save space and keep data fresh. This is called log compaction.

🎯 Goal: You will create a Kafka topic with log compaction enabled, produce messages with keys, and then consume the messages to see how log compaction keeps only the latest message for each key.

📋 What You'll Learn

Create a Kafka topic named user-updates with log compaction enabled

Produce messages with keys user1, user2, and user1 again with updated values

Consume messages from the user-updates topic to observe log compaction effect

💡 Why This Matters

🌍 Real World

Log compaction is used in real systems to keep only the latest state for each key, saving storage and making data processing efficient.

💼 Career

Understanding log compaction is important for roles working with Kafka in data engineering, backend development, and real-time data processing.

1

Create Kafka topic with log compaction

Use the Kafka command line tool to create a topic called user-updates with --config cleanup.policy=compact to enable log compaction.

Kafka

# Run the command to create the topic with log compaction enabled
# Your code here

Need a hint?

Use kafka-topics --create with --config cleanup.policy=compact to enable log compaction.

2

Produce messages with keys to the topic

Use the Kafka console producer to send three messages to user-updates with keys and values exactly as: user1:Alice, user2:Bob, and user1:Alicia.

Kafka

# Use kafka-console-producer with --property to send messages with keys
# Your code here

Need a hint?

Use kafka-console-producer with --property parse.key=true and --property key.separator=: to send keyed messages.

3

Consume messages from the topic

Use the Kafka console consumer to read messages from user-updates from the beginning, showing keys and values.

Kafka

# Use kafka-console-consumer with --property to show keys and values from beginning
# Your code here

Need a hint?

Use kafka-console-consumer with --from-beginning and --property print.key=true to see keys and values.

4

Observe log compaction effect

Run the consumer command and observe that only the latest message for user1 is kept, showing user1:Alicia and user2:Bob.

Kafka

# Run the consumer command and observe output
# Your code here

Need a hint?

Log compaction keeps only the latest message per key. You should see user2:Bob and user1:Alicia in the output.