Log compaction in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is log compaction in Kafka?

Log compaction is a Kafka feature that retains only the latest value for each key in a topic, removing older duplicates to save storage and ensure data freshness.

beginner

How does Kafka decide which messages to keep during log compaction?

Kafka keeps the latest message for each unique key and removes older messages with the same key during log compaction.

intermediate

What is the difference between log compaction and log retention by time?

Log retention by time deletes messages older than a set time, while log compaction keeps the latest message per key regardless of age.

intermediate

Why is log compaction useful in Kafka?

It helps keep topics small by removing outdated data, supports stateful stream processing, and ensures consumers can recover the latest state efficiently.

advanced

What happens if a message with a key is sent with a null value in a compacted topic?

A message with a null value acts as a delete marker (tombstone) for that key, so the key and its data are removed during compaction.

What does Kafka's log compaction keep for each key?

A The latest message only

B All messages regardless of key

C Only messages older than a set time

D Messages with null values only

Log compaction keeps only the latest message for each unique key.

What is the purpose of a tombstone message in a compacted topic?

A To add a new key

B To mark a key for deletion

C To duplicate a message

D To reset the topic

A tombstone message has a null value and marks the key for deletion during compaction.

Which scenario best fits using log compaction?

A Archiving logs to cold storage

B Storing all sensor readings forever

C Deleting messages after 7 days

D Keeping the latest user profile updates

Log compaction is ideal for keeping the latest state like user profiles.

How does log compaction affect topic size?

A It increases size by adding more messages

B It has no effect on size

C It reduces size by removing old duplicates

D It deletes all messages periodically

Log compaction removes older duplicates, reducing topic size.

Can log compaction remove messages with unique keys that have never been duplicated?

A Yes, if the message value is null

B Yes, it removes all messages older than a threshold

C No, it keeps the latest message for each key

D No, it removes only messages with duplicate keys

Messages with null values act as tombstones and are removed during compaction.

Explain how Kafka's log compaction works and why it is useful.

Think about how Kafka keeps only the freshest data for each key.

Describe the role of tombstone messages in log compaction.

Consider how Kafka deletes keys using special messages.

Practice

1. What is the main purpose of log compaction in Kafka?

easy

A. To compress messages to save disk space

B. To delete all messages older than a certain time

C. To replicate messages across multiple brokers

D. To keep only the latest message for each key in a topic

Solution

1. Step 1: Understand log compaction concept

   Log compaction keeps the latest message per key, removing older duplicates.

2. Step 2: Compare with other options

   Deleting old messages is retention, compression is different, replication is for fault tolerance.

3. Final Answer:

   To keep only the latest message for each key in a topic -> Option D

4. Quick Check:

   Log compaction = latest message per key [OK]

Hint: Log compaction keeps latest per key, not all old messages [OK]

Common Mistakes:

• Confusing log compaction with time-based retention
• Thinking compression means log compaction
• Mixing replication with compaction

2. Which configuration enables log compaction on a Kafka topic?

easy

A. cleanup.policy=delete

B. cleanup.policy=compact

C. log.compaction.enabled=true

D. retention.ms=0

Solution

1. Step 1: Identify correct cleanup policy

   Log compaction is enabled by setting cleanup.policy=compact on the topic.

2. Step 2: Eliminate incorrect options

   cleanup.policy=delete is default retention, log.compaction.enabled is invalid, retention.ms controls time retention.

3. Final Answer:

   cleanup.policy=compact -> Option B

4. Quick Check:

   Enable compaction with cleanup.policy=compact [OK]

Hint: Remember: compact, not delete, enables log compaction [OK]

Common Mistakes:

• Using cleanup.policy=delete instead of compact
• Assuming a separate 'log.compaction.enabled' setting exists
• Confusing retention.ms with compaction setting

3. Given a compacted Kafka topic with keys and values:
key1: v1, key2: v2, key1: v3, key3: v4
What messages remain after log compaction?

medium

A. key1: v3, key2: v2, key3: v4

B. key1: v1, key2: v2, key3: v4

C. key1: v3, key2: v2

D. key2: v2, key3: v4

Solution

1. Step 1: Identify latest message per key

   For key1, latest is v3; key2 is v2; key3 is v4.

2. Step 2: List messages after compaction

   Only latest per key remain: key1:v3, key2:v2, key3:v4.

3. Final Answer:

   key1: v3, key2: v2, key3: v4 -> Option A

4. Quick Check:

   Latest message per key kept [OK]

Hint: Keep only last message per key after compaction [OK]

Common Mistakes:

• Keeping all messages instead of only latest per key
• Confusing order and keeping first message per key
• Dropping keys incorrectly

4. You set cleanup.policy=compact on a topic but notice old messages are not removed. What is a likely cause?

medium

A. The topic has no keys in messages

B. The retention.ms is set to 0

C. The broker is running in debug mode

D. The topic has too many partitions

Solution

1. Step 1: Understand log compaction requirement

   Log compaction requires messages to have keys to identify duplicates.

2. Step 2: Analyze why old messages remain

   If messages lack keys, compaction cannot remove duplicates, so old messages stay.

3. Final Answer:

   The topic has no keys in messages -> Option A

4. Quick Check:

   Compaction needs keys to remove old messages [OK]

Hint: Compaction needs keys; no keys means no compaction [OK]

Common Mistakes:

• Assuming retention.ms=0 disables compaction
• Blaming debug mode for compaction issues
• Thinking partition count affects compaction directly

5. You want to store the current state of user profiles in Kafka using log compaction. Which approach is best?

hard

A. Use a topic with cleanup.policy=delete and produce messages without keys

B. Use a topic with retention.ms=0 and produce messages with random keys

C. Use a topic with cleanup.policy=compact and produce messages with user ID as key

D. Use a topic with cleanup.policy=compact but produce messages without keys

Solution

1. Step 1: Identify requirements for storing current state

   Current state needs latest update per user, so keys must be user IDs.

2. Step 2: Choose correct topic configuration

   Log compaction enabled with cleanup.policy=compact keeps latest per key.

3. Step 3: Eliminate wrong options

   Delete policy removes old messages by time, no keys prevent compaction, random keys break state tracking.

4. Final Answer:

   Use a topic with cleanup.policy=compact and produce messages with user ID as key -> Option C

5. Quick Check:

   Compaction + keys = current state storage [OK]

Hint: Use compact policy and keys for current state topics [OK]

Common Mistakes:

• Using delete policy for state storage
• Producing messages without keys
• Using random keys that break compaction logic