Log compaction in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of the Kafka log compaction status check?

Given the following Kafka topic configuration and status check command, what will be the output?

Kafka

kafka-topics.sh --describe --topic user-updates --bootstrap-server localhost:9092

Topic: user-updates	PartitionCount: 3	ReplicationFactor: 2	Configs: cleanup.policy=compact
Topic: user-updates	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2
Topic: user-updates	Partition: 1	Leader: 2	Replicas: 2,3	Isr: 2,3
Topic: user-updates	Partition: 2	Leader: 3	Replicas: 3,1	Isr: 3,1

A The topic 'user-updates' has log compaction enabled with 3 partitions and replication factor 2.

B The topic 'user-updates' has log compaction disabled and 3 partitions.

C The topic 'user-updates' has 1 partition with log compaction enabled.

D The topic 'user-updates' has 3 partitions but replication factor is 1.

💡 Hint

Look for the cleanup.policy configuration in the topic description.

The output shows 'cleanup.policy=compact' which means log compaction is enabled. The topic has 3 partitions and replication factor 2 as shown.

🧠 Conceptual

intermediate

What does Kafka log compaction guarantee?

Choose the correct statement about Kafka log compaction guarantees.

A Log compaction guarantees message order across all partitions.

B Log compaction guarantees that the latest value for each key is retained in the topic.

C Log compaction compresses messages to reduce disk usage without deleting any data.

D Log compaction deletes all messages older than a configured time.

💡 Hint

Think about what happens to messages with the same key after compaction.

Log compaction keeps the latest message for each key, removing older duplicates, but does not delete messages based on time or compress data.

❓ Predict Output

advanced

What error occurs when enabling log compaction with invalid cleanup.policy?

What error will Kafka produce if you set cleanup.policy=compacted instead of compact for a topic?

Kafka

kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1 --config cleanup.policy=compacted

A Error: Replication factor must be greater than 1 for compaction

B Topic created successfully with log compaction enabled

C Warning: Unknown cleanup.policy value, defaulting to delete

D Error: Invalid value 'compacted' for configuration 'cleanup.policy'

💡 Hint

Check the exact allowed values for cleanup.policy.

Kafka only accepts 'compact', 'delete', or both combined. 'compacted' is invalid and causes an error.

🚀 Application

advanced

How to configure a topic for both log compaction and time-based deletion?

Which configuration correctly enables both log compaction and time-based deletion on a Kafka topic?

A cleanup.policy=compact delete

B cleanup.policy=compact;delete

C cleanup.policy=compact,delete

D cleanup.policy=compact|delete

💡 Hint

Multiple policies are comma-separated in Kafka configs.

Kafka expects multiple cleanup policies separated by commas without spaces.

🧠 Conceptual

expert

Why might Kafka log compaction not immediately remove old records?

Select the best explanation why Kafka log compaction does not instantly delete old records after a new record with the same key is written.

A Compaction runs periodically and only removes older duplicates during background cleanup, not immediately.

B Kafka keeps all records forever regardless of compaction settings.

C Compaction requires manual trigger to delete old records.

D Old records are deleted immediately but only after consumer acknowledges.

💡 Hint

Think about how Kafka manages background tasks.

Kafka log compaction is a background process that runs periodically, so old duplicates remain until compaction runs.

Practice

1. What is the main purpose of log compaction in Kafka?

easy

A. To compress messages to save disk space

B. To delete all messages older than a certain time

C. To replicate messages across multiple brokers

D. To keep only the latest message for each key in a topic

Solution

1. Step 1: Understand log compaction concept

   Log compaction keeps the latest message per key, removing older duplicates.

2. Step 2: Compare with other options

   Deleting old messages is retention, compression is different, replication is for fault tolerance.

3. Final Answer:

   To keep only the latest message for each key in a topic -> Option D

4. Quick Check:

   Log compaction = latest message per key [OK]

Hint: Log compaction keeps latest per key, not all old messages [OK]

Common Mistakes:

• Confusing log compaction with time-based retention
• Thinking compression means log compaction
• Mixing replication with compaction

2. Which configuration enables log compaction on a Kafka topic?

easy

A. cleanup.policy=delete

B. cleanup.policy=compact

C. log.compaction.enabled=true

D. retention.ms=0

Solution

1. Step 1: Identify correct cleanup policy

   Log compaction is enabled by setting cleanup.policy=compact on the topic.

2. Step 2: Eliminate incorrect options

   cleanup.policy=delete is default retention, log.compaction.enabled is invalid, retention.ms controls time retention.

3. Final Answer:

   cleanup.policy=compact -> Option B

4. Quick Check:

   Enable compaction with cleanup.policy=compact [OK]

Hint: Remember: compact, not delete, enables log compaction [OK]

Common Mistakes:

• Using cleanup.policy=delete instead of compact
• Assuming a separate 'log.compaction.enabled' setting exists
• Confusing retention.ms with compaction setting

3. Given a compacted Kafka topic with keys and values:
key1: v1, key2: v2, key1: v3, key3: v4
What messages remain after log compaction?

medium

A. key1: v3, key2: v2, key3: v4

B. key1: v1, key2: v2, key3: v4

C. key1: v3, key2: v2

D. key2: v2, key3: v4

Solution

1. Step 1: Identify latest message per key

   For key1, latest is v3; key2 is v2; key3 is v4.

2. Step 2: List messages after compaction

   Only latest per key remain: key1:v3, key2:v2, key3:v4.

3. Final Answer:

   key1: v3, key2: v2, key3: v4 -> Option A

4. Quick Check:

   Latest message per key kept [OK]

Hint: Keep only last message per key after compaction [OK]

Common Mistakes:

• Keeping all messages instead of only latest per key
• Confusing order and keeping first message per key
• Dropping keys incorrectly

4. You set cleanup.policy=compact on a topic but notice old messages are not removed. What is a likely cause?

medium

A. The topic has no keys in messages

B. The retention.ms is set to 0

C. The broker is running in debug mode

D. The topic has too many partitions

Solution

1. Step 1: Understand log compaction requirement

   Log compaction requires messages to have keys to identify duplicates.

2. Step 2: Analyze why old messages remain

   If messages lack keys, compaction cannot remove duplicates, so old messages stay.

3. Final Answer:

   The topic has no keys in messages -> Option A

4. Quick Check:

   Compaction needs keys to remove old messages [OK]

Hint: Compaction needs keys; no keys means no compaction [OK]

Common Mistakes:

• Assuming retention.ms=0 disables compaction
• Blaming debug mode for compaction issues
• Thinking partition count affects compaction directly

5. You want to store the current state of user profiles in Kafka using log compaction. Which approach is best?

hard

A. Use a topic with cleanup.policy=delete and produce messages without keys

B. Use a topic with retention.ms=0 and produce messages with random keys

C. Use a topic with cleanup.policy=compact and produce messages with user ID as key

D. Use a topic with cleanup.policy=compact but produce messages without keys

Solution

1. Step 1: Identify requirements for storing current state

   Current state needs latest update per user, so keys must be user IDs.

2. Step 2: Choose correct topic configuration

   Log compaction enabled with cleanup.policy=compact keeps latest per key.

3. Step 3: Eliminate wrong options

   Delete policy removes old messages by time, no keys prevent compaction, random keys break state tracking.

4. Final Answer:

   Use a topic with cleanup.policy=compact and produce messages with user ID as key -> Option C

5. Quick Check:

   Compaction + keys = current state storage [OK]

Hint: Use compact policy and keys for current state topics [OK]

Common Mistakes:

• Using delete policy for state storage
• Producing messages without keys
• Using random keys that break compaction logic