Retention policies (time-based, size-based) in Kafka - Mini Project: Build & Apply

Kafka Retention Policies: Time-based and Size-based

📖 Scenario: You are managing a Kafka topic that collects user activity logs. To keep the storage efficient, you want to apply retention policies that automatically delete old messages based on time and size.

🎯 Goal: Configure a Kafka topic with both time-based and size-based retention policies, then verify the configuration by printing the topic settings.

📋 What You'll Learn

Create a Kafka topic configuration dictionary with default settings

Add a time-based retention policy of 7 days (in milliseconds)

Add a size-based retention policy of 1 GB (in bytes)

Print the final topic configuration dictionary

💡 Why This Matters

🌍 Real World

Kafka topics often collect large amounts of data. Retention policies help manage storage by deleting old or excess data automatically.

💼 Career

Understanding Kafka retention policies is important for roles like data engineers, backend developers, and system administrators who manage data pipelines and streaming platforms.

1

Create initial Kafka topic configuration

Create a dictionary called topic_config with these exact entries: 'cleanup.policy': 'delete' and 'segment.ms': 604800000 (7 days in milliseconds).

Kafka

# Create the initial topic_config dictionary with cleanup.policy and segment.ms
# Your code here

Need a hint?

Use a dictionary with keys 'cleanup.policy' and 'segment.ms'. The value for 'segment.ms' is 7 days in milliseconds (7 * 24 * 60 * 60 * 1000).

2

Add time-based retention policy

Add a new entry to topic_config with key 'retention.ms' and value 604800000 (7 days in milliseconds).

Kafka

topic_config = {
    'cleanup.policy': 'delete',
    'segment.ms': 604800000
}
# Add 'retention.ms' with value 604800000 to topic_config
# Your code here

Need a hint?

Use dictionary key assignment to add 'retention.ms' with the value 604800000.

3

Add size-based retention policy

Add a new entry to topic_config with key 'retention.bytes' and value 1073741824 (1 GB in bytes).

Kafka

topic_config = {
    'cleanup.policy': 'delete',
    'segment.ms': 604800000
}
topic_config['retention.ms'] = 604800000
# Add 'retention.bytes' with value 1073741824 to topic_config
# Your code here

Need a hint?

Use dictionary key assignment to add 'retention.bytes' with the value 1073741824.

4

Print the final topic configuration

Write print(topic_config) to display the final topic configuration dictionary.

Kafka

topic_config = {
    'cleanup.policy': 'delete',
    'segment.ms': 604800000
}
topic_config['retention.ms'] = 604800000
topic_config['retention.bytes'] = 1073741824
# Print the topic_config dictionary
# Your code here

Need a hint?

Use the print function to show the dictionary.