How to Set Retention for Topic in Kafka: Simple Guide

To set retention for a Kafka topic, use the kafka-configs.sh tool with the --alter option and specify retention.ms or retention.bytes. For example, kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name your_topic --add-config retention.ms=60000 sets retention to 1 minute.

Syntax

The retention settings for a Kafka topic can be configured using the kafka-configs.sh command-line tool or by setting properties when creating the topic.

• retention.ms: Time in milliseconds to keep a log segment before deletion.
• retention.bytes: Maximum size in bytes for the log before deletion.

Using kafka-configs.sh, the syntax to alter retention is:

bash

kafka-configs.sh --bootstrap-server <broker_host>:<port> --alter --entity-type topics --entity-name <topic_name> --add-config retention.ms=<milliseconds>,retention.bytes=<bytes>

Example

This example sets the retention time of the topic my_topic to 2 hours (7200000 milliseconds) using the Kafka command-line tool.

bash

kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name my_topic --add-config retention.ms=7200000

Output

Updated configs for topic 'my_topic'.

Common Pitfalls

• Forgetting to specify the correct --bootstrap-server or --zookeeper parameter depending on your Kafka version.
• Using retention.ms and retention.bytes together without understanding which limit triggers deletion first.
• Not restarting Kafka brokers when changing retention settings in server config files (if not using dynamic configs).
• Setting retention too low can cause data loss; too high can cause disk space issues.

bash

Wrong way:
kafka-configs.sh --alter --entity-type topics --entity-name my_topic --add-config retention.ms=60000

Right way:
kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name my_topic --add-config retention.ms=60000

Quick Reference

Summary of key retention configuration properties for Kafka topics:

Property	Description	Default Value
retention.ms	Time in milliseconds to retain logs	604800000 (7 days)
retention.bytes	Maximum size in bytes to retain logs	-1 (no size limit)
log.retention.check.interval.ms	Interval to check retention policies	300000 (5 minutes)

Key Takeaways

Use kafka-configs.sh with --alter and --add-config to set topic retention dynamically.

retention.ms controls time-based retention; retention.bytes controls size-based retention.

Always specify --bootstrap-server with the correct broker address for the command to work.

Beware of setting retention too low to avoid unintended data loss.

Check Kafka broker logs and disk usage after changing retention settings.