How to Set Retention Time in Kafka: Simple Guide

To set the retention time in Kafka, use the retention.ms configuration on a topic, which defines how long Kafka keeps messages before deleting them. You can set it when creating a topic or update it later using the kafka-configs.sh tool or Kafka Admin API.

Syntax

The retention.ms property controls the retention time in milliseconds for messages in a Kafka topic.

• retention.ms: Time in milliseconds to keep a log segment before deletion.
• Set it to -1 to keep data indefinitely.
• Can be set per topic or globally in broker config.

bash

kafka-topics.sh --create --topic <topic-name> --bootstrap-server <broker-address> --config retention.ms=<milliseconds>

Example

This example creates a Kafka topic named my-topic with a retention time of 1 hour (3600000 milliseconds).

bash

kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1 --config retention.ms=3600000

Output

Created topic my-topic.

Common Pitfalls

Common mistakes when setting retention time include:

• Setting retention time too low, causing data loss.
• Forgetting to specify --config retention.ms when updating topic configs.
• Confusing retention.ms with retention.bytes, which limits size, not time.
• Not restarting Kafka broker when changing global retention settings.

bash

## Wrong: Trying to update retention without --config
kafka-topics.sh --alter --topic my-topic --bootstrap-server localhost:9092 --retention.ms 3600000

## Right: Use --config to update retention
kafka-topics.sh --alter --topic my-topic --bootstrap-server localhost:9092 --config retention.ms=3600000

Quick Reference

Command	Description
kafka-topics.sh --create --topic --config retention.ms=	Create topic with retention time
kafka-topics.sh --alter --topic --config retention.ms=	Update retention time of existing topic
retention.ms = -1	Keep messages indefinitely
Default retention.ms	Usually 7 days (604800000 ms) unless changed

Key Takeaways

Use the retention.ms property to set how long Kafka keeps messages in a topic.

Set retention.ms in milliseconds; for example, 3600000 ms equals 1 hour.

Update retention.ms using kafka-topics.sh with the --config option.

Setting retention.ms to -1 disables automatic deletion, keeping data forever.

Be careful not to confuse retention.ms (time) with retention.bytes (size).