How to Set Retention Size in Kafka: Configuration Guide

To set retention size in Kafka, use the retention.bytes property in the topic configuration. This property limits the total size of log data retained for a topic, deleting older data when the size exceeds the limit.

Syntax

The retention.bytes property defines the maximum size in bytes for the log data retained per topic partition. When the log size exceeds this value, Kafka deletes the oldest segments to free space.

It can be set at the topic level or broker level (default for all topics).

properties

retention.bytes=<size_in_bytes>

Example

This example shows how to set the retention size to 100 MB for a Kafka topic named my-topic using the Kafka command-line tool.

bash

kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic --alter --add-config retention.bytes=104857600

Output

Updated configs for topic 'my-topic'.

Common Pitfalls

• Setting retention.bytes too low can cause frequent data deletion, losing important messages.
• Not setting retention.bytes at the topic level means the broker default applies, which might not fit your needs.
• Confusing retention.bytes with retention.ms which controls retention by time, not size.

bash

Wrong:
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic --alter --add-config retention.bytes=100

Right:
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic --alter --add-config retention.bytes=104857600

Quick Reference

Property	Description	Example Value
retention.bytes	Max size in bytes to retain per partition	104857600 (100 MB)
retention.ms	Max time to retain data (alternative)	604800000 (7 days)
log.segment.bytes	Size of each log segment file	1073741824 (1 GB)

Key Takeaways

Use retention.bytes to limit Kafka topic data size by bytes.

Set retention size per topic with kafka-configs.sh or in topic creation configs.

Beware of setting too low retention size to avoid unwanted data loss.

Retention size controls storage, different from retention time (retention.ms).

Check broker defaults if topic-level retention size is not set.