Retention policies (time-based, size-based) in Kafka - Step-by-Step Execution

Process Flow - Retention policies (time-based, size-based)

Message Produced

↓

Stored in Partition

↓

Check Retention Policy

↓

Time-based?

↓

Delete if older than configured time

↓

Size-based?

↓

Delete oldest if size limit exceeded

↓

Message Retained or Deleted

Messages are stored in partitions. Kafka checks if messages exceed time or size limits and deletes old messages accordingly.

Execution Sample

Kafka

topic.retention.ms=60000
# 1 minute retention time

topic.retention.bytes=1048576
# 1 MB max size

This config sets a topic to keep messages for 1 minute or until size exceeds 1 MB.

Process Table

Step	Action	Message Timestamp	Partition Size (bytes)	Retention Check	Result
1	Message produced	0 ms	500000	No deletion	Message stored
2	Message produced	30000 ms	900000	No deletion	Message stored
3	Message produced	61000 ms	1200000	Time check: 61000 > 60000	Oldest message deleted
4	Retention size check	N/A	1100000	Size check: 1100000 > 1048576	Oldest message deleted
5	Retention time check	120000 ms	900000	Time check: 120000 > 60000	Oldest message deleted
6	Retention size check	N/A	700000	Size check: 700000 < 1048576	No deletion
7	No new messages	N/A	700000	No retention triggers	Partition stable

💡 No new messages and partition size/time within limits, retention stops.

Status Tracker

Variable	Start	After 1	After 2	After 3	After 4	After 5	After 6	Final
Partition Size (bytes)	0	500000	900000	1200000	1100000	900000	700000	700000
Oldest Message Timestamp (ms)	0	0	0	30000	30000	61000	120000	120000

Key Moments - 2 Insights

Why does Kafka delete messages at step 3 even though the partition size is below the size limit?

At step 3, the message timestamp exceeds the time-based retention limit (61000 ms > 60000 ms), so Kafka deletes the oldest message regardless of size, as shown in the execution_table row 3.

Why is there a size-based deletion at step 4 after time-based deletion?

After deleting old messages by time, the partition size is still above the size limit (1100000 > 1048576), so Kafka deletes oldest messages to reduce size, as seen in execution_table row 4.

Visual Quiz - 3 Questions

Look at the execution table, what is the partition size after step 2?

A 900000 bytes

B 500000 bytes

C 1200000 bytes

D 700000 bytes

💡 Hint

Check the 'Partition Size (bytes)' column at step 2 in the execution_table.

At which step does the time-based retention policy first delete messages?

A Step 2

B Step 4

C Step 3

D Step 5

💡 Hint

Look for 'Time check' in the 'Retention Check' column in the execution_table.

If the retention time was increased to 2 minutes, what would happen at step 3?

A Oldest message deleted due to time

B No deletion due to time, only size-based deletion if needed

C Partition size would reset to zero

D All messages deleted

💡 Hint

Refer to the time-based retention check at step 3 and consider the new retention time.

Concept Snapshot

Retention policies in Kafka control how long or how much data is kept.
Time-based retention deletes messages older than a set time (e.g., 1 minute).
Size-based retention deletes oldest messages if partition size exceeds a limit (e.g., 1 MB).
Kafka checks both policies regularly and deletes messages accordingly.
This helps manage storage and keep data fresh.

Full Transcript

Kafka stores messages in partitions. Each message has a timestamp and size. Kafka uses retention policies to decide when to delete old messages. Time-based retention deletes messages older than a configured time, like 1 minute. Size-based retention deletes oldest messages if the partition size grows beyond a set limit, like 1 MB. In the example, messages are produced with timestamps and sizes. At step 3, the time-based policy deletes messages older than 1 minute. At step 4, size-based policy deletes messages because the partition size is too big. This process repeats to keep data within limits. Understanding these policies helps manage Kafka storage efficiently.