Retention policies (time-based, size-based) in Kafka - Step-by-Step Execution

Process Flow - Retention policies (time-based, size-based)

Message Produced

↓

Stored in Partition

↓

Check Retention Policy

↓

Time-based?

↓

Delete if older than configured time

↓

Size-based?

↓

Delete oldest if size limit exceeded

↓

Message Retained or Deleted

Messages are stored in partitions. Kafka checks if messages exceed time or size limits and deletes old messages accordingly.

Execution Sample

Kafka

topic.retention.ms=60000
# 1 minute retention time

topic.retention.bytes=1048576
# 1 MB max size

This config sets a topic to keep messages for 1 minute or until size exceeds 1 MB.

Process Table

Step	Action	Message Timestamp	Partition Size (bytes)	Retention Check	Result
1	Message produced	0 ms	500000	No deletion	Message stored
2	Message produced	30000 ms	900000	No deletion	Message stored
3	Message produced	61000 ms	1200000	Time check: 61000 > 60000	Oldest message deleted
4	Retention size check	N/A	1100000	Size check: 1100000 > 1048576	Oldest message deleted
5	Retention time check	120000 ms	900000	Time check: 120000 > 60000	Oldest message deleted
6	Retention size check	N/A	700000	Size check: 700000 < 1048576	No deletion
7	No new messages	N/A	700000	No retention triggers	Partition stable

💡 No new messages and partition size/time within limits, retention stops.

Status Tracker

Variable	Start	After 1	After 2	After 3	After 4	After 5	After 6	Final
Partition Size (bytes)	0	500000	900000	1200000	1100000	900000	700000	700000
Oldest Message Timestamp (ms)	0	0	0	30000	30000	61000	120000	120000

Key Moments - 2 Insights

Why does Kafka delete messages at step 3 even though the partition size is below the size limit?

At step 3, the message timestamp exceeds the time-based retention limit (61000 ms > 60000 ms), so Kafka deletes the oldest message regardless of size, as shown in the execution_table row 3.

Why is there a size-based deletion at step 4 after time-based deletion?

After deleting old messages by time, the partition size is still above the size limit (1100000 > 1048576), so Kafka deletes oldest messages to reduce size, as seen in execution_table row 4.

Visual Quiz - 3 Questions

Look at the execution table, what is the partition size after step 2?

A 900000 bytes

B 500000 bytes

C 1200000 bytes

D 700000 bytes

💡 Hint

Check the 'Partition Size (bytes)' column at step 2 in the execution_table.

At which step does the time-based retention policy first delete messages?

A Step 2

B Step 4

C Step 3

D Step 5

💡 Hint

Look for 'Time check' in the 'Retention Check' column in the execution_table.

If the retention time was increased to 2 minutes, what would happen at step 3?

A Oldest message deleted due to time

B No deletion due to time, only size-based deletion if needed

C Partition size would reset to zero

D All messages deleted

💡 Hint

Refer to the time-based retention check at step 3 and consider the new retention time.

Concept Snapshot

Retention policies in Kafka control how long or how much data is kept.
Time-based retention deletes messages older than a set time (e.g., 1 minute).
Size-based retention deletes oldest messages if partition size exceeds a limit (e.g., 1 MB).
Kafka checks both policies regularly and deletes messages accordingly.
This helps manage storage and keep data fresh.

Full Transcript

Kafka stores messages in partitions. Each message has a timestamp and size. Kafka uses retention policies to decide when to delete old messages. Time-based retention deletes messages older than a configured time, like 1 minute. Size-based retention deletes oldest messages if the partition size grows beyond a set limit, like 1 MB. In the example, messages are produced with timestamps and sizes. At step 3, the time-based policy deletes messages older than 1 minute. At step 4, size-based policy deletes messages because the partition size is too big. This process repeats to keep data within limits. Understanding these policies helps manage Kafka storage efficiently.

Practice

1. What does the retention.ms setting control in Kafka?

easy

A. The maximum size of a Kafka topic

B. How long Kafka keeps data before deleting it

C. The number of partitions in a topic

D. The replication factor of a topic

Solution

1. Step 1: Understand retention.ms meaning

   retention.ms sets the time duration Kafka retains data before deletion.

2. Step 2: Compare with other options

   Other options relate to size, partitions, or replication, not time retention.

3. Final Answer:

   How long Kafka keeps data before deleting it -> Option B

4. Quick Check:

   retention.ms = time-based retention [OK]

Hint: Remember ms means milliseconds, so it's about time [OK]

Common Mistakes:

• Confusing retention.ms with retention.bytes
• Thinking it controls topic size
• Mixing it with partition count

2. Which of the following is the correct way to set a size-based retention policy in Kafka's topic configuration?

easy

A. "retention.bytes" = 1073741824

B. "retention.time" = 604800000

C. "retention.size" = 1073741824

D. "retention.ms" = 1073741824

Solution

1. Step 1: Identify correct property for size-based retention

   The correct property is retention.bytes for size limits.

2. Step 2: Check option values and names

   "retention.bytes" = 1073741824 uses retention.bytes with a size value (1GB). Others use wrong keys or time-based keys.

3. Final Answer:

   "retention.bytes" = 1073741824 -> Option A

4. Quick Check:

   Size retention uses retention.bytes [OK]

Hint: Size retention uses bytes, so look for retention.bytes [OK]

Common Mistakes:

• Using retention.time instead of retention.ms
• Using retention.size which is invalid
• Confusing retention.ms with size setting

3. Given a Kafka topic with retention.ms=60000 and retention.bytes=1048576, what happens when the topic data reaches 2MB before 60 seconds pass?

medium

A. Kafka waits until 60 seconds pass before deleting any data

B. Kafka deletes data only if both time and size limits are reached

C. Kafka deletes old data immediately because size limit is reached

D. Kafka ignores retention settings and keeps all data

Solution

1. Step 1: Understand retention.ms and retention.bytes interaction

   Kafka deletes data when either time or size limit is reached, whichever comes first.

2. Step 2: Analyze given values

   Size limit is 1MB, data reached 2MB before 60 seconds, so size limit triggers deletion immediately.

3. Final Answer:

   Kafka deletes old data immediately because size limit is reached -> Option C

4. Quick Check:

   Retention triggers on earliest limit reached [OK]

Hint: Retention deletes when time OR size limit is reached [OK]

Common Mistakes:

• Thinking both limits must be reached to delete
• Assuming time limit always overrides size
• Ignoring size limit when time not reached

4. You set retention.ms to a negative value in Kafka topic config. What is the likely outcome?

medium

A. Kafka treats it as infinite retention (never deletes data)

B. Kafka throws a configuration error and refuses to start

C. Kafka deletes data immediately after creation

D. Kafka uses default retention time instead

Solution

1. Step 1: Check Kafka behavior for negative retention.ms

   Kafka interprets negative retention.ms as infinite retention, meaning no data deletion.

2. Step 2: Confirm other options

   Kafka does not throw error or delete immediately; it does not revert to default automatically.

3. Final Answer:

   Kafka treats it as infinite retention (never deletes data) -> Option A

4. Quick Check:

   Negative retention.ms means infinite retention [OK]

Hint: Negative retention.ms disables deletion, keeps data forever [OK]

Common Mistakes:

• Expecting an error on negative value
• Assuming immediate deletion
• Thinking default value applies automatically

5. You want to configure a Kafka topic to keep data for 2 hours or up to 500MB, whichever limit is reached first. Which configuration is correct?

hard

A. "retention.ms"=7200000 only, size limit is not needed

B. "retention.ms"=7200000 and "retention.bytes"=5000000000

C. "retention.ms"=120 and "retention.bytes"=524288000

D. "retention.ms"=7200000 and "retention.bytes"=524288000

Solution

1. Step 1: Convert 2 hours to milliseconds

   2 hours = 2 * 60 * 60 * 1000 = 7,200,000 ms.

2. Step 2: Convert 500MB to bytes

   500MB = 500 * 1024 * 1024 = 524,288,000 bytes.

3. Step 3: Match values to options

   "retention.ms"=7200000 and "retention.bytes"=524288000 matches both correct values; others have wrong size or time values.

4. Final Answer:

   "retention.ms"=7200000 and "retention.bytes"=524288000 -> Option D

5. Quick Check:

   2 hours = 7200000 ms, 500MB = 524288000 bytes [OK]

Hint: Convert hours to ms and MB to bytes carefully [OK]

Common Mistakes:

• Using seconds instead of milliseconds for retention.ms
• Confusing MB with bytes without conversion
• Setting only one limit when both are needed