Key expiry for memory management in Redis - Deep Dive

Overview - Key expiry for memory management

What is it?

Key expiry in Redis is a feature that allows you to set a time limit on how long a piece of data (a key) should live in memory. After this time passes, Redis automatically deletes the key to free up space. This helps manage memory efficiently by removing data that is no longer needed. It works like a countdown timer attached to each key.

Why it matters

Without key expiry, Redis would keep all data forever, which could fill up memory and slow down or crash your system. Key expiry helps prevent memory overload by cleaning up old or temporary data automatically. This makes Redis reliable and fast, especially for applications that handle lots of changing data like sessions or caches.

Where it fits

Before learning key expiry, you should understand basic Redis data types and commands for storing and retrieving data. After mastering key expiry, you can explore advanced memory management techniques like eviction policies and persistence options to optimize Redis performance.

Mental Model

Core Idea

Key expiry is like setting an automatic timer on data that tells Redis when to delete it to save memory.

Think of it like...

Imagine putting food in a fridge with an expiration date sticker. Once the date passes, you throw the food away to keep the fridge clean and fresh. Key expiry works the same way for data in Redis memory.

┌───────────────┐
│   Redis Key   │
│  ┌─────────┐  │
│  │  Value  │  │
│  └─────────┘  │
│  Expiry Timer │
│  (countdown)  │
└──────┬────────┘
       │
       ▼
  Time passes → Timer hits zero → Key deleted → Memory freed

Build-Up - 6 Steps

1

Foundation

What is Key Expiry in Redis

Concept: Introducing the basic idea that keys can have a time limit before they are removed.

In Redis, you can assign a time-to-live (TTL) to any key. This TTL is the number of seconds or milliseconds after which Redis will delete the key automatically. You set this using commands like EXPIRE or PEXPIRE. If no TTL is set, the key stays forever unless deleted manually.

Result

Keys with TTL will disappear after the set time, freeing memory without manual cleanup.

Understanding that Redis can automatically remove data after a set time helps you manage memory without extra code.

2

Foundation

How to Set and Check Expiry

Concept: Learning the commands to assign and check expiry times on keys.

Use EXPIRE key seconds to set expiry in seconds. Use TTL key to check how many seconds remain before expiry. For example, EXPIRE session123 60 sets the key 'session123' to expire in 60 seconds. TTL session123 returns the remaining time.

Result

You can control and monitor when keys will be deleted.

Knowing these commands lets you control data lifetime precisely and verify expiry settings.

3

Intermediate

Expiry Precision and Milliseconds

🤔Before reading on: do you think Redis expiry can be set with millisecond precision or only seconds? Commit to your answer.

Concept: Redis supports expiry times in milliseconds for finer control.

Besides EXPIRE (seconds), Redis offers PEXPIRE which sets expiry in milliseconds. This is useful when you need very short-lived keys, like caching web requests that expire quickly. PTTL returns the remaining time in milliseconds.

Result

You can set very precise expiry times, down to milliseconds.

Understanding millisecond expiry allows you to optimize performance for fast-changing data.

4

Intermediate

How Expiry Affects Memory Management

🤔Before reading on: does Redis immediately delete expired keys or wait until accessed? Commit to your answer.

Concept: Expired keys free memory automatically, but deletion timing varies.

Redis does not delete expired keys instantly at expiry time. Instead, it deletes them lazily when accessed or during periodic scans. This means expired keys might still use memory briefly after expiry. This lazy deletion balances performance and memory cleanup.

Result

Memory is freed over time as expired keys are accessed or scanned.

Knowing lazy deletion helps you understand why expired keys might linger briefly and how Redis balances speed with cleanup.

5

Advanced

Persistence and Expiry Interaction

🤔Before reading on: do you think expired keys are saved in Redis snapshots or AOF files? Commit to your answer.

Concept: Expiry interacts with Redis persistence mechanisms in subtle ways.

When Redis saves data to disk (RDB snapshots or AOF logs), keys with expiry are saved with their remaining TTL. On restart, Redis restores keys and their TTLs. However, expired keys might still appear briefly after restart until Redis cleans them up. This ensures data consistency but can cause temporary memory use.

Result

Expired keys are preserved with TTL on disk but cleaned after restart.

Understanding persistence interaction prevents surprises with expired keys after Redis restarts.

6

Expert

Internal Expiry Mechanism and Performance

🤔Before reading on: do you think Redis scans all keys continuously to delete expired ones? Commit to your answer.

Concept: Redis uses a combination of passive and active expiry strategies internally.

Redis deletes expired keys in two ways: passive expiry happens when a key is accessed and found expired, it is deleted immediately. Active expiry runs in the background, randomly sampling keys with expiry to delete expired ones. This avoids scanning all keys continuously, which would hurt performance. The sampling rate adapts to system load.

Result

Expired keys are removed efficiently without slowing Redis.

Knowing Redis expiry uses smart sampling explains how it balances memory cleanup with high performance.

Under the Hood

Redis stores expiry times as timestamps alongside keys internally. When a key is accessed, Redis checks the current time against the expiry timestamp to decide if the key is expired. For keys not accessed, Redis runs a background task that randomly samples keys with expiry to delete expired ones. This avoids scanning the entire dataset, keeping Redis fast. Expiry timestamps are stored in a separate dictionary for quick lookup.

Why designed this way?

The design balances memory cleanup with speed. Immediate deletion of all expired keys would require scanning all keys constantly, which is slow. Lazy deletion on access plus random sampling reduces CPU load and avoids latency spikes. This approach was chosen to keep Redis extremely fast and responsive even with millions of keys.

┌───────────────┐       ┌───────────────┐
│   Key Store   │──────▶│ Expiry Times  │
│ (keys+values) │       │ (timestamps)  │
└──────┬────────┘       └──────┬────────┘
       │                       │
       │ Access key            │
       ▼                       ▼
  Check expiry?           Background
       │                 random sampling
       ▼                       │
  If expired → delete ◀───────┘
       │
       ▼
  Return value or nil

Myth Busters - 4 Common Misconceptions

Quick: Does Redis delete expired keys exactly at the expiry time or sometime later? Commit to your answer.

Common Belief: Redis deletes keys immediately the moment they expire.

Reality: Redis deletes expired keys lazily when accessed or during periodic background scans, not exactly at expiry time.

Why it matters: Expecting immediate deletion can cause confusion when expired keys still appear briefly, leading to wrong assumptions about memory usage.

Quick: If a key has no expiry set, will Redis ever delete it automatically? Commit to your answer.

Common Belief: Keys without expiry will be deleted automatically when memory is low.

Reality: Keys without expiry stay forever unless deleted manually or evicted by memory policies, but expiry itself does not apply.

Why it matters: Assuming keys without expiry auto-delete can cause memory to fill unexpectedly, leading to crashes.

Quick: Are expired keys saved in Redis persistence files? Commit to your answer.

Common Belief: Expired keys are never saved to disk because they are deleted immediately.

Reality: Expired keys with remaining TTL are saved in persistence files and cleaned up after restart.

Why it matters: Not knowing this can cause confusion when expired keys reappear after restart, affecting memory planning.

Quick: Does setting expiry on a key guarantee it will be deleted exactly after that time? Commit to your answer.

Common Belief: Expiry guarantees exact deletion timing down to the second or millisecond.

Reality: Expiry sets a minimum lifetime; actual deletion can be delayed due to lazy and active expiry strategies.

Why it matters: Relying on exact expiry timing can cause bugs in time-sensitive applications.

Expert Zone

1

Expiry times are stored separately from keys, allowing quick expiry checks without scanning full data.

2

Active expiry sampling rate adapts dynamically based on system load to avoid performance degradation.

3

Setting expiry on volatile keys can interact with eviction policies, affecting which keys Redis removes under memory pressure.

When NOT to use

Avoid relying solely on key expiry for critical data deletion guarantees. For guaranteed deletion, use explicit delete commands or combine with Redis streams or external cleanup jobs. Also, for very large datasets with complex expiry needs, consider external cache layers or databases with built-in TTL support.

Production Patterns

In production, key expiry is widely used for session management, caching, rate limiting, and temporary data storage. Combining expiry with eviction policies ensures Redis memory stays within limits. Monitoring TTL distributions helps optimize expiry settings and avoid memory bloat.

Connections

Garbage Collection in Programming

Both manage memory by removing unused data automatically.

Understanding key expiry as a form of garbage collection helps grasp how Redis frees memory without manual intervention.

Cache Invalidation

Key expiry is a method of cache invalidation by removing stale data after a set time.

Knowing expiry helps understand how caches stay fresh and avoid serving outdated information.

Perishable Goods Management

Expiry in Redis parallels managing perishable goods by discarding expired items to maintain quality.

This cross-domain view highlights the importance of timely cleanup to keep systems efficient and reliable.

Common Pitfalls

#1 Setting expiry but expecting keys to be deleted immediately at expiry time.

Wrong approach:

EXPIRE session123 60
// Expect session123 to disappear exactly after 60 seconds

Correct approach:

EXPIRE session123 60
// Understand key may persist briefly after 60 seconds until accessed or scanned

Root cause:

Misunderstanding Redis lazy and active expiry mechanisms causes wrong expectations about deletion timing.

#2 Not setting expiry on temporary data, causing memory to fill up.

Wrong approach:

SET cache_item 'data'
// No expiry set, key stays forever

Correct approach:

SET cache_item 'data'
EXPIRE cache_item 300
// Key expires after 5 minutes, freeing memory

Root cause:

Forgetting to set TTL on temporary keys leads to memory leaks.

#3 Assuming expired keys are never saved to disk.

Wrong approach:

Relying on expired keys not appearing after restart without cleanup

Correct approach:

Understand expired keys with TTL are saved and cleaned after restart

Root cause:

Lack of knowledge about persistence and expiry interaction causes unexpected key presence.

Key Takeaways

Key expiry in Redis automatically removes data after a set time to manage memory efficiently.

Expiry times can be set in seconds or milliseconds, allowing precise control over data lifetime.

Redis deletes expired keys lazily on access and actively via background sampling to balance performance and cleanup.

Expired keys are saved with their TTL in persistence files and cleaned after restart, which can cause temporary memory use.

Understanding expiry mechanisms helps prevent common mistakes and optimize Redis memory management in real-world applications.