Maxmemory setting in Redis - Deep Dive

Overview - Maxmemory setting

What is it?

The maxmemory setting in Redis controls the maximum amount of memory the Redis server can use. When Redis reaches this limit, it uses a policy to decide which data to remove to free up space. This setting helps manage memory usage and keeps Redis from using too much memory on a system.

Why it matters

Without maxmemory, Redis could use all available memory on a server, causing the system to slow down or crash. This setting protects the server by limiting Redis's memory use and ensures Redis continues to work smoothly by removing old or less important data when needed.

Where it fits

Before learning maxmemory, you should understand basic Redis concepts like keys, values, and how Redis stores data in memory. After maxmemory, you can learn about eviction policies and persistence to manage data durability and memory efficiently.

Mental Model

Core Idea

Maxmemory sets a memory limit for Redis, and when reached, Redis removes data based on a chosen policy to stay within that limit.

Think of it like...

Imagine a backpack with a fixed size. You can only carry so much inside it. When it gets full, you have to decide what to take out before adding new items. Maxmemory is like the backpack size, and eviction policies decide what to remove.

┌───────────────┐
│   Redis RAM   │
│  Maxmemory    │
│  Limit Set   │
├───────────────┤
│ Data Stored   │
│ (Keys/Values) │
├───────────────┤
│ When Full →   │
│ Eviction      │
│ Policy Runs   │
└───────────────┘

Build-Up - 7 Steps

1

Foundation

What is Maxmemory in Redis

Concept: Introducing the maxmemory setting as a limit on Redis memory usage.

Redis stores all data in memory for fast access. Maxmemory is a configuration that sets the maximum memory Redis can use. If Redis tries to use more than this limit, it triggers actions to free memory.

Result

Redis will not exceed the set memory limit, protecting the server from running out of memory.

Understanding maxmemory is key to controlling Redis's memory footprint and preventing system crashes.

2

Foundation

How Redis Uses Memory

Concept: Explaining that Redis keeps all data in RAM and why memory management is critical.

Redis keeps data in RAM for speed. Unlike databases that store data on disk, Redis's speed depends on fitting data in memory. Without limits, Redis can consume all RAM, affecting other programs.

Result

Learners see why memory limits like maxmemory are necessary for stable Redis operation.

Knowing Redis is an in-memory database clarifies why maxmemory is a fundamental setting.

3

Intermediate

Configuring Maxmemory Setting

🤔Before reading on: Do you think maxmemory is set in bytes, megabytes, or as a percentage? Commit to your answer.

Concept: How to set maxmemory in Redis configuration using different units.

You can set maxmemory in the redis.conf file or at runtime with commands. It accepts bytes or human-readable units like 'mb' for megabytes. For example, maxmemory 100mb limits Redis to 100 megabytes of RAM.

Result

Redis enforces the memory limit you set, measured in bytes or convenient units.

Knowing how to configure maxmemory precisely lets you tailor Redis memory use to your server's capacity.

4

Intermediate

Eviction Policies When Maxmemory Reached

🤔Before reading on: Do you think Redis stops accepting writes when maxmemory is full, or does it remove some data? Commit to your answer.

Concept: Introducing eviction policies that decide what data Redis removes when maxmemory is reached.

When Redis hits maxmemory, it uses an eviction policy to free space. Policies include removing least recently used keys (LRU), least frequently used (LFU), or no eviction (reject writes). You choose the policy based on your needs.

Result

Redis continues working by removing data according to the chosen policy, preventing crashes.

Understanding eviction policies is crucial to controlling data loss and performance under memory pressure.

5

Intermediate

Noeviction Policy and Its Effects

Concept: Explaining the noeviction policy where Redis refuses writes when maxmemory is reached.

If you set maxmemory-policy to noeviction, Redis will reject commands that add data once the limit is reached. This protects existing data but can cause errors in your application if not handled.

Result

Redis stops accepting new data when memory is full, preserving current data but risking write failures.

Knowing noeviction helps you decide if data safety or availability is more important for your use case.

6

Advanced

Memory Fragmentation and Maxmemory

🤔Before reading on: Do you think maxmemory counts only used data memory or also includes fragmentation? Commit to your answer.

Concept: Memory fragmentation can cause Redis to use more memory than expected, affecting maxmemory enforcement.

Memory fragmentation happens when free memory is split into small pieces, making it hard to allocate large blocks. Redis's maxmemory counts total allocated memory including fragmentation, so actual usable memory can be less than maxmemory.

Result

Redis may hit maxmemory sooner than expected due to fragmentation, triggering eviction or errors.

Understanding fragmentation helps in tuning maxmemory and interpreting memory usage metrics correctly.

7

Expert

Maxmemory and Persistence Trade-offs

🤔Before reading on: Does maxmemory affect Redis persistence files on disk? Commit to your answer.

Concept: How maxmemory interacts with Redis persistence mechanisms like RDB and AOF.

Maxmemory limits RAM usage but does not limit disk space used for persistence. However, eviction can cause data loss before persistence saves. Also, persistence files can grow large independently, so monitoring both memory and disk is important.

Result

You learn that maxmemory controls RAM but persistence requires separate management to avoid data loss or disk issues.

Knowing this prevents surprises where data is evicted from memory but still saved on disk, or disk fills up despite memory limits.

Under the Hood

Redis tracks memory usage internally using allocator statistics and counts all memory allocated for keys, values, and overhead. When maxmemory is reached, Redis triggers the eviction process based on the configured policy. It scans keys according to the policy (e.g., LRU) and removes candidates until enough memory is freed. This process is efficient to avoid blocking Redis operations.

Why designed this way?

Redis was designed as an in-memory database prioritizing speed. Memory limits prevent Redis from destabilizing the host system. Eviction policies provide flexible trade-offs between data retention and availability. Alternatives like blocking writes were less flexible, so eviction policies became the standard.

┌───────────────┐
│ Redis Memory  │
│ Usage Track   │
├───────────────┤
│ Maxmemory Set │
├───────────────┤
│ If Usage >    │
│ Maxmemory →   │
│ Eviction Run  │
├───────────────┤
│ Eviction      │
│ Policy Check  │
├───────────────┤
│ Remove Keys   │
│ Until Enough  │
│ Memory Freed  │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does setting maxmemory guarantee Redis will never use more memory? Commit yes or no.

Common Belief: Setting maxmemory means Redis will never exceed that memory limit.

Reality: Redis can temporarily exceed maxmemory due to memory fragmentation or internal overhead before eviction frees space.

Why it matters: Assuming strict limits can cause unexpected crashes or performance issues if memory usage spikes beyond maxmemory.

Quick: When maxmemory is full, does Redis always delete the oldest data? Commit yes or no.

Common Belief: Redis always removes the oldest data when maxmemory is reached.

Reality: Redis removes data based on the configured eviction policy, which may be least recently used, least frequently used, or other strategies, not necessarily oldest.

Why it matters: Misunderstanding eviction can lead to wrong assumptions about which data will be lost under memory pressure.

Quick: Does maxmemory affect how much disk space Redis uses for persistence? Commit yes or no.

Common Belief: Maxmemory also limits the disk space Redis uses for saving data.

Reality: Maxmemory only limits RAM usage; disk persistence files like RDB or AOF are not limited by maxmemory.

Why it matters: Ignoring disk usage can cause disk space exhaustion even if memory is controlled, leading to system failures.

Quick: If maxmemory-policy is set to noeviction, will Redis accept all writes? Commit yes or no.

Common Belief: With noeviction policy, Redis will keep accepting writes even if memory is full.

Reality: Redis rejects write commands when memory is full under noeviction policy, causing errors.

Why it matters: Not handling write rejections can cause application failures or data loss.

Expert Zone

1

Eviction policies can be combined with volatile or allkeys modes, affecting whether only keys with expiration or all keys are candidates for eviction.

2

Memory fragmentation can vary greatly depending on workload and allocator, so maxmemory tuning requires monitoring fragmentation metrics.

3

Redis's internal memory accounting includes overhead for data structures, so actual usable memory for data is less than maxmemory.

When NOT to use

Maxmemory is not suitable if you need guaranteed data persistence without loss; in such cases, consider Redis in append-only mode with no eviction or use disk-based databases. Also, for workloads with unpredictable memory spikes, external memory management or sharding may be better.

Production Patterns

In production, maxmemory is set based on available system RAM minus overhead for OS and other processes. Eviction policies like allkeys-lru are common for caching use cases. Monitoring tools track memory usage and eviction events to adjust settings dynamically.

Connections

Cache Eviction Algorithms

Maxmemory eviction policies implement cache eviction algorithms like LRU and LFU.

Understanding cache eviction helps grasp how Redis decides which data to remove under memory pressure.

Operating System Memory Management

Maxmemory setting interacts with OS memory limits and fragmentation behavior.

Knowing OS memory management clarifies why Redis memory usage can exceed limits temporarily and how fragmentation affects performance.

Project Management Resource Limits

Both set limits to prevent overuse of resources and require policies to handle excess demand.

Recognizing resource limits and policies in different fields helps appreciate why maxmemory and eviction policies are essential for system stability.

Common Pitfalls

#1 Setting maxmemory too high, leaving no RAM for OS and other processes.

Wrong approach:

maxmemory 100% # Trying to use all system RAM

Correct approach:

maxmemory 75% # Reserving RAM for OS and other apps

Root cause:

Misunderstanding that Redis shares system memory and needs headroom for OS and other applications.

#2 Using noeviction policy without handling write errors in the application.

Wrong approach:

maxmemory-policy noeviction
# Application ignores write failures

Correct approach:

maxmemory-policy noeviction
# Application checks and handles write errors gracefully

Root cause:

Not anticipating that Redis will reject writes when memory is full under noeviction.

#3 Assuming maxmemory limits disk usage and ignoring persistence file growth.

Wrong approach:

# Set maxmemory but no disk monitoring
maxmemory 100mb

Correct approach:

# Set maxmemory and monitor disk space for persistence
maxmemory 100mb
# Use monitoring tools for disk usage

Root cause:

Confusing memory limits with disk storage limits.

Key Takeaways

Maxmemory sets a hard limit on how much RAM Redis can use to store data.

When Redis reaches maxmemory, it removes data based on eviction policies to free space.

Choosing the right eviction policy balances data retention and availability under memory pressure.

Memory fragmentation can cause Redis to exceed maxmemory temporarily, affecting performance.

Maxmemory controls RAM usage but does not limit disk space used for persistence.