AOF rewrite process in Redis - Deep Dive

Overview - AOF rewrite process

What is it?

The AOF rewrite process in Redis is a method to compact the Append-Only File (AOF) that logs every write operation. Instead of keeping all commands since the server started, it creates a smaller file that represents the current database state. This helps reduce file size and speeds up server restarts. The process runs in the background without blocking Redis from serving clients.

Why it matters

Without the AOF rewrite process, the AOF file would grow endlessly, making restarts slow and consuming more disk space. This would lead to longer downtime and inefficient storage use. The rewrite process keeps Redis fast and reliable by maintaining a compact log of changes, ensuring quick recovery and minimal resource use.

Where it fits

Before learning about the AOF rewrite process, you should understand Redis persistence basics, especially how the Append-Only File works. After this, you can explore Redis replication and snapshotting for full data durability strategies.

Mental Model

Core Idea

The AOF rewrite process compacts Redis's command log by rewriting it into a minimal set of commands that recreate the current data state.

Think of it like...

Imagine writing down every step you take during a day in a notebook. Over time, the notebook gets very thick. The AOF rewrite is like summarizing your whole day into a short list of key activities, so the notebook becomes thin again but still tells the full story.

┌───────────────────────────────┐
│ Original AOF file (all commands)│
│  SET key1 val1                │
│  INCR counter                 │
│  SET key1 val2                │
│  ...                         │
└──────────────┬────────────────┘
               │ Rewrite process
               ▼
┌───────────────────────────────┐
│ Rewritten AOF file (compact)  │
│  SET key1 val2                │
│  INCR counter (final value)   │
└───────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

What is Redis AOF Persistence

Concept: Introduce the Append-Only File as a way Redis saves every write command to disk.

Redis uses the Append-Only File (AOF) to record every change made to the database. Each write command is added to this file so Redis can replay them to restore data after a restart.

Result

You understand that AOF is a log of all write commands, ensuring data durability.

Knowing that AOF logs every command helps you see why the file can grow very large over time.

2

Foundation

Why AOF File Grows Over Time

Concept: Explain that every write command is appended, causing the file to grow continuously.

Every time Redis changes data, it adds a new command to the AOF file. Even if a key is updated multiple times, all commands are kept. This means the file size keeps increasing as Redis runs.

Result

You realize that without intervention, the AOF file becomes very large and inefficient.

Understanding file growth sets the stage for why rewriting is necessary.

3

Intermediate

How AOF Rewrite Compacts Data

🤔Before reading on: do you think the rewrite copies all commands or only the latest needed? Commit to your answer.

Concept: The rewrite process creates a new AOF file with only the minimal commands needed to recreate the current database state.

Instead of copying every command, Redis scans the current database and writes commands that set keys to their latest values. This removes old commands that are no longer needed, making the file smaller.

Result

The new AOF file is much smaller but still fully restores the database.

Knowing that rewrite uses the current data snapshot to generate commands explains how it reduces file size drastically.

4

Intermediate

Background Rewrite Without Blocking

🤔Before reading on: do you think Redis stops serving clients during rewrite? Commit to your answer.

Concept: Redis performs the rewrite in a child process so the main server keeps running without delay.

When rewriting starts, Redis forks a child process to create the new AOF file. The parent continues handling client requests. After the child finishes, Redis merges new commands received during rewrite into the new file.

Result

Redis remains responsive during rewrite, avoiding downtime.

Understanding the fork-and-merge approach explains how Redis balances durability and performance.

5

Intermediate

Merging Commands After Rewrite

Concept: Explain how Redis handles commands received during the rewrite to keep data consistent.

While the child process rewrites the AOF, the parent keeps logging new commands. After rewrite finishes, Redis appends these new commands to the rewritten file before replacing the old AOF.

Result

The final AOF file includes all changes, old and new, ensuring no data loss.

Knowing this merge step prevents confusion about missing recent commands after rewrite.

6

Advanced

Triggers and Configuration of Rewrite

🤔Before reading on: do you think rewrite happens automatically or only manually? Commit to your answer.

Concept: Rewrite can be triggered automatically based on file size growth or manually by the user.

Redis can be set to rewrite the AOF when it grows a certain size compared to the last rewrite. Users can also force rewrite commands. This flexibility helps balance performance and durability needs.

Result

You understand how to control when rewrites happen to optimize Redis operation.

Knowing triggers helps you tune Redis for your workload and avoid unexpected delays.

7

Expert

Edge Cases and Performance Surprises

🤔Before reading on: do you think rewrite always reduces file size? Commit to your answer.

Concept: Rewrite may not always shrink the file if data is large or fragmented; also, fork overhead can impact performance on big datasets.

If the dataset is very large or has many small changes, the rewritten AOF might be similar in size. Forking creates a copy of memory, which can be costly on systems with limited RAM. Understanding these helps plan hardware and rewrite frequency.

Result

You gain insight into when rewrite might not help and how to mitigate performance hits.

Recognizing rewrite limits prevents surprises in production and guides better resource planning.

Under the Hood

Redis uses a fork system call to create a child process that has a copy of the server's memory. The child scans the in-memory data structures and writes commands that recreate the current state to a temporary AOF file. Meanwhile, the parent continues to accept writes and logs them separately. After the child finishes, the parent appends the new commands to the rewritten file and atomically replaces the old AOF file.

Why designed this way?

This design balances durability and performance by avoiding blocking the main server during rewrite. Forking leverages copy-on-write memory to minimize overhead. Alternatives like blocking rewrites would cause downtime, which is unacceptable for many Redis use cases.

┌───────────────┐          fork          ┌───────────────┐
│ Parent Redis  │──────────────────────▶│ Child Process │
│ (serving cmds)│                       │ (rewriting AOF)│
└──────┬────────┘                       └──────┬────────┘
       │                                      │
       │ Logs new commands                    │ Scans DB and writes compact AOF
       │                                      │
       │◀───────────── Merge new cmds ──────┤
       │                                      │
       ▼                                      ▼
┌───────────────────────────────────────────────────────────┐
│ Replace old AOF file with rewritten compact AOF file atomically │
└───────────────────────────────────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does AOF rewrite delete any data? Commit yes or no.

Common Belief: AOF rewrite deletes old data commands, so some data might be lost.

Reality: AOF rewrite only removes redundant commands but keeps all current data by rewriting commands that recreate the full database state.

Why it matters: Believing rewrite deletes data causes unnecessary fear and avoidance, risking huge AOF files and slow restarts.

Quick: Does Redis stop responding during AOF rewrite? Commit yes or no.

Common Belief: Redis blocks all client requests during AOF rewrite.

Reality: Redis continues serving clients during rewrite by forking a child process; only a brief pause happens during final file replacement.

Why it matters: Thinking Redis blocks leads to poor design decisions and misunderstanding of Redis's high availability.

Quick: Does AOF rewrite always reduce file size? Commit yes or no.

Common Belief: AOF rewrite always makes the file much smaller.

Reality: Sometimes the rewritten file is similar in size if data is large or frequently changing, so rewrite doesn't always shrink the file.

Why it matters: Expecting guaranteed size reduction can cause confusion and misinterpretation of Redis performance.

Quick: Is manual rewrite the only way to compact AOF? Commit yes or no.

Common Belief: You must manually trigger AOF rewrite to compact the file.

Reality: Redis can automatically trigger rewrite based on configuration thresholds, reducing manual work.

Why it matters: Not knowing about automatic rewrite can lead to neglecting configuration and inefficient Redis operation.

Expert Zone

1

The fork system call uses copy-on-write memory, so the child process shares memory pages with the parent until writes happen, minimizing overhead.

2

During rewrite, Redis buffers new commands in memory to append later, so heavy write loads can increase memory usage temporarily.

3

The atomic rename of the rewritten AOF file ensures no partial or corrupted files replace the original, preserving data integrity.

When NOT to use

AOF rewrite is not suitable if your dataset is extremely large and fork overhead causes latency spikes; in such cases, consider using RDB snapshots or Redis Cluster to distribute load.

Production Patterns

In production, AOF rewrite is often scheduled during low traffic periods or triggered automatically when the AOF grows 100% larger than the last rewrite. Monitoring tools alert if rewrite takes too long or fails, ensuring reliability.

Connections

Database Checkpointing

Both create a compact snapshot of data to speed recovery.

Understanding AOF rewrite helps grasp how databases save state efficiently without replaying full logs.

Operating System Forking

AOF rewrite relies on OS-level fork to create a child process with copy-on-write memory.

Knowing OS forking mechanics clarifies how Redis achieves non-blocking rewrites with minimal memory cost.

Version Control Systems

Like git compresses commit history into snapshots, AOF rewrite compresses command history into minimal state.

Seeing this parallel helps appreciate how systems manage growing histories efficiently.

Common Pitfalls

#1 Running AOF rewrite too frequently causing high CPU and memory usage.

Wrong approach:

Configuring Redis to rewrite AOF every few seconds regardless of file size.

Correct approach:

Set rewrite triggers based on meaningful file size growth thresholds to balance performance and durability.

Root cause:

Misunderstanding rewrite cost leads to over-triggering and resource exhaustion.

#2 Assuming rewrite deletes data and disabling it out of fear.

Wrong approach:

Disabling AOF rewrite in redis.conf to avoid data loss.

Correct approach:

Enable rewrite to keep AOF file manageable and ensure fast restarts without data loss.

Root cause:

Confusing rewrite compaction with data deletion causes harmful configuration.

#3 Ignoring memory usage spikes during rewrite causing crashes.

Wrong approach:

Running rewrite on a server with limited RAM without monitoring.

Correct approach:

Monitor memory and schedule rewrites during low load or upgrade hardware to handle fork overhead.

Root cause:

Not accounting for copy-on-write memory usage during fork leads to unexpected failures.

Key Takeaways

The AOF rewrite process compacts Redis's command log by rewriting only the commands needed to recreate the current data state.

It runs in a child process using OS fork, allowing Redis to serve clients without blocking during rewrite.

Rewrite merges new commands received during the process to ensure no data loss.

Automatic and manual triggers control when rewrite happens, balancing performance and durability.

Understanding rewrite internals helps avoid common pitfalls like resource exhaustion and misconfiguration.