AOF (Append Only File) in Redis - Deep Dive

Overview - AOF (Append Only File)

What is it?

AOF, or Append Only File, is a way Redis saves all the changes made to its data by recording each write operation in a file. Instead of saving the whole database at once, it keeps a log of commands that can rebuild the data step-by-step. This helps Redis recover data after a restart or crash by replaying the saved commands.

Why it matters

Without AOF, if Redis crashes or restarts, all recent changes since the last save could be lost, causing data loss. AOF ensures durability by keeping a detailed history of changes, so Redis can restore the exact state it had before failure. This is crucial for applications that need reliable data storage and cannot afford to lose updates.

Where it fits

Before learning AOF, you should understand basic Redis data storage and commands. After mastering AOF, you can explore Redis persistence strategies like RDB snapshots and hybrid persistence, as well as performance tuning and recovery techniques.

Mental Model

Core Idea

AOF is like a diary that records every change made to Redis data, so it can replay those changes to restore the exact state after a restart.

Think of it like...

Imagine writing down every step you take during a recipe in a notebook. If you lose the final dish, you can follow your notes step-by-step to recreate it exactly as before.

┌───────────────┐
│ Redis Server  │
├───────────────┤
│ Write Command │
│ (e.g. SET)    │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Append to AOF │
│ (log command) │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Data in Redis │
│ Memory        │
└───────────────┘

Build-Up - 7 Steps

1

Foundation

What is Redis Persistence

Concept: Introduce the idea that Redis can save data to disk to survive restarts.

Redis stores data in memory for speed, but memory is volatile. Persistence means saving data to disk so it is not lost if Redis stops. Redis offers two main ways: snapshots (RDB) and command logging (AOF).

Result

Learner understands why Redis needs to save data beyond memory.

Understanding persistence is key to trusting Redis with important data.

2

Foundation

Basics of Append Only File

Concept: Explain that AOF logs every write command to a file in order.

Each time Redis changes data, it writes the command to the AOF file. This file grows over time and can be replayed to rebuild the database state exactly as it was.

Result

Learner sees how AOF records changes step-by-step.

Knowing that AOF logs commands, not snapshots, clarifies how Redis recovers data.

3

Intermediate

AOF Rewrite and Compaction

🤔Before reading on: do you think the AOF file keeps growing forever or does Redis manage its size? Commit to your answer.

Concept: Introduce the process of rewriting AOF to keep it compact and efficient.

Over time, the AOF file can become very large because it logs every command. Redis periodically rewrites the AOF file by creating a new file with only the minimal commands needed to recreate the current data, removing redundant commands.

Result

Learner understands how Redis prevents AOF from growing without limit.

Knowing about AOF rewriting explains how Redis balances durability with performance.

4

Intermediate

AOF Write and Sync Policies

🤔Before reading on: do you think Redis writes to AOF immediately on every command or batches writes? Commit to your answer.

Concept: Explain different ways Redis controls when AOF data is written and saved to disk.

Redis can write AOF data to disk in three modes: always (every command), every second, or never (letting OS decide). This affects durability and performance. Writing every command is safest but slower; writing less often is faster but risks losing recent commands on crash.

Result

Learner grasps trade-offs between safety and speed in AOF syncing.

Understanding sync policies helps choose the right balance for your application's needs.

5

Intermediate

AOF vs RDB Persistence Comparison

🤔Before reading on: do you think AOF or RDB is better for data safety? Commit to your answer.

Concept: Compare AOF and RDB persistence methods and their pros and cons.

RDB saves snapshots at intervals, which is fast but can lose recent data. AOF logs every command, offering better durability but can be slower and produce larger files. Redis can use both together for safety and speed.

Result

Learner can decide when to use AOF, RDB, or both.

Knowing strengths and weaknesses of persistence methods guides better system design.

6

Advanced

AOF Recovery Process

🤔Before reading on: do you think Redis loads AOF by replaying commands or restoring a snapshot? Commit to your answer.

Concept: Explain how Redis uses the AOF file to restore data after restart.

On startup, Redis reads the AOF file line by line and executes each command to rebuild the in-memory data. This ensures the database state matches exactly before shutdown or crash.

Result

Learner understands the step-by-step recovery mechanism.

Knowing the replay process clarifies why AOF files must be consistent and well-formed.

7

Expert

AOF Internal Format and Performance

🤔Before reading on: do you think AOF stores commands as plain text or a special format? Commit to your answer.

Concept: Dive into the Redis protocol format used in AOF and its impact on performance.

AOF stores commands in Redis Serialization Protocol (RESP) format, which is a compact, binary-safe text format. This allows fast parsing and appending. Redis optimizes AOF writes with buffering and background rewriting to minimize latency.

Result

Learner gains insight into how AOF balances durability with speed internally.

Understanding the RESP format and buffering explains why AOF is efficient despite logging every command.

Under the Hood

AOF works by intercepting every write command executed by Redis and appending it to a file in a special format called RESP. This file acts as a log of all changes. On restart, Redis reads this file sequentially and re-executes each command to restore the exact data state. To keep the file size manageable, Redis periodically rewrites the AOF file by creating a minimal set of commands that produce the current state, removing redundant or overwritten commands.

Why designed this way?

AOF was designed to provide a durable, crash-safe way to persist Redis data without blocking the server for long periods. Unlike snapshots that save the whole dataset at once, AOF logs incremental changes, reducing data loss risk. The RESP format was chosen for its simplicity and speed, allowing Redis to parse and write commands efficiently. The rewrite mechanism balances durability with file size and performance, avoiding unbounded growth.

┌───────────────┐
│ Client Writes │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Redis Executes│
│ Command       │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Append Command│
│ to AOF File   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ AOF File on   │
│ Disk          │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Redis Restart │
│ Reads AOF     │
│ Replays Cmds  │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does AOF guarantee zero data loss in all configurations? Commit yes or no.

Common Belief: AOF always guarantees no data loss because it logs every command.

Reality: AOF durability depends on the fsync policy. If fsync is set to every second or never, recent commands can be lost on crash.

Why it matters: Assuming zero data loss can lead to unexpected data loss in production if sync policies are not configured properly.

Quick: Is AOF file always smaller than RDB snapshot? Commit yes or no.

Common Belief: AOF files are always smaller than RDB snapshots because they only log commands.

Reality: AOF files can be larger than RDB snapshots because they log every command, including repeated or overwritten data, until rewritten.

Why it matters: Misunderstanding file size can cause storage planning errors and performance issues.

Quick: Does Redis block all commands during AOF rewrite? Commit yes or no.

Common Belief: Redis stops processing all commands during AOF rewrite to keep data consistent.

Reality: Redis performs AOF rewrite in the background without blocking clients, using copy-on-write to maintain performance.

Why it matters: Believing Redis blocks can lead to unnecessary fears about downtime and poor system design.

Quick: Is AOF replay always faster than loading RDB snapshot? Commit yes or no.

Common Belief: AOF replay is always faster because it just replays commands.

Reality: AOF replay can be slower than loading an RDB snapshot because it must execute every command sequentially.

Why it matters: Expecting fast restarts with AOF alone can cause surprises in recovery time.

Expert Zone

1

AOF rewrite can be triggered manually or automatically based on file size growth, allowing fine control over performance and durability trade-offs.

2

The choice of fsync policy affects not only durability but also latency spikes and throughput, requiring careful tuning for production workloads.

3

Redis supports hybrid persistence by loading RDB snapshot first and then applying AOF commands, speeding up recovery while maintaining durability.

When NOT to use

AOF is not ideal when extremely low latency is required and some data loss is acceptable; in such cases, RDB snapshots or no persistence might be preferred. For very large datasets with infrequent writes, RDB snapshots can be more efficient. Also, if disk space is limited, AOF file size and rewrite overhead may be problematic.

Production Patterns

In production, many Redis setups use AOF with 'everysec' fsync policy for a balance of durability and performance. They combine AOF with periodic RDB snapshots for faster recovery. Monitoring AOF file size and rewrite frequency is common to avoid performance degradation. Some use AOF only for critical data and disable it for ephemeral caches.

Connections

Write-Ahead Logging (WAL)

AOF is a form of write-ahead logging used in databases.

Understanding AOF helps grasp how databases ensure durability by logging changes before applying them.

Event Sourcing (Software Design)

AOF logs every change as an event, similar to event sourcing patterns.

Knowing AOF clarifies how event sourcing captures state changes as a sequence of events for replay and recovery.

Version Control Systems

AOF's command log resembles commit history in version control.

Seeing AOF as a history of changes helps understand incremental data recovery and compaction.

Common Pitfalls

#1 Assuming AOF fsync 'always' mode is enabled by default.

Wrong approach:

appendonly yes
appendfsync everysec

Correct approach:

appendonly yes
appendfsync always

Root cause:

Misunderstanding default fsync settings leads to weaker durability guarantees.

#2 Ignoring AOF rewrite causing large disk usage.

Wrong approach:

appendonly yes
no monitoring or rewrite configuration

Correct approach:

appendonly yes
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

Root cause:

Not configuring rewrite triggers causes uncontrolled AOF growth and disk exhaustion.

#3 Stopping Redis during AOF rewrite due to fear of blocking.

Wrong approach:

Manually stopping Redis to rewrite AOF file.

Correct approach:

Use BGREWRITEAOF command to rewrite AOF in background without downtime.

Root cause:

Misunderstanding Redis's non-blocking rewrite leads to unnecessary downtime.

Key Takeaways

AOF is a Redis persistence method that logs every write command to a file to ensure data durability.

It allows Redis to recover the exact data state by replaying commands after a restart or crash.

AOF files can grow large, so Redis rewrites them in the background to keep size manageable without blocking operations.

Choosing the right fsync policy balances durability and performance based on application needs.

Understanding AOF internals and trade-offs is essential for designing reliable and efficient Redis systems.