Snapshot and restore in Elasticsearch - Deep Dive

Overview - Snapshot and restore

What is it?

Snapshot and restore in Elasticsearch is a way to save a copy of your data and settings at a certain point in time. A snapshot is like a backup that you can store safely outside your main system. Later, if something goes wrong or you want to move data, you can restore from this snapshot to bring your data back. This helps protect your data and makes managing large amounts of information easier.

Why it matters

Without snapshot and restore, losing data due to mistakes, hardware failure, or upgrades would be risky and costly. It ensures that you can recover your data quickly and reliably, avoiding downtime and data loss. This is crucial for businesses that depend on Elasticsearch for search, analytics, or logging, where data integrity and availability are vital.

Where it fits

Before learning snapshot and restore, you should understand basic Elasticsearch concepts like indices, clusters, and nodes. After mastering snapshot and restore, you can explore advanced topics like disaster recovery, cross-cluster replication, and data lifecycle management.

Mental Model

Core Idea

Snapshot and restore is like taking a photo of your Elasticsearch data at a moment, so you can rewind and recover that exact state anytime.

Think of it like...

Imagine you are writing a long document and you save versions of it as you go. If you make a mistake, you can open an earlier saved version to fix it. Snapshots are these saved versions for your data.

┌───────────────┐       ┌───────────────┐
│ Elasticsearch │──────▶│ Snapshot Repo │
│    Cluster    │       │ (Backup Store)│
└───────────────┘       └───────────────┘
       ▲                        ▲
       │                        │
       │                        │
Restore from snapshot      Save snapshot
       │                        │
       ▼                        ▼
┌───────────────┐       ┌───────────────┐
│ Elasticsearch │◀─────│ Snapshot Repo │
│    Cluster    │       │ (Backup Store)│
└───────────────┘       └───────────────┘

Build-Up - 6 Steps

1

Foundation

What is a snapshot in Elasticsearch

Concept: Introduce the idea of a snapshot as a backup of data and metadata.

A snapshot in Elasticsearch is a copy of your indices and cluster metadata saved to a repository. It captures the state of your data at a specific time. Snapshots are incremental, meaning after the first full snapshot, only changes are saved to save space.

Result

You understand that snapshots are backups that can be stored and reused later.

Understanding snapshots as point-in-time backups helps you see how data safety and recovery are possible in Elasticsearch.

2

Foundation

Setting up a snapshot repository

Concept: Learn how to create a place to store snapshots.

Before taking snapshots, you must register a snapshot repository. This is a shared storage location like a file system, Amazon S3, or other cloud storage. Elasticsearch needs access to this repository to save and restore snapshots.

Result

You can configure Elasticsearch to save snapshots to a safe external location.

Knowing that snapshots require a repository clarifies that backups are stored outside the cluster for safety and durability.

3

Intermediate

Taking and managing snapshots

🤔Before reading on: Do you think snapshots capture live changes instantly or only at the moment you start the snapshot? Commit to your answer.

Concept: Learn how to create snapshots and understand their incremental nature.

You can create snapshots manually or schedule them. When a snapshot starts, Elasticsearch copies data files that represent the current state. Because snapshots are incremental, only new or changed data since the last snapshot is saved, making the process efficient.

Result

You can create snapshots that save your data efficiently without duplicating unchanged data.

Understanding incremental snapshots helps you appreciate how backups can be fast and storage-friendly.

4

Intermediate

Restoring data from snapshots

🤔Before reading on: Do you think restoring a snapshot overwrites existing data or merges with it? Commit to your answer.

Concept: Learn how to bring back data from a snapshot into your cluster.

Restoring a snapshot recovers indices and metadata saved in that snapshot. You can restore all or some indices. Restoring can overwrite existing indices or create new ones with different names to avoid conflicts.

Result

You can recover lost or corrupted data by restoring from snapshots safely.

Knowing how restore works prevents accidental data loss and helps plan recovery strategies.

5

Advanced

Snapshot lifecycle management

🤔Before reading on: Do you think snapshots must be managed manually forever or can Elasticsearch automate this? Commit to your answer.

Concept: Learn about automating snapshot creation and deletion to manage backups efficiently.

Elasticsearch offers Snapshot Lifecycle Management (SLM) to automate snapshot schedules and retention policies. You can define when snapshots happen and how long to keep them, reducing manual work and storage costs.

Result

You can keep your backups up-to-date and storage-efficient automatically.

Understanding SLM helps maintain reliable backups without manual errors or forgotten snapshots.

6

Expert

Handling snapshot consistency and failures

🤔Before reading on: Do you think snapshots lock your cluster or allow normal operations during backup? Commit to your answer.

Concept: Explore how Elasticsearch ensures snapshot consistency without downtime and how it handles failures.

Snapshots are taken without locking the cluster, allowing reads and writes during backup. Elasticsearch uses a copy-on-write mechanism to ensure data consistency. If a snapshot fails, partial data is cleaned up, and you can retry without corrupting your repository.

Result

You understand how snapshots protect data integrity while keeping your cluster available.

Knowing snapshot internals helps design robust backup strategies and troubleshoot snapshot issues effectively.

Under the Hood

Elasticsearch snapshots work by copying data files from the Lucene segments that make up indices. It uses incremental backups by tracking which files have changed since the last snapshot. The snapshot process is coordinated by the master node, which instructs data nodes to copy files to the repository. During snapshotting, Elasticsearch allows normal operations by using copy-on-write, so ongoing changes do not affect the snapshot's consistency.

Why designed this way?

This design balances data safety with cluster availability. Early Elasticsearch versions required locking indices during backup, causing downtime. Incremental snapshots reduce storage and network load. Using shared repositories allows snapshots to be stored outside the cluster, protecting against node failures. Alternatives like full backups were too slow and storage-heavy, so incremental snapshots became the standard.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Master Node │──────▶│ Data Nodes    │──────▶│ Snapshot Repo │
│  Coordinates  │       │ Copy Files    │       │ Stores Files  │
└───────────────┘       └───────────────┘       └───────────────┘
       │                      ▲                      ▲
       │                      │                      │
       │      Copy-on-write   │                      │
       │      ensures data    │                      │
       │      consistency    │                      │

Myth Busters - 4 Common Misconceptions

Quick: Do you think snapshots lock your Elasticsearch cluster during backup? Commit to yes or no.

Common Belief: Snapshots lock the cluster, so no writes or reads can happen during backup.

Reality: Snapshots are taken without locking the cluster; reads and writes continue normally.

Why it matters: Believing snapshots lock the cluster may cause unnecessary downtime planning and fear of using snapshots in production.

Quick: Do you think every snapshot saves a full copy of all data? Commit to yes or no.

Common Belief: Each snapshot is a full backup, duplicating all data every time.

Reality: Snapshots are incremental; after the first full snapshot, only changed data is saved.

Why it matters: Thinking snapshots are full backups wastes storage and network resources and leads to inefficient backup strategies.

Quick: Do you think restoring a snapshot always overwrites existing indices? Commit to yes or no.

Common Belief: Restoring a snapshot will overwrite any existing indices with the same name automatically.

Reality: You can choose to rename indices during restore to avoid overwriting, or selectively restore indices.

Why it matters: Assuming automatic overwrite can cause accidental data loss or conflicts during restore.

Quick: Do you think snapshots can be stored only on local disks? Commit to yes or no.

Common Belief: Snapshots must be stored on local disks attached to Elasticsearch nodes.

Reality: Snapshots can be stored on various shared repositories like network file systems, Amazon S3, or Google Cloud Storage.

Why it matters: Limiting storage options reduces flexibility and disaster recovery capabilities.

Expert Zone

1

Snapshot repositories must be accessible by all nodes to avoid snapshot failures, which is often overlooked in multi-node clusters.

2

Restoring snapshots can be done to a different cluster, enabling data migration or disaster recovery across environments.

3

Snapshot lifecycle management policies can be combined with index lifecycle management to automate full data lifecycle from creation to deletion.

When NOT to use

Snapshot and restore is not suitable for real-time replication or high-frequency backups due to snapshot duration and resource use. For real-time data sync, use cross-cluster replication or other streaming methods.

Production Patterns

In production, snapshots are scheduled during low-traffic periods using Snapshot Lifecycle Management. Snapshots are stored in cloud repositories for durability. Restores are tested regularly as part of disaster recovery drills. Partial restores are used to recover specific indices without downtime.

Connections

Version Control Systems

Both use incremental snapshots to save changes efficiently over time.

Understanding how Git stores changes incrementally helps grasp how Elasticsearch snapshots avoid duplicating unchanged data.

Disaster Recovery Planning

Snapshot and restore is a core technique in disaster recovery strategies.

Knowing snapshot and restore deepens understanding of how organizations prepare for and recover from data loss events.

Photography

Snapshot and restore conceptually mirrors taking photos to capture moments for later review.

Recognizing this connection helps appreciate the importance of capturing exact states for recovery and analysis.

Common Pitfalls

#1 Trying to create a snapshot without registering a repository first.

Wrong approach:

POST /_snapshot/my_backup/snapshot_1
{
"indices": "*"
}

Correct approach:

PUT /_snapshot/my_backup
{
"type": "fs",
"settings": {
"location": "/mount/backups/my_backup"
}
}

POST /_snapshot/my_backup/snapshot_1
{
"indices": "*"
}

Root cause:

Not understanding that snapshots require a repository to store data before creating snapshots.

#2 Restoring a snapshot without handling existing indices, causing conflicts.

Wrong approach:

POST /_snapshot/my_backup/snapshot_1/_restore
{
"indices": "logs"
}

Correct approach:

POST /_snapshot/my_backup/snapshot_1/_restore
{
"indices": "logs",
"rename_pattern": "logs",
"rename_replacement": "restored_logs"
}

Root cause:

Assuming restore overwrites existing indices without conflict, leading to errors or data loss.

#3 Scheduling snapshots too frequently without considering cluster load.

Wrong approach:

Setting snapshot lifecycle to run every minute on a large cluster.

Correct approach:

Scheduling snapshots during off-peak hours with reasonable intervals like daily or hourly based on data change rate.

Root cause:

Not considering resource usage and snapshot duration, causing performance degradation.

Key Takeaways

Snapshots in Elasticsearch are incremental backups that capture your data and metadata at a point in time without locking the cluster.

You must configure a snapshot repository before creating snapshots to store backups safely outside the cluster.

Restoring snapshots can recover lost data selectively and safely, with options to avoid overwriting existing indices.

Snapshot Lifecycle Management automates backup schedules and retention, making data protection reliable and efficient.

Understanding snapshot internals and common pitfalls helps design robust backup and recovery strategies for production Elasticsearch clusters.