etcd backup and recovery in Kubernetes - Step-by-Step Execution

Process Flow - etcd backup and recovery

Start etcd backup

↓

Run etcdctl snapshot save

↓

Verify snapshot file

↓

Store snapshot safely

↓

If recovery needed?

No→End

Yes↓

Stop etcd service

↓

Run etcdctl snapshot restore

↓

Restart etcd service

↓

Verify etcd health

↓

Recovery complete

This flow shows how to create a backup snapshot of etcd, store it, and restore it if needed to recover the cluster state.

Execution Sample

Kubernetes

ETCDCTL_API=3 etcdctl snapshot save backup.db
systemctl stop etcd
ETCDCTL_API=3 etcdctl snapshot restore backup.db --data-dir restored_etcd
mv /var/lib/etcd /var/lib/etcd-old
mv restored_etcd /var/lib/etcd
systemctl start etcd
ETCDCTL_API=3 etcdctl endpoint health

This sequence saves a snapshot, stops etcd service, restores it to a new data directory, backs up and replaces the old data directory, starts etcd, and checks health.

Process Table

Step	Command	Action	Result/Output
1	ETCDCTL_API=3 etcdctl snapshot save backup.db	Create snapshot file	Snapshot saved to backup.db
2	ls backup.db	Verify snapshot file exists	backup.db listed
3	systemctl stop etcd	Stop etcd service before restore	etcd service stopped
4	ETCDCTL_API=3 etcdctl snapshot restore backup.db --data-dir restored_etcd	Restore snapshot to new data directory	Snapshot restored to restored_etcd
5	mv /var/lib/etcd /var/lib/etcd-old	Backup old data directory	Old data backed up
6	mv restored_etcd /var/lib/etcd	Replace data directory with restored data	Data directory replaced
7	systemctl start etcd	Start etcd service	etcd service started
8	ETCDCTL_API=3 etcdctl endpoint health	Check etcd health	endpoint is healthy
9	-	Recovery complete	etcd cluster restored and healthy

💡 Recovery ends after etcd service is healthy and running with restored data

Status Tracker

Variable	Start	After Step 1	After Step 4	After Step 7	Final
snapshot_file	none	backup.db created	backup.db unchanged	backup.db unchanged	backup.db unchanged
etcd_service	running	running	stopped	started	running
data_directory	/var/lib/etcd	/var/lib/etcd	restored_etcd (new)	/var/lib/etcd (restored)	/var/lib/etcd (restored)
etcd_health	healthy	healthy	unknown (stopped)	unknown (starting)	healthy

Key Moments - 3 Insights

Why do we stop the etcd service before restoring the snapshot?

Stopping etcd ensures no data changes happen during restore, preventing corruption. See execution_table step 3 where etcd stops before restore.

What happens if the snapshot file is missing or corrupted?

The restore command will fail and etcd cannot recover. Step 2 verifies snapshot existence to avoid this issue.

Why do we move the old data directory before replacing it with restored data?

This backup step prevents data loss if restore fails, allowing rollback. See step 5 in execution_table.

Visual Quiz - 3 Questions

Look at the execution table, what is the state of etcd_service after step 4?

A running

B stopped

C starting

D unknown

💡 Hint

Check the 'etcd_service' variable in variable_tracker after step 4

At which step is the snapshot file created?

A Step 1

B Step 3

C Step 5

D Step 7

💡 Hint

Look at execution_table step 1 command and result

If we skip moving the old data directory (step 5), what risk increases?

A etcd service won't start

B Snapshot file will be deleted

C No backup of old data, risking data loss

D etcd health check will fail

💡 Hint

Refer to key_moments explanation about step 5

Concept Snapshot

etcd backup and recovery:
- Use 'etcdctl snapshot save <file>' to backup
- Stop etcd before restore
- Restore with 'etcdctl snapshot restore <file> --data-dir <dir>'
- Replace old data dir with restored dir
- Restart etcd and verify health
- Always keep backup of old data before restore

Full Transcript

This visual execution shows how to backup and recover etcd data in Kubernetes. First, we create a snapshot file using 'etcdctl snapshot save'. We verify the snapshot exists. Before restoring, we stop the etcd service to avoid data corruption. Then we restore the snapshot to a new data directory. We backup the old data directory by moving it, then replace it with the restored data. After that, we start the etcd service again and check its health to confirm recovery success. Key points include stopping etcd before restore and backing up old data to prevent loss. The execution table and variable tracker clearly show each step and state change for easy understanding.