Backup and disaster recovery in Hadoop - Step-by-Step Execution

Concept Flow - Backup and disaster recovery

Start: Data in Hadoop Cluster

↓

Create Backup Snapshot

↓

Store Backup in Safe Location

↓

Monitor Cluster Health

↓

Disaster Occurs?

No→Continue Operations

Yes↓

Restore Data from Backup

↓

Verify Data Integrity

↓

Resume Normal Operations

This flow shows how Hadoop data is backed up, monitored, and restored after a disaster to keep data safe and operations running.

Execution Sample

Hadoop

hdfs dfsadmin -saveNamespace
hdfs dfs -createSnapshot /data backup1
hdfs dfs -cp /data/.snapshot/backup1/* /backup_location/
# Disaster happens
hdfs dfs -rm -r /data/*
hdfs dfs -cp /backup_location/* /data/
hdfs dfsadmin -saveNamespace

This code creates a snapshot backup of Hadoop data, copies it to a safe location, simulates data loss, and restores data from the backup.

Execution Table

Step	Command	Action	Result
1	hdfs dfsadmin -saveNamespace	Save current namespace metadata	Namespace saved successfully
2	hdfs dfs -createSnapshot /data backup1	Create snapshot named 'backup1' of /data	Snapshot 'backup1' created
3	hdfs dfs -cp /data/.snapshot/backup1/* /backup_location/	Copy snapshot to backup location	Snapshot copied to /backup_location/
4	# Disaster happens	Disaster occurs	Disaster initiated
5	hdfs dfs -rm -r /data/*	Remove contents of /data directory	/data contents removed
6	hdfs dfs -cp /backup_location/* /data/	Copy backup snapshot back to /data	Backup restored to /data
7	hdfs dfsadmin -saveNamespace	Save namespace metadata after restore	Namespace saved successfully

💡 Data restored from backup, namespace restored, system ready for normal operations

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 5	After Step 6	Final
/data content	Original data present	Snapshot created (data unchanged)	Backup copied (data unchanged)	Deleted (empty)	Restored from backup	Restored data present
Namespace	Current metadata	Saved metadata	Saved metadata	Saved metadata	Saved metadata	Saved metadata

Key Moments - 3 Insights

Why do we create a snapshot before copying data to backup location?

Creating a snapshot freezes the data state at a point in time, ensuring the backup is consistent and not affected by ongoing changes. See execution_table step 2 and 3.

What happens if we try to restore data without saving the namespace first?

Namespace metadata might be out of sync causing inconsistencies. Saving namespace (step 1) and saving it again (step 7) keeps metadata aligned with data.

Why do we delete /data before restoring from backup?

Deleting /data simulates data loss or disaster. It ensures the restore process actually replaces lost data. See execution_table step 5.

Visual Quiz - 3 Questions

Look at the execution_table, what is the result after step 3?

A Snapshot copied to /backup_location/

B Namespace saved successfully

C Snapshot 'backup1' created

D Backup restored to /data

💡 Hint

Check the 'Result' column for step 3 in execution_table

At which step is the /data directory deleted?

A Step 4

B Step 5

C Step 6

D Step 7

💡 Hint

Look for the command 'hdfs dfs -rm -r /data' in execution_table

If we skip saving the namespace at step 1, what might happen?

A Backup snapshot will fail

B Data will not be deleted

C Namespace metadata may be inconsistent after restore

D Snapshot cannot be copied

💡 Hint

Refer to key_moments about namespace saving and restoring

Concept Snapshot

Backup and disaster recovery in Hadoop:
- Use 'hdfs dfs -createSnapshot' to freeze data state
- Copy snapshot to safe backup location
- Save namespace metadata with 'hdfs dfsadmin -saveNamespace'
- On disaster, delete lost data and restore from backup
- Save namespace metadata again ('hdfs dfsadmin -saveNamespace') to keep system consistent
- Verify data integrity before resuming operations

Full Transcript

This visual execution shows how Hadoop handles backup and disaster recovery. First, the system saves the namespace metadata to keep track of the file system state. Then, it creates a snapshot of the data directory to capture a consistent copy. This snapshot is copied to a backup location for safekeeping. When a disaster occurs, such as data loss, the contents of the original data directory are deleted to simulate the loss. The backup snapshot is then copied back to the original location to restore the data. Finally, the namespace metadata is saved again ('hdfs dfsadmin -saveNamespace') to ensure the file system metadata matches the restored data. This process helps keep Hadoop data safe and recoverable.