Backup and disaster recovery in Hadoop - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main goal of backup in Hadoop?

The main goal of backup in Hadoop is to create copies of important data to protect it from loss or corruption, so it can be restored if needed.

beginner

Define disaster recovery in the context of Hadoop.

Disaster recovery in Hadoop means having a plan and tools to restore data and services quickly after a failure or disaster, ensuring minimal downtime.

intermediate

Name two common backup methods used in Hadoop.

Two common backup methods in Hadoop are: 1) DistCp (distributed copy) to copy data between clusters, and 2) Snapshots to capture the state of data at a point in time.

intermediate

How do Hadoop snapshots help in disaster recovery?

Hadoop snapshots save the exact state of the file system at a moment, allowing quick rollback to that state if data is lost or corrupted.

beginner

Why is it important to test your disaster recovery plan regularly?

Testing ensures the plan works correctly, helps find problems early, and trains the team to respond quickly during a real disaster.

What Hadoop tool is commonly used to copy large amounts of data between clusters for backup?

A DistCp

B Hive

C Pig

D Sqoop

DistCp is designed for distributed copying of large data sets between Hadoop clusters.

What does a Hadoop snapshot capture?

A Current state of the file system

B User permissions only

C Network configuration

D Job execution logs

Snapshots capture the exact state of the file system at a specific time.

Which of the following is NOT a goal of disaster recovery?

A Minimize downtime

B Increase data duplication

C Prevent data loss

D Restore services quickly

Increasing data duplication is not a direct goal; the focus is on recovery and minimizing impact.

Why should backup data be stored in a different location?

A To avoid data encryption

B To reduce storage costs

C To speed up data processing

D To protect against site-wide disasters

Storing backups offsite protects data if the main site is damaged.

What is a key benefit of regularly testing your disaster recovery plan?

A It guarantees no disasters will happen

B It increases data storage capacity

C It helps identify weaknesses in the plan

D It reduces the need for backups

Testing helps find and fix problems before a real disaster occurs.

Explain how Hadoop snapshots and DistCp work together to support backup and disaster recovery.

Think about capturing data state and copying it safely.

Describe why having a disaster recovery plan is critical for Hadoop data environments.

Consider what happens if data or services suddenly stop working.

Practice

1. What is the main purpose of backup in Hadoop's disaster recovery?

easy

A. To delete old data automatically

B. To save copies of data to prevent loss

C. To speed up data processing

D. To encrypt data for security

Solution

1. Step 1: Understand backup purpose

   Backup means saving copies of data so it can be recovered if lost.

2. Step 2: Match with options

   Only To save copies of data to prevent loss describes saving copies to prevent data loss.

3. Final Answer:

   To save copies of data to prevent loss -> Option B

4. Quick Check:

   Backup = Save copies [OK]

Hint: Backup means saving data copies to avoid loss [OK]

Common Mistakes:

• Confusing backup with data processing speed
• Thinking backup deletes data
• Mixing backup with encryption

2. Which Hadoop command is used to copy files from HDFS to local for backup?

easy

A. hdfs dfs -get

B. hdfs dfs -ls

C. hdfs dfs -rm

D. hdfs dfs -put

Solution

1. Step 1: Identify command to copy from HDFS to local

   The command hdfs dfs -get copies files from HDFS to local filesystem.

2. Step 2: Check other commands

   -put copies local to HDFS, -rm removes files, -ls lists files.

3. Final Answer:

   hdfs dfs -get -> Option A

4. Quick Check:

   Copy HDFS to local = -get [OK]

Hint: Use -get to copy from HDFS to local [OK]

Common Mistakes:

• Using -put instead of -get
• Confusing -rm with copy commands
• Thinking -ls copies files

3. What will be the output of this command if the source file does not exist?
hdfs dfs -cp /data/file1.txt /backup/file1.txt

medium

A. cp: `/data/file1.txt`: No such file or directory

B. File copied successfully

C. Backup created with empty file

D. Command runs silently without error

Solution

1. Step 1: Understand the cp command behavior

   The hdfs dfs -cp command copies files within HDFS. If the source file does not exist, it throws an error.

2. Step 2: Match error message

   The error message for missing source file is "No such file or directory" as in cp: `/data/file1.txt`: No such file or directory.

3. Final Answer:

   cp: `/data/file1.txt`: No such file or directory -> Option A

4. Quick Check:

   Missing source file = error message [OK]

Hint: Missing source file causes 'No such file' error [OK]

Common Mistakes:

• Assuming silent success
• Expecting empty file creation
• Confusing error with copy success

4. Identify the error in this backup command:
hdfs dfs -copyToLocal /backup/data.txt /local/backup/

medium

A. The command should be -copyFromLocal to backup

B. The destination path should be HDFS, not local

C. The command is correct as is

D. The source path should be local, not HDFS

Solution

1. Step 1: Understand -copyToLocal command

   This command copies files from HDFS to local filesystem, useful for backup.

2. Step 2: Check source and destination paths

   Source is HDFS (/backup/data.txt), destination is local (/local/backup/), which matches command purpose.

3. Final Answer:

   The command is correct as is -> Option C

4. Quick Check:

   copyToLocal copies HDFS to local [OK]

Hint: copyToLocal copies from HDFS to local correctly [OK]

Common Mistakes:

• Confusing copyToLocal with copyFromLocal
• Swapping source and destination paths
• Thinking backup requires copyFromLocal

5. You want to create a disaster recovery plan that backs up daily changes in Hadoop data to a remote cluster. Which approach is best?

hard

A. Use hdfs dfs -rm to remove data after backup

B. Manually copy files with hdfs dfs -cp on local machine

C. Delete old backups and keep only latest file in HDFS

D. Use hdfs dfs -distcp to copy changed files to remote cluster daily

Solution

1. Step 1: Understand disaster recovery needs

   Disaster recovery requires copying data safely to a remote cluster regularly.

2. Step 2: Evaluate commands for remote copying

   hdfs dfs -distcp is designed for efficient distributed copying between clusters, ideal for daily backups.

3. Step 3: Check other options

   Manual copy is error-prone and slow; deleting backups or removing data is risky.

4. Final Answer:

   Use hdfs dfs -distcp to copy changed files to remote cluster daily -> Option D

5. Quick Check:

   distcp = best for remote backup [OK]

Hint: Use distcp for efficient remote cluster backup [OK]

Common Mistakes:

• Using local copy commands for remote backup
• Deleting backups too early
• Confusing remove command with backup