Backup and disaster recovery in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Hadoop Backup Strategies

Which of the following best describes the primary purpose of Hadoop's DistCp tool in backup and disaster recovery?

A It compresses Hadoop data files to save storage space during backup.

B It copies large amounts of data efficiently between Hadoop clusters for backup or migration.

C It monitors Hadoop cluster health to prevent data loss.

D It encrypts Hadoop data before backup to ensure security.

💡 Hint

Think about tools designed to move data between clusters.

DistCp (Distributed Copy) is designed to copy large datasets efficiently across Hadoop clusters, which is essential for backup and disaster recovery.

❓ Predict Output

intermediate

HDFS Snapshot Output

What will be the output of the following HDFS command sequence?

hdfs dfs -mkdir /data
hdfs dfs -put file1.txt /data/
hdfs dfs -createSnapshot /data snap1
hdfs dfs -rm /data/file1.txt
hdfs dfs -ls /data/.snapshot/snap1

A Lists all files except file1.txt in the snapshot.

B Shows an empty directory because file1.txt was deleted.

C Lists file1.txt inside the snapshot directory.

D Returns an error because snapshots cannot be created on /data.

💡 Hint

Remember what snapshots do in HDFS.

HDFS snapshots preserve the state of the directory at the time of snapshot creation. Even if file1.txt is deleted later, it remains visible inside the snapshot.

❓ data_output

advanced

Analyzing Backup Data Size Reduction

You run a Hadoop backup job that uses compression. The original data size is 500 GB. After backup, the compressed backup size is 150 GB. What is the compression ratio?

A 3.33

B 0.3

C 350

D 0.03

💡 Hint

Compression ratio = original size / compressed size.

The compression ratio is calculated by dividing the original data size by the compressed size: 500 / 150 = 3.33.

🔧 Debug

advanced

Identifying Error in Hadoop Backup Script

Consider this Hadoop backup script snippet:

hdfs dfs -mkdir /backup
hadoop distcp /user/data /backup/data_backup
hdfs dfs -rm -r /user/data

What is the main risk or error in this script?

A The rm command will fail because /user/data is not empty.

B The distcp command syntax is incorrect and will cause a syntax error.

C The backup directory /backup does not exist before copying.

D The script deletes original data immediately after copying without verifying backup success.

💡 Hint

Think about safe backup practices.

Deleting original data immediately after copying without checking if the backup succeeded risks data loss if the copy fails.

🚀 Application

expert

Designing a Disaster Recovery Plan for Hadoop

You are tasked with designing a disaster recovery plan for a Hadoop cluster that must minimize downtime and data loss. Which combination of strategies is best?

A Use HDFS snapshots regularly and replicate data to a remote cluster using DistCp.

B Only rely on HDFS replication factor set to 3 within the same cluster.

C Backup data manually once a month and store on local disks.

D Use a single-node cluster backup to external USB drives weekly.

💡 Hint

Consider both data safety and recovery speed.

Combining HDFS snapshots for quick recovery and remote replication with DistCp ensures minimal downtime and protects against cluster-wide failures.

Practice

1. What is the main purpose of backup in Hadoop's disaster recovery?

easy

A. To delete old data automatically

B. To save copies of data to prevent loss

C. To speed up data processing

D. To encrypt data for security

Solution

1. Step 1: Understand backup purpose

   Backup means saving copies of data so it can be recovered if lost.

2. Step 2: Match with options

   Only To save copies of data to prevent loss describes saving copies to prevent data loss.

3. Final Answer:

   To save copies of data to prevent loss -> Option B

4. Quick Check:

   Backup = Save copies [OK]

Hint: Backup means saving data copies to avoid loss [OK]

Common Mistakes:

• Confusing backup with data processing speed
• Thinking backup deletes data
• Mixing backup with encryption

2. Which Hadoop command is used to copy files from HDFS to local for backup?

easy

A. hdfs dfs -get

B. hdfs dfs -ls

C. hdfs dfs -rm

D. hdfs dfs -put

Solution

1. Step 1: Identify command to copy from HDFS to local

   The command hdfs dfs -get copies files from HDFS to local filesystem.

2. Step 2: Check other commands

   -put copies local to HDFS, -rm removes files, -ls lists files.

3. Final Answer:

   hdfs dfs -get -> Option A

4. Quick Check:

   Copy HDFS to local = -get [OK]

Hint: Use -get to copy from HDFS to local [OK]

Common Mistakes:

• Using -put instead of -get
• Confusing -rm with copy commands
• Thinking -ls copies files

3. What will be the output of this command if the source file does not exist?
hdfs dfs -cp /data/file1.txt /backup/file1.txt

medium

A. cp: `/data/file1.txt`: No such file or directory

B. File copied successfully

C. Backup created with empty file

D. Command runs silently without error

Solution

1. Step 1: Understand the cp command behavior

   The hdfs dfs -cp command copies files within HDFS. If the source file does not exist, it throws an error.

2. Step 2: Match error message

   The error message for missing source file is "No such file or directory" as in cp: `/data/file1.txt`: No such file or directory.

3. Final Answer:

   cp: `/data/file1.txt`: No such file or directory -> Option A

4. Quick Check:

   Missing source file = error message [OK]

Hint: Missing source file causes 'No such file' error [OK]

Common Mistakes:

• Assuming silent success
• Expecting empty file creation
• Confusing error with copy success

4. Identify the error in this backup command:
hdfs dfs -copyToLocal /backup/data.txt /local/backup/

medium

A. The command should be -copyFromLocal to backup

B. The destination path should be HDFS, not local

C. The command is correct as is

D. The source path should be local, not HDFS

Solution

1. Step 1: Understand -copyToLocal command

   This command copies files from HDFS to local filesystem, useful for backup.

2. Step 2: Check source and destination paths

   Source is HDFS (/backup/data.txt), destination is local (/local/backup/), which matches command purpose.

3. Final Answer:

   The command is correct as is -> Option C

4. Quick Check:

   copyToLocal copies HDFS to local [OK]

Hint: copyToLocal copies from HDFS to local correctly [OK]

Common Mistakes:

• Confusing copyToLocal with copyFromLocal
• Swapping source and destination paths
• Thinking backup requires copyFromLocal

5. You want to create a disaster recovery plan that backs up daily changes in Hadoop data to a remote cluster. Which approach is best?

hard

A. Use hdfs dfs -rm to remove data after backup

B. Manually copy files with hdfs dfs -cp on local machine

C. Delete old backups and keep only latest file in HDFS

D. Use hdfs dfs -distcp to copy changed files to remote cluster daily

Solution

1. Step 1: Understand disaster recovery needs

   Disaster recovery requires copying data safely to a remote cluster regularly.

2. Step 2: Evaluate commands for remote copying

   hdfs dfs -distcp is designed for efficient distributed copying between clusters, ideal for daily backups.

3. Step 3: Check other options

   Manual copy is error-prone and slow; deleting backups or removing data is risky.

4. Final Answer:

   Use hdfs dfs -distcp to copy changed files to remote cluster daily -> Option D

5. Quick Check:

   distcp = best for remote backup [OK]

Hint: Use distcp for efficient remote cluster backup [OK]

Common Mistakes:

• Using local copy commands for remote backup
• Deleting backups too early
• Confusing remove command with backup