Backup and disaster recovery in Hadoop - Mini Project: Build & Apply

Backup and Disaster Recovery in Hadoop

📖 Scenario: You work as a data engineer managing a Hadoop cluster that stores important company data. To protect against data loss from hardware failure or accidental deletion, you need to create a backup of critical files and set up a simple disaster recovery plan.

🎯 Goal: Build a basic Hadoop backup and disaster recovery script that copies important files from the main Hadoop Distributed File System (HDFS) directory to a backup directory. Then, verify the backup by listing the files in the backup location.

📋 What You'll Learn

Create a list of important HDFS file paths to back up

Set a backup directory path variable

Write a loop to copy each important file to the backup directory using Hadoop commands

Print the list of files in the backup directory to confirm backup success

💡 Why This Matters

🌍 Real World

Backing up data in Hadoop is critical to prevent data loss from hardware failures or accidental deletions. This project simulates a simple backup and recovery process.

💼 Career

Data engineers and Hadoop administrators regularly create backup and disaster recovery plans to ensure data safety and business continuity.

1

Create a list of important HDFS files

Create a list called important_files with these exact HDFS file paths: /data/sales.csv, /data/customers.csv, and /data/products.csv.

Hadoop

# Create the list of important HDFS files
# Your code here

Hint

Use square brackets to create a list and include the file paths as strings.

2

Set the backup directory path

Create a variable called backup_dir and set it to the string /backup/ which will be the HDFS backup directory.

Hadoop

important_files = ['/data/sales.csv', '/data/customers.csv', '/data/products.csv']
# Set the backup directory path
# Your code here

Hint

Assign the string '/backup/' to the variable backup_dir.

3

Copy important files to the backup directory

Write a for loop using the variable file_path to iterate over important_files. Inside the loop, use the Hadoop command hadoop fs -cp to copy each file_path to backup_dir. Use an f-string to build the command string exactly as: hadoop fs -cp {file_path} {backup_dir}. Use os.system() to run the command.

Hadoop

important_files = ['/data/sales.csv', '/data/customers.csv', '/data/products.csv']
backup_dir = '/backup/'
import os
# Copy each important file to the backup directory
# Your code here

Hint

Use a for loop with file_path and build the copy command with an f-string. Then run it with os.system().

4

List files in the backup directory to verify

Use os.system() to run the Hadoop command hadoop fs -ls {backup_dir} to list the files in the backup directory and confirm the backup.

Hadoop

important_files = ['/data/sales.csv', '/data/customers.csv', '/data/products.csv']
backup_dir = '/backup/'
import os
for file_path in important_files:
    command = f'hadoop fs -cp {file_path} {backup_dir}'
    os.system(command)
# List files in backup directory
# Your code here

Hint

Use os.system() with the list command to show files in the backup directory.

Practice

1. What is the main purpose of backup in Hadoop's disaster recovery?

easy

A. To delete old data automatically

B. To save copies of data to prevent loss

C. To speed up data processing

D. To encrypt data for security

Solution

1. Step 1: Understand backup purpose

   Backup means saving copies of data so it can be recovered if lost.

2. Step 2: Match with options

   Only To save copies of data to prevent loss describes saving copies to prevent data loss.

3. Final Answer:

   To save copies of data to prevent loss -> Option B

4. Quick Check:

   Backup = Save copies [OK]

Hint: Backup means saving data copies to avoid loss [OK]

Common Mistakes:

• Confusing backup with data processing speed
• Thinking backup deletes data
• Mixing backup with encryption

2. Which Hadoop command is used to copy files from HDFS to local for backup?

easy

A. hdfs dfs -get

B. hdfs dfs -ls

C. hdfs dfs -rm

D. hdfs dfs -put

Solution

1. Step 1: Identify command to copy from HDFS to local

   The command hdfs dfs -get copies files from HDFS to local filesystem.

2. Step 2: Check other commands

   -put copies local to HDFS, -rm removes files, -ls lists files.

3. Final Answer:

   hdfs dfs -get -> Option A

4. Quick Check:

   Copy HDFS to local = -get [OK]

Hint: Use -get to copy from HDFS to local [OK]

Common Mistakes:

• Using -put instead of -get
• Confusing -rm with copy commands
• Thinking -ls copies files

3. What will be the output of this command if the source file does not exist?
hdfs dfs -cp /data/file1.txt /backup/file1.txt

medium

A. cp: `/data/file1.txt`: No such file or directory

B. File copied successfully

C. Backup created with empty file

D. Command runs silently without error

Solution

1. Step 1: Understand the cp command behavior

   The hdfs dfs -cp command copies files within HDFS. If the source file does not exist, it throws an error.

2. Step 2: Match error message

   The error message for missing source file is "No such file or directory" as in cp: `/data/file1.txt`: No such file or directory.

3. Final Answer:

   cp: `/data/file1.txt`: No such file or directory -> Option A

4. Quick Check:

   Missing source file = error message [OK]

Hint: Missing source file causes 'No such file' error [OK]

Common Mistakes:

• Assuming silent success
• Expecting empty file creation
• Confusing error with copy success

4. Identify the error in this backup command:
hdfs dfs -copyToLocal /backup/data.txt /local/backup/

medium

A. The command should be -copyFromLocal to backup

B. The destination path should be HDFS, not local

C. The command is correct as is

D. The source path should be local, not HDFS

Solution

1. Step 1: Understand -copyToLocal command

   This command copies files from HDFS to local filesystem, useful for backup.

2. Step 2: Check source and destination paths

   Source is HDFS (/backup/data.txt), destination is local (/local/backup/), which matches command purpose.

3. Final Answer:

   The command is correct as is -> Option C

4. Quick Check:

   copyToLocal copies HDFS to local [OK]

Hint: copyToLocal copies from HDFS to local correctly [OK]

Common Mistakes:

• Confusing copyToLocal with copyFromLocal
• Swapping source and destination paths
• Thinking backup requires copyFromLocal

5. You want to create a disaster recovery plan that backs up daily changes in Hadoop data to a remote cluster. Which approach is best?

hard

A. Use hdfs dfs -rm to remove data after backup

B. Manually copy files with hdfs dfs -cp on local machine

C. Delete old backups and keep only latest file in HDFS

D. Use hdfs dfs -distcp to copy changed files to remote cluster daily

Solution

1. Step 1: Understand disaster recovery needs

   Disaster recovery requires copying data safely to a remote cluster regularly.

2. Step 2: Evaluate commands for remote copying

   hdfs dfs -distcp is designed for efficient distributed copying between clusters, ideal for daily backups.

3. Step 3: Check other options

   Manual copy is error-prone and slow; deleting backups or removing data is risky.

4. Final Answer:

   Use hdfs dfs -distcp to copy changed files to remote cluster daily -> Option D

5. Quick Check:

   distcp = best for remote backup [OK]

Hint: Use distcp for efficient remote cluster backup [OK]

Common Mistakes:

• Using local copy commands for remote backup
• Deleting backups too early
• Confusing remove command with backup