Why HDFS handles petabyte-scale storage in Hadoop - Test Your Understanding

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create an HDFS directory using the Hadoop command.

Hadoop

hdfs dfs -mkdir [1]

Drag options to blanks, or click blank then click option'

A/tmp/file

B/user/data

C-ls

Ddatafile.txt

Attempts:

3 left

💡 Hint

Common Mistakes

Using a filename instead of a directory path

Using a command option instead of a path

The -mkdir command creates a directory in HDFS. Here, /user/data is a valid directory path.

2fill in blank

medium

Complete the code to set the replication factor for a file in HDFS.

Hadoop

hdfs dfs -setrep [1] /user/data/file.txt

Drag options to blanks, or click blank then click option'

A0

B-rm

Cfile.txt

D3

Attempts:

3 left

💡 Hint

Common Mistakes

Using zero or negative numbers

Using command options instead of numbers

Setting replication factor to 3 means each data block is stored on 3 different nodes for fault tolerance.

3fill in blank

hard

Fix the error in the command to copy a local file to HDFS.

Hadoop

hdfs dfs -put [1] /user/data/

Drag options to blanks, or click blank then click option'

A/local/path/file.txt

B-copyFromLocal

Chdfs://file.txt

D/user/data/file.txt

Attempts:

3 left

💡 Hint

Common Mistakes

Using HDFS path as source

Using command options as source

The -put command copies a local file to HDFS. The source must be a local file path.

4fill in blank

hard

Fill both blanks to create a dictionary comprehension that maps file names to their sizes if size is greater than 1000 bytes.

Hadoop

file_sizes = [1]: [2] for [1], [2] in files.items() if [2] > 1000}

Drag options to blanks, or click blank then click option'

Afilename

Bsize

Cfiles

Ditems

Attempts:

3 left

💡 Hint

Common Mistakes

Using the dictionary name as variable

Mixing keys and values

The dictionary comprehension iterates over files.items() with filename and size as key and value. It filters sizes greater than 1000.

5fill in blank

hard

Fill all three blanks to create a dictionary comprehension that maps uppercase file names to their sizes if size is less than 5000 bytes.

Hadoop

filtered_files = [1]: [2] for [3], [2] in files.items() if [2] < 5000}

Drag options to blanks, or click blank then click option'

Afilename.upper()

Bsize

Cfilename

Dfiles

Attempts:

3 left

💡 Hint

Common Mistakes

Using dictionary name as key

Not converting to uppercase

Mixing variable names

This comprehension converts file names to uppercase as keys, keeps sizes as values, and filters sizes less than 5000.

Practice

1. Why does HDFS split large files into smaller blocks when storing data?

easy

A. To make files harder to access

B. To distribute data across many machines for better storage and speed

C. To reduce the total amount of data stored

D. To encrypt data automatically

Solution

1. Step 1: Understand HDFS file storage method

   HDFS breaks large files into blocks to spread them across multiple machines.

2. Step 2: Recognize benefits of block splitting

   This spreading allows parallel access and efficient storage management.

3. Final Answer:

   To distribute data across many machines for better storage and speed -> Option B

4. Quick Check:

   Splitting files = distributed storage [OK]

Hint: Think: splitting helps spread data across machines [OK]

Common Mistakes:

• Thinking splitting reduces data size
• Assuming splitting encrypts data
• Believing splitting makes access harder

2. Which of the following is the correct way HDFS ensures data reliability?

easy

A. By replicating data blocks multiple times across different machines

B. By compressing data blocks to save space

C. By storing only one copy of each data block

D. By deleting old data blocks automatically

Solution

1. Step 1: Identify HDFS data protection method

   HDFS keeps multiple copies (replicas) of each block on different machines.

2. Step 2: Understand why replication is used

   This replication protects data if one machine fails.

3. Final Answer:

   By replicating data blocks multiple times across different machines -> Option A

4. Quick Check:

   Replication = data safety [OK]

Hint: Remember: multiple copies protect data [OK]

Common Mistakes:

• Choosing single copy storage
• Confusing compression with reliability
• Thinking deletion ensures reliability

3. Consider this simplified code snippet representing HDFS block replication logic:

blocks = ['block1', 'block2']
replication_factor = 3
replicated_blocks = [block for block in blocks for _ in range(replication_factor)]
print(replicated_blocks)

What will be the output?

medium

A. ['block1', 'block2']

B. ['block1', 'block1', 'block2', 'block2', 'block2', 'block2']

C. ['block1', 'block2', 'block1', 'block2', 'block1', 'block2']

D. ['block1', 'block1', 'block1', 'block2', 'block2', 'block2']

Solution

1. Step 1: Understand the list comprehension structure

   For each block in blocks, the inner loop repeats replication_factor times, adding the block multiple times.

2. Step 2: Apply replication factor to each block

   'block1' repeats 3 times, then 'block2' repeats 3 times, resulting in 6 elements total.

3. Final Answer:

   ['block1', 'block1', 'block1', 'block2', 'block2', 'block2'] -> Option D

4. Quick Check:

   Replication factor 3 means 3 copies each [OK]

Hint: Repeat each block replication_factor times [OK]

Common Mistakes:

• Mixing order of repeated blocks
• Repeating blocks fewer or more times
• Confusing inner and outer loops

4. This code tries to simulate HDFS block replication but has a bug:

blocks = ['b1', 'b2']
replication_factor = 2
replicated = []
for block in blocks:
    for i in range(replication_factor):
        replicated.append(blocks[i])
print(replicated)

What is the main error?

medium

A. Using blocks[i] instead of block inside the inner loop

B. Wrong replication_factor value

C. Missing initialization of replicated list

D. Incorrect print statement syntax

Solution

1. Step 1: Analyze the inner loop append statement

   The code appends blocks[i], which uses the loop index i, not the current block variable.

2. Step 2: Understand correct variable usage

   It should append the current block variable to replicate it, not blocks[i].

3. Final Answer:

   Using blocks[i] instead of block inside the inner loop -> Option A

4. Quick Check:

   Append current block, not indexed block [OK]

Hint: Append 'block', not blocks[i], inside inner loop [OK]

Common Mistakes:

• Confusing loop variables
• Changing replication_factor unnecessarily
• Forgetting to initialize list

5. HDFS stores petabytes of data by splitting files into blocks and replicating them. If a cluster has 100 machines and each block is 128MB with replication factor 3, what is the maximum raw data size (in terabytes) the cluster can store assuming all machines have 1TB disk space fully used for HDFS?

hard

A. Approximately 900 TB

B. Approximately 100 TB

C. Approximately 33.3 TB

D. Approximately 300 TB

Solution

1. Step 1: Calculate total raw storage in TB

   100 machines * 1TB each = 100 TB total physical storage.

2. Step 2: Account for replication factor

   Replication factor 3 means each data block is stored 3 times, so usable raw data = 100 TB / 3 ≈ 33.3 TB.

3. Final Answer:

   Approximately 33.3 TB -> Option C

4. Quick Check:

   Total storage ÷ replication factor = usable data [OK]

Hint: Divide total storage by replication factor [OK]

Common Mistakes:

• Ignoring replication factor
• Multiplying instead of dividing
• Confusing block size with total storage