Why HDFS handles petabyte-scale storage in Hadoop - See It in Action

Why HDFS Handles Petabyte-Scale Storage

📖 Scenario: Imagine you work for a company that collects huge amounts of data every day, like videos, logs, and sensor readings. You need a way to store all this data safely and access it quickly, even when it grows to petabytes (millions of gigabytes).

🎯 Goal: You will create a simple simulation to understand how HDFS (Hadoop Distributed File System) manages very large data by splitting it into blocks and storing copies across many machines.

📋 What You'll Learn

Create a dictionary to represent files and their sizes in gigabytes

Set a block size variable to split files into blocks

Calculate how many blocks each file needs

Print the number of blocks per file to see how HDFS handles large data

💡 Why This Matters

🌍 Real World

Big companies like Netflix and Facebook store massive data using HDFS to keep it safe and accessible.

💼 Career

Understanding HDFS block management is key for roles in big data engineering and data infrastructure.

1

Create a dictionary of files with their sizes

Create a dictionary called files with these exact entries: 'video1.mp4': 1500, 'log_data.txt': 3000, 'sensor_readings.csv': 500. The sizes are in gigabytes.

Hadoop

# Create the dictionary called files with given file names and sizes
# Your code here

Hint

Use curly braces {} to create a dictionary with file names as keys and sizes as values.

2

Set the block size for splitting files

Create a variable called block_size_gb and set it to 128 to represent the block size in gigabytes.

Hadoop

files = {'video1.mp4': 1500, 'log_data.txt': 3000, 'sensor_readings.csv': 500}
# Set block_size_gb to 128
# Your code here

Hint

Just assign the number 128 to the variable block_size_gb.

3

Calculate the number of blocks for each file

Create a new dictionary called file_blocks. Use a for loop with variables file_name and size to iterate over files.items(). For each file, calculate the number of blocks by dividing size by block_size_gb and rounding up using math.ceil(). Store the result in file_blocks[file_name]. Import the math module first.

Hadoop

files = {'video1.mp4': 1500, 'log_data.txt': 3000, 'sensor_readings.csv': 500}
block_size_gb = 128
import math
# Calculate number of blocks per file and store in file_blocks
# Your code here

Hint

Use math.ceil() to round up the division result so partial blocks count as full blocks.

4

Print the number of blocks per file

Use a for loop with variables file_name and blocks to iterate over file_blocks.items(). Print each file name and its number of blocks in the format: File: video1.mp4, Blocks: 12.

Hadoop

files = {'video1.mp4': 1500, 'log_data.txt': 3000, 'sensor_readings.csv': 500}
block_size_gb = 128
import math
file_blocks = {}
for file_name, size in files.items():
    file_blocks[file_name] = math.ceil(size / block_size_gb)
# Print file names and their block counts
# Your code here

Hint

Use an f-string inside the print statement to format the output exactly as shown.

Practice

1. Why does HDFS split large files into smaller blocks when storing data?

easy

A. To make files harder to access

B. To distribute data across many machines for better storage and speed

C. To reduce the total amount of data stored

D. To encrypt data automatically

Solution

1. Step 1: Understand HDFS file storage method

   HDFS breaks large files into blocks to spread them across multiple machines.

2. Step 2: Recognize benefits of block splitting

   This spreading allows parallel access and efficient storage management.

3. Final Answer:

   To distribute data across many machines for better storage and speed -> Option B

4. Quick Check:

   Splitting files = distributed storage [OK]

Hint: Think: splitting helps spread data across machines [OK]

Common Mistakes:

• Thinking splitting reduces data size
• Assuming splitting encrypts data
• Believing splitting makes access harder

2. Which of the following is the correct way HDFS ensures data reliability?

easy

A. By replicating data blocks multiple times across different machines

B. By compressing data blocks to save space

C. By storing only one copy of each data block

D. By deleting old data blocks automatically

Solution

1. Step 1: Identify HDFS data protection method

   HDFS keeps multiple copies (replicas) of each block on different machines.

2. Step 2: Understand why replication is used

   This replication protects data if one machine fails.

3. Final Answer:

   By replicating data blocks multiple times across different machines -> Option A

4. Quick Check:

   Replication = data safety [OK]

Hint: Remember: multiple copies protect data [OK]

Common Mistakes:

• Choosing single copy storage
• Confusing compression with reliability
• Thinking deletion ensures reliability

3. Consider this simplified code snippet representing HDFS block replication logic:

blocks = ['block1', 'block2']
replication_factor = 3
replicated_blocks = [block for block in blocks for _ in range(replication_factor)]
print(replicated_blocks)

What will be the output?

medium

A. ['block1', 'block2']

B. ['block1', 'block1', 'block2', 'block2', 'block2', 'block2']

C. ['block1', 'block2', 'block1', 'block2', 'block1', 'block2']

D. ['block1', 'block1', 'block1', 'block2', 'block2', 'block2']

Solution

1. Step 1: Understand the list comprehension structure

   For each block in blocks, the inner loop repeats replication_factor times, adding the block multiple times.

2. Step 2: Apply replication factor to each block

   'block1' repeats 3 times, then 'block2' repeats 3 times, resulting in 6 elements total.

3. Final Answer:

   ['block1', 'block1', 'block1', 'block2', 'block2', 'block2'] -> Option D

4. Quick Check:

   Replication factor 3 means 3 copies each [OK]

Hint: Repeat each block replication_factor times [OK]

Common Mistakes:

• Mixing order of repeated blocks
• Repeating blocks fewer or more times
• Confusing inner and outer loops

4. This code tries to simulate HDFS block replication but has a bug:

blocks = ['b1', 'b2']
replication_factor = 2
replicated = []
for block in blocks:
    for i in range(replication_factor):
        replicated.append(blocks[i])
print(replicated)

What is the main error?

medium

A. Using blocks[i] instead of block inside the inner loop

B. Wrong replication_factor value

C. Missing initialization of replicated list

D. Incorrect print statement syntax

Solution

1. Step 1: Analyze the inner loop append statement

   The code appends blocks[i], which uses the loop index i, not the current block variable.

2. Step 2: Understand correct variable usage

   It should append the current block variable to replicate it, not blocks[i].

3. Final Answer:

   Using blocks[i] instead of block inside the inner loop -> Option A

4. Quick Check:

   Append current block, not indexed block [OK]

Hint: Append 'block', not blocks[i], inside inner loop [OK]

Common Mistakes:

• Confusing loop variables
• Changing replication_factor unnecessarily
• Forgetting to initialize list

5. HDFS stores petabytes of data by splitting files into blocks and replicating them. If a cluster has 100 machines and each block is 128MB with replication factor 3, what is the maximum raw data size (in terabytes) the cluster can store assuming all machines have 1TB disk space fully used for HDFS?

hard

A. Approximately 900 TB

B. Approximately 100 TB

C. Approximately 33.3 TB

D. Approximately 300 TB

Solution

1. Step 1: Calculate total raw storage in TB

   100 machines * 1TB each = 100 TB total physical storage.

2. Step 2: Account for replication factor

   Replication factor 3 means each data block is stored 3 times, so usable raw data = 100 TB / 3 ≈ 33.3 TB.

3. Final Answer:

   Approximately 33.3 TB -> Option C

4. Quick Check:

   Total storage ÷ replication factor = usable data [OK]

Hint: Divide total storage by replication factor [OK]

Common Mistakes:

• Ignoring replication factor
• Multiplying instead of dividing
• Confusing block size with total storage