Block storage and replication in Hadoop - Step-by-Step Execution

Concept Flow - Block storage and replication

File Input

↓

Split into Blocks

↓

Store Blocks on DataNodes

↓

Replicate Blocks

↓

Maintain Replication Factor

↓

Handle Node Failures

↓

Re-replicate Missing Blocks

↓

File Available for Access

A file is split into blocks, stored on different nodes, and each block is replicated to ensure reliability and availability.

Execution Sample

Hadoop

file = 'bigdata.txt'
blocks = split(file, block_size=128 * 1024 * 1024)  # 128MB in bytes
for block in blocks:
    store(block, datanode)
    replicate(block, replication_factor=3)

This code splits a file into blocks and stores each block on data nodes with replication.

Execution Table

Step	Action	Block	DataNode(s) Stored	Replication Count	Result
1	Split file into blocks	block_1	-	-	File split into block_1, block_2, block_3
2	Store block_1	block_1	DataNode1	1	block_1 stored on DataNode1
3	Replicate block_1	block_1	DataNode2, DataNode3	3	block_1 replicated to DataNode2 and DataNode3
4	Store block_2	block_2	DataNode2	1	block_2 stored on DataNode2
5	Replicate block_2	block_2	DataNode1, DataNode3	3	block_2 replicated to DataNode1 and DataNode3
6	Store block_3	block_3	DataNode3	1	block_3 stored on DataNode3
7	Replicate block_3	block_3	DataNode1, DataNode2	3	block_3 replicated to DataNode1 and DataNode2
8	DataNode2 fails	block_1, block_2, block_3	DataNode2 lost	Replication drops to 2 for affected blocks	Replication factor below 3 for blocks on DataNode2
9	Re-replicate missing blocks	block_1, block_2, block_3	New DataNode4	3	Blocks re-replicated to maintain replication factor
10	File available	-	-	-	All blocks have replication factor 3, file accessible

💡 All blocks have replication factor 3, ensuring data reliability and availability

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 8	After Step 9	Final
block_1_replication_count	0	1	3	2	3	3
block_2_replication_count	0	1	3	2	3	3
block_3_replication_count	0	1	3	2	3	3
DataNodes_active	DataNode1, DataNode2, DataNode3	Same	Same	DataNode1, DataNode3	DataNode1, DataNode3, DataNode4	Same

Key Moments - 3 Insights

Why do we replicate blocks on multiple DataNodes?

Replication ensures that if one DataNode fails, the data is still available on other nodes, as shown in steps 3, 5, 7, and recovery in step 9.

What happens when a DataNode fails?

The replication count for blocks on that node drops (step 8), triggering re-replication to maintain the replication factor (step 9).

Why split a file into blocks instead of storing whole file on one node?

Splitting allows parallel storage and processing, improves fault tolerance, and fits large files across multiple nodes, as shown in step 1 and subsequent storage steps.

Visual Quiz - 3 Questions

Look at the execution table at step 3. What is the replication count of block_1 after replication?

A 1

B 2

C 3

D 0

💡 Hint

Check the 'Replication Count' column in row for step 3.

At which step does a DataNode failure occur causing replication to drop?

A Step 5

B Step 8

C Step 9

D Step 10

💡 Hint

Look for the step mentioning DataNode failure and replication drop.

If the replication factor was set to 2 instead of 3, how would the replication count change after step 3?

A Replication count would be 2

B Replication count would be 3

C Replication count would be 1

D Replication count would be 0

💡 Hint

Replication count matches the replication factor set during replication.

Concept Snapshot

Block storage splits large files into fixed-size blocks.
Each block is stored on multiple DataNodes.
Replication factor defines how many copies exist.
Replication ensures data availability and fault tolerance.
If a node fails, missing blocks are re-replicated.
This keeps data safe and accessible in Hadoop clusters.

Full Transcript

In Hadoop, large files are split into blocks for easier storage and processing. Each block is stored on a DataNode and replicated to other nodes to ensure reliability. The replication factor, often 3, means each block has three copies on different nodes. If a DataNode fails, the system detects the drop in replication and re-replicates the missing blocks to other nodes. This process keeps the file available and safe from data loss. The execution table shows steps from splitting the file, storing blocks, replicating them, handling node failure, and re-replication to maintain the replication factor.

Practice

1. What is the main purpose of splitting files into blocks in Hadoop's block storage?

easy

A. To delete duplicate files automatically

B. To reduce the file size permanently

C. To encrypt the data for security

D. To make large files easier to manage and process

Solution

1. Step 1: Understand file splitting in Hadoop

   Hadoop splits large files into smaller blocks to handle them efficiently across many machines.

2. Step 2: Identify the purpose of block storage

   This splitting helps in parallel processing and easier management of big data.

3. Final Answer:

   To make large files easier to manage and process -> Option D

4. Quick Check:

   Block storage = easier file management [OK]

Hint: Blocks split files for easier handling and processing [OK]

Common Mistakes:

• Thinking blocks reduce file size permanently
• Confusing block splitting with encryption
• Assuming blocks delete duplicates automatically

2. Which of the following is the correct way to set the replication factor to 3 in Hadoop configuration?

easy

A. replication=3

B. replication.factor=3

C. dfs.replication=3

D. dfs.replication_factor=3

Solution

1. Step 1: Recall Hadoop replication configuration property

   The correct property to set replication factor is dfs.replication.

2. Step 2: Match the exact syntax

   The value is assigned directly as dfs.replication=3 in the configuration file.

3. Final Answer:

   <code>dfs.replication=3</code> -> Option C

4. Quick Check:

   Replication factor property = dfs.replication [OK]

Hint: Use dfs.replication to set replication factor [OK]

Common Mistakes:

• Using incorrect property names like replication.factor
• Adding extra underscores in property name
• Omitting 'dfs.' prefix

3. Consider a Hadoop cluster with block size 128 MB and replication factor 3. If a file of 640 MB is stored, how many total blocks are stored across the cluster?

medium

A. 12 blocks

B. 15 blocks

C. 3 blocks

D. 5 blocks

Solution

1. Step 1: Calculate number of blocks for the file

   File size 640 MB divided by block size 128 MB gives 640 / 128 = 5 blocks.

2. Step 2: Multiply by replication factor

   Each block is replicated 3 times, so total blocks stored = 5 * 3 = 15 blocks.

3. Final Answer:

   15 blocks -> Option B

4. Quick Check:

   Total blocks = file blocks x replication factor = 15 [OK]

Hint: Multiply file blocks by replication factor [OK]

Common Mistakes:

• Counting only original blocks without replication
• Dividing file size by replication factor
• Using wrong block size in calculation

4. A Hadoop admin sets the replication factor to 2 but notices only one copy of each block is stored. What is the likely cause?

medium

A. The cluster has only one DataNode available

B. The block size is too large

C. The file is too small to replicate

D. Replication factor must be set to 1

Solution

1. Step 1: Understand replication behavior

   Replication factor sets desired copies, but actual copies depend on available DataNodes.

2. Step 2: Identify cluster limitation

   If only one DataNode exists, Hadoop cannot store multiple copies, so only one block copy is stored.

3. Final Answer:

   The cluster has only one DataNode available -> Option A

4. Quick Check:

   Replication copies depend on DataNodes count [OK]

Hint: Replication needs multiple DataNodes to store copies [OK]

Common Mistakes:

• Thinking block size affects replication copies
• Assuming small files don't replicate
• Believing replication factor must be 1

5. You want to optimize Hadoop storage for a critical dataset by increasing fault tolerance but have limited disk space. Which approach balances safety and storage best?

hard

A. Increase replication factor moderately and reduce block size slightly

B. Set replication factor to maximum and keep default block size

C. Keep replication factor low and increase block size greatly

D. Disable replication and rely on backups

Solution

1. Step 1: Understand trade-offs of replication and block size

   Higher replication increases fault tolerance but uses more disk space. Smaller block size can improve parallelism but may increase metadata overhead.

2. Step 2: Balance replication and block size

   Moderately increasing replication improves safety without excessive space use. Slightly reducing block size helps manage data efficiently.

3. Final Answer:

   Increase replication factor moderately and reduce block size slightly -> Option A

4. Quick Check:

   Moderate replication + smaller blocks = balanced safety and storage [OK]

Hint: Balance replication and block size for best storage use [OK]

Common Mistakes:

• Maxing replication wastes disk space
• Large blocks reduce fault tolerance
• Disabling replication risks data loss