Rack awareness in HDFS in Hadoop - Step-by-Step Execution

Concept Flow - Rack awareness in HDFS

Client requests file write

↓

NameNode identifies racks

↓

Select DataNodes on different racks

↓

Write replicas to chosen DataNodes

↓

Client confirms write success

↓

Done

The client asks to write data, the NameNode picks DataNodes on different racks to store replicas, ensuring fault tolerance across racks.

Execution Sample

Hadoop

replica_placement = []
racks = ['rack1', 'rack2', 'rack3']
data_nodes = {'rack1': ['DN1', 'DN2'], 'rack2': ['DN3'], 'rack3': ['DN4', 'DN5']}

for rack in racks:
    replica_placement.append(data_nodes[rack][0])

This code selects one DataNode from each rack to place replicas for fault tolerance.

Execution Table

Step	Rack	DataNode Selected	Replica Placement List	Action
1	rack1	DN1	['DN1']	Select first DataNode from rack1
2	rack2	DN3	['DN1', 'DN3']	Select first DataNode from rack2
3	rack3	DN4	['DN1', 'DN3', 'DN4']	Select first DataNode from rack3
4	-	-	['DN1', 'DN3', 'DN4']	Replica placement complete

💡 All racks processed, replicas placed on one DataNode per rack

Variable Tracker

Variable	Start	After 1	After 2	After 3	Final
replica_placement	[]	['DN1']	['DN1', 'DN3']	['DN1', 'DN3', 'DN4']	['DN1', 'DN3', 'DN4']

Key Moments - 3 Insights

Why does HDFS place replicas on different racks?

Placing replicas on different racks protects data if one rack fails, as shown in execution_table rows 1-3 where replicas are chosen from separate racks.

What happens if all replicas are on the same rack?

If all replicas are on the same rack, a rack failure can cause data loss. The code and execution_table show how HDFS avoids this by selecting DataNodes from different racks.

Why do we select only one DataNode per rack in this example?

Selecting one DataNode per rack ensures replicas are spread out. This simple example picks the first DataNode per rack, as seen in the loop in execution_sample and execution_table.

Visual Quiz - 3 Questions

Look at the execution_table, which DataNode is selected from rack2 at step 2?

A DN2

B DN4

C DN3

D DN1

💡 Hint

Check the 'DataNode Selected' column at step 2 in execution_table

At which step is the replica placement complete according to the execution_table?

A Step 3

B Step 4

C Step 2

D Step 1

💡 Hint

Look at the 'Action' column for the step indicating completion

If rack3 had no DataNodes, how would the replica_placement list change after step 3?

A It would have only ['DN1', 'DN3']

B It would have ['DN1', 'DN3', 'DN4']

C It would be empty

D It would have duplicates

💡 Hint

Refer to variable_tracker and consider missing DataNodes in rack3

Concept Snapshot

Rack awareness in HDFS:
- NameNode knows rack locations of DataNodes
- Replicas placed on different racks
- Protects data from rack failure
- Select one DataNode per rack for replicas
- Ensures fault tolerance and availability

Full Transcript

Rack awareness in HDFS means the system knows which DataNodes belong to which racks. When a client writes data, the NameNode chooses DataNodes on different racks to store replicas. This spreads copies across racks to protect against rack failures. The example code shows selecting one DataNode from each rack. The execution table traces this selection step-by-step, showing the replica placement list growing as each rack is processed. Key moments clarify why spreading replicas matters and how the selection works. The visual quiz tests understanding of which DataNodes are chosen and what happens if racks lack DataNodes. The snapshot summarizes the main points for quick review.