Rack awareness in HDFS in Hadoop - Mini Project: Build & Apply

Rack Awareness in HDFS

📖 Scenario: You are working with a Hadoop cluster that has multiple racks. To improve data reliability and network efficiency, Hadoop uses rack awareness to decide where to place data blocks.Imagine you have a small cluster with nodes distributed across two racks. You want to simulate how Hadoop places data blocks considering rack awareness.

🎯 Goal: Build a simple Python simulation that models rack awareness in HDFS by assigning data blocks to nodes on different racks to ensure fault tolerance.

📋 What You'll Learn

Create a dictionary representing nodes and their racks

Set a replication factor for data blocks

Write logic to assign replicas to nodes ensuring replicas are on different racks

Print the final assignment of data blocks to nodes

💡 Why This Matters

🌍 Real World

Rack awareness in HDFS helps improve data reliability and network efficiency by spreading data copies across different racks to avoid data loss if a rack fails.

💼 Career

Understanding rack awareness is important for Hadoop administrators and data engineers to optimize cluster configuration and ensure fault tolerance.

1

Create the cluster node to rack mapping

Create a dictionary called nodes with these exact entries: 'node1': 'rack1', 'node2': 'rack1', 'node3': 'rack2', 'node4': 'rack2'.

Hadoop

# Create the nodes dictionary mapping nodes to racks
# Your code here

Need a hint?

Use a Python dictionary with node names as keys and rack names as values.

2

Set the replication factor

Create a variable called replication_factor and set it to 3.

Hadoop

nodes = {'node1': 'rack1', 'node2': 'rack1', 'node3': 'rack2', 'node4': 'rack2'}
# Set the replication factor to 3
# Your code here

Need a hint?

Just assign the number 3 to the variable replication_factor.

3

Assign replicas to nodes ensuring rack awareness

Write code to create a list called replica_nodes that contains exactly replication_factor nodes selected from nodes such that no two nodes are from the same rack. Use a simple approach to pick nodes from different racks.

Hadoop

nodes = {'node1': 'rack1', 'node2': 'rack1', 'node3': 'rack2', 'node4': 'rack2'}
replication_factor = 3
# Select replica_nodes list with nodes from different racks
# Your code here

Need a hint?

Use a loop over nodes.items() and keep track of racks used to avoid duplicates.

4

Print the replica node assignments

Write a print statement to display the list replica_nodes.

Hadoop

nodes = {'node1': 'rack1', 'node2': 'rack1', 'node3': 'rack2', 'node4': 'rack2'}
replication_factor = 3

replica_nodes = []
racks_used = set()
for node, rack in nodes.items():
    if rack not in racks_used:
        replica_nodes.append(node)
        racks_used.add(rack)
    if len(replica_nodes) == replication_factor:
        break

# Print the replica_nodes list
# Your code here

Need a hint?

The output should be a list of nodes from different racks. Since replication_factor is 3 but only 2 racks exist, the list will have 2 nodes.