HDFS high availability in Hadoop - Mini Project: Build & Apply

HDFS High Availability Setup Simulation

📖 Scenario: You are working in a company that uses Hadoop to store big data. To make sure the data is always available even if one server fails, you want to set up HDFS High Availability. This means having two NameNodes where one is active and the other is standby, so the system keeps working without interruption.

🎯 Goal: You will simulate the setup of HDFS High Availability by creating a data structure to represent the two NameNodes and their states, configure a threshold for failover, write logic to switch the active NameNode if needed, and finally display the current active NameNode.

📋 What You'll Learn

Create a dictionary called name_nodes with two entries: 'nn1' and 'nn2', each having a state of 'active' or 'standby'.

Create a variable called failover_threshold set to 1 to represent the number of failures allowed before switching.

Write a function called check_and_failover that takes name_nodes and failover_threshold and switches the active NameNode to standby and standby to active if the threshold is reached.

Print the name of the currently active NameNode after running the failover check.

💡 Why This Matters

🌍 Real World

HDFS High Availability is critical in big data systems to avoid downtime and data loss by having backup NameNodes ready to take over.

💼 Career

Understanding how to manage and simulate failover in distributed systems is important for roles like Data Engineer, Hadoop Administrator, and Big Data Developer.

1

Create the NameNodes dictionary

Create a dictionary called name_nodes with these exact entries: 'nn1': 'active' and 'nn2': 'standby'.

Hadoop

# Create the name_nodes dictionary with 'nn1' as 'active' and 'nn2' as 'standby'
# Your code here

Need a hint?

Use curly braces to create a dictionary. Assign 'active' to 'nn1' and 'standby' to 'nn2'.

2

Set the failover threshold

Create a variable called failover_threshold and set it to 1.

Hadoop

name_nodes = {'nn1': 'active', 'nn2': 'standby'}
# Set failover_threshold to 1
# Your code here

Need a hint?

Just assign the number 1 to the variable failover_threshold.

3

Write the failover logic function

Write a function called check_and_failover that takes name_nodes and failover_threshold as parameters. Inside, if failover_threshold is 1, switch the 'active' NameNode to 'standby' and the 'standby' NameNode to 'active'.

Hadoop

name_nodes = {'nn1': 'active', 'nn2': 'standby'}
failover_threshold = 1

# Define check_and_failover function
# Your code here

Need a hint?

Use a for loop to check each NameNode's state and switch it accordingly if the threshold is 1.

4

Run failover check and print active NameNode

Call the function check_and_failover(name_nodes, failover_threshold). Then, print the name of the NameNode that is currently 'active'.

Hadoop

name_nodes = {'nn1': 'active', 'nn2': 'standby'}
failover_threshold = 1

def check_and_failover(name_nodes, failover_threshold):
    if failover_threshold == 1:
        for nn, state in name_nodes.items():
            if state == 'active':
                name_nodes[nn] = 'standby'
            else:
                name_nodes[nn] = 'active'

# Call check_and_failover and print the active NameNode
# Your code here

Need a hint?

After calling the function, loop through name_nodes and print the key where the value is 'active'.