NameNode and DataNode roles in Hadoop - Mini Project: Build & Apply

Understanding NameNode and DataNode Roles in Hadoop

📖 Scenario: You are working with a Hadoop cluster that stores large amounts of data. The cluster has two main types of nodes: NameNode and DataNode. Understanding their roles helps you manage and analyze data efficiently.

🎯 Goal: Build a simple Python dictionary to represent the roles of NameNode and DataNode in a Hadoop cluster. Then, filter and display the key responsibilities of each node type.

📋 What You'll Learn

Create a dictionary called hadoop_nodes with keys 'NameNode' and 'DataNode' and their role descriptions as values.

Create a variable called filter_keyword to select roles containing the word 'manage'.

Use a dictionary comprehension to filter hadoop_nodes based on filter_keyword.

Print the filtered dictionary.

💡 Why This Matters

🌍 Real World

Hadoop clusters use NameNode and DataNode to manage and store big data efficiently. Understanding their roles helps in cluster management and troubleshooting.

💼 Career

Data engineers and big data analysts need to understand Hadoop architecture to optimize data storage and processing.

1

Create the Hadoop nodes dictionary

Create a dictionary called hadoop_nodes with these exact entries: 'NameNode' mapped to 'Manages the file system namespace and regulates access to files', and 'DataNode' mapped to 'Stores actual data and handles read/write requests from clients'.

Hadoop

# Create the hadoop_nodes dictionary with NameNode and DataNode roles
# Your code here

Need a hint?

Use curly braces {} to create a dictionary with the exact keys and values given.

2

Set the filter keyword

Create a variable called filter_keyword and set it to the string 'manage' (all lowercase).

Hadoop

hadoop_nodes = {
    'NameNode': 'Manages the file system namespace and regulates access to files',
    'DataNode': 'Stores actual data and handles read/write requests from clients'
}
# Create filter_keyword variable with value 'manage'
# Your code here

Need a hint?

Assign the string 'manage' to the variable filter_keyword.

3

Filter roles containing the keyword

Use a dictionary comprehension to create a new dictionary called filtered_roles that includes only those entries from hadoop_nodes where the role description contains the filter_keyword (case insensitive).

Hadoop

hadoop_nodes = {
    'NameNode': 'Manages the file system namespace and regulates access to files',
    'DataNode': 'Stores actual data and handles read/write requests from clients'
}
filter_keyword = 'manage'
# Create filtered_roles dictionary using dictionary comprehension
# Your code here

Need a hint?

Use hadoop_nodes.items() to loop through the dictionary and check if filter_keyword is in the lowercase version of the role description.

4

Print the filtered roles

Print the filtered_roles dictionary to display the filtered Hadoop node roles.

Hadoop

hadoop_nodes = {
    'NameNode': 'Manages the file system namespace and regulates access to files',
    'DataNode': 'Stores actual data and handles read/write requests from clients'
}
filter_keyword = 'manage'
filtered_roles = {node: role for node, role in hadoop_nodes.items() if filter_keyword in role.lower()}
# Print the filtered_roles dictionary
# Your code here

Need a hint?

Use print(filtered_roles) to show the filtered dictionary.