Hadoop ecosystem overview - Mini Project: Build & Apply

Hadoop Ecosystem Overview

📖 Scenario: You work in a company that collects a lot of data from different sources like websites, sensors, and apps. You want to organize and understand this data using the Hadoop ecosystem, which is a set of tools that help store and analyze big data efficiently.

🎯 Goal: Build a simple overview of the Hadoop ecosystem by creating a dictionary that lists key Hadoop components and their main functions. Then, filter the components to find those related to data storage.

📋 What You'll Learn

Create a dictionary with Hadoop components and their descriptions

Create a list of components related to data storage

Use a comprehension to filter the storage components

Print the filtered list of storage components

💡 Why This Matters

🌍 Real World

Companies use the Hadoop ecosystem to store and process large amounts of data from many sources efficiently.

💼 Career

Understanding Hadoop components helps data engineers and analysts manage big data workflows and choose the right tools for data storage and processing.

1

Create a dictionary of Hadoop components

Create a dictionary called hadoop_components with these exact entries: 'HDFS': 'Distributed file system', 'YARN': 'Resource management', 'MapReduce': 'Data processing framework', 'Hive': 'Data warehouse', 'HBase': 'NoSQL database', 'Pig': 'Data flow scripting', 'Sqoop': 'Data transfer tool'.

Hadoop

# Create the dictionary hadoop_components with the specified entries
# Your code here

Need a hint?

Use curly braces {} to create the dictionary with the exact keys and values.

2

Create a list of storage-related components

Create a list called storage_components containing these exact strings: 'HDFS', 'HBase', and 'Hive'.

Hadoop

hadoop_components = {
    'HDFS': 'Distributed file system',
    'YARN': 'Resource management',
    'MapReduce': 'Data processing framework',
    'Hive': 'Data warehouse',
    'HBase': 'NoSQL database',
    'Pig': 'Data flow scripting',
    'Sqoop': 'Data transfer tool'
}

# Create the list storage_components with the specified items
# Your code here

Need a hint?

Use square brackets [] to create the list with the exact component names.

3

Filter storage components using comprehension

Create a list called filtered_storage using a list comprehension that includes only the keys from hadoop_components if they are in storage_components.

Hadoop

hadoop_components = {
    'HDFS': 'Distributed file system',
    'YARN': 'Resource management',
    'MapReduce': 'Data processing framework',
    'Hive': 'Data warehouse',
    'HBase': 'NoSQL database',
    'Pig': 'Data flow scripting',
    'Sqoop': 'Data transfer tool'
}

storage_components = ['HDFS', 'HBase', 'Hive']

# Use a list comprehension to create filtered_storage
# Your code here

Need a hint?

Use a list comprehension with for component in hadoop_components and an if condition.

4

Print the filtered storage components

Write a print statement to display the list filtered_storage.

Hadoop

hadoop_components = {
    'HDFS': 'Distributed file system',
    'YARN': 'Resource management',
    'MapReduce': 'Data processing framework',
    'Hive': 'Data warehouse',
    'HBase': 'NoSQL database',
    'Pig': 'Data flow scripting',
    'Sqoop': 'Data transfer tool'
}

storage_components = ['HDFS', 'HBase', 'Hive']

filtered_storage = [component for component in hadoop_components if component in storage_components]

# Print the filtered_storage list
# Your code here

Need a hint?

Use print(filtered_storage) to show the result.