Hadoop in cloud (EMR, Dataproc, HDInsight) - Mini Project: Build & Apply

Analyzing Sales Data Using Hadoop in Cloud (EMR, Dataproc, HDInsight)

📖 Scenario: You work for a retail company that wants to analyze its sales data stored in the cloud. The data is large, so you will use Hadoop services like AWS EMR, Google Dataproc, or Azure HDInsight to process it efficiently.This project guides you through setting up a simple sales dataset, configuring a threshold for high sales, processing the data using Hadoop MapReduce style logic, and finally outputting the filtered results.

🎯 Goal: Build a simple Hadoop-style data processing pipeline in Python that mimics how cloud Hadoop services process big data. You will create sales data, set a sales threshold, filter sales above the threshold, and print the results.

📋 What You'll Learn

Create a dictionary with sales data for products and their sales numbers

Add a sales threshold variable to filter high sales

Use a for loop to filter products with sales above the threshold

Print the filtered high sales products and their sales

💡 Why This Matters

🌍 Real World

Retail companies use cloud Hadoop services like EMR, Dataproc, or HDInsight to process large sales data quickly and find important trends.

💼 Career

Data analysts and engineers use these cloud tools to handle big data and extract useful insights for business decisions.

1

Create the sales data dictionary

Create a dictionary called sales_data with these exact entries: 'Laptop': 120, 'Smartphone': 250, 'Tablet': 90, 'Headphones': 150, 'Smartwatch': 80.

Hadoop

# Create the sales_data dictionary with product names and sales numbers
# Your code here

Hint

Use curly braces {} to create a dictionary with product names as keys and sales numbers as values.

2

Set the sales threshold

Create a variable called sales_threshold and set it to 100 to filter products with sales above this number.

Hadoop

sales_data = {'Laptop': 120, 'Smartphone': 250, 'Tablet': 90, 'Headphones': 150, 'Smartwatch': 80}
# Create the sales_threshold variable and set it to 100
# Your code here

Hint

Just assign the number 100 to the variable sales_threshold.

3

Filter products with sales above the threshold

Create an empty dictionary called high_sales. Use a for loop with variables product and sales to iterate over sales_data.items(). Inside the loop, add products with sales greater than sales_threshold to high_sales.

Hadoop

sales_data = {'Laptop': 120, 'Smartphone': 250, 'Tablet': 90, 'Headphones': 150, 'Smartwatch': 80}
sales_threshold = 100
# Create an empty dictionary high_sales
# Use a for loop to add products with sales > sales_threshold to high_sales
# Your code here

Hint

Use for product, sales in sales_data.items(): to loop through the dictionary. Use an if statement to check sales.

4

Print the filtered high sales products

Write a print statement to display the high_sales dictionary.

Hadoop

sales_data = {'Laptop': 120, 'Smartphone': 250, 'Tablet': 90, 'Headphones': 150, 'Smartwatch': 80}
sales_threshold = 100
high_sales = {}
for product, sales in sales_data.items():
    if sales > sales_threshold:
        high_sales[product] = sales
# Print the high_sales dictionary
# Your code here

Hint

Use print(high_sales) to show the filtered results.

Practice

1. Which cloud service is NOT commonly used to run Hadoop clusters in the cloud?

easy

A. AWS EMR

B. Google Dataproc

C. Azure HDInsight

D. Firebase Realtime Database

Solution

1. Step 1: Identify common Hadoop cloud services

   AWS EMR, Google Dataproc, and Azure HDInsight are popular cloud services for Hadoop clusters.

2. Step 2: Recognize Firebase's purpose

   Firebase Realtime Database is a NoSQL database service, not a Hadoop cluster service.

3. Final Answer:

   Firebase Realtime Database -> Option D

4. Quick Check:

   Hadoop clusters use EMR, Dataproc, HDInsight, not Firebase [OK]

Hint: Remember: Firebase is for databases, not Hadoop clusters [OK]

Common Mistakes:

• Confusing Firebase with Hadoop services
• Thinking all cloud services support Hadoop
• Mixing database services with cluster services

2. Which of the following is the correct way to start an AWS EMR cluster using AWS CLI?

easy

A. aws emr launch --cluster TestCluster --app Hadoop

B. aws emr create-cluster --name "TestCluster" --release-label emr-6.5.0 --applications Name=Hadoop

C. aws emr start-cluster --cluster-name TestCluster --version 6.5.0

D. aws emr init --name TestCluster --app Hadoop

Solution

1. Step 1: Understand AWS EMR CLI command structure

   The correct command to create a cluster is 'aws emr create-cluster' with parameters for name, release label, and applications.

2. Step 2: Check each option's syntax

   aws emr create-cluster --name "TestCluster" --release-label emr-6.5.0 --applications Name=Hadoop matches the correct syntax. Options A, B, and D use incorrect commands or parameters.

3. Final Answer:

   aws emr create-cluster --name "TestCluster" --release-label emr-6.5.0 --applications Name=Hadoop -> Option B

4. Quick Check:

   Use 'create-cluster' to start EMR clusters [OK]

Hint: Remember: 'create-cluster' starts EMR clusters via CLI [OK]

Common Mistakes:

• Using wrong CLI commands like 'start-cluster'
• Missing required parameters like release-label
• Confusing application names or syntax

3. What will be the output of this Python code snippet using Google Cloud Dataproc API to list clusters?

from google.cloud import dataproc_v1
client = dataproc_v1.ClusterControllerClient()
project_id = "my-project"
region = "us-central1"
clusters = client.list_clusters(project_id=project_id, region=region)
print([cluster.cluster_name for cluster in clusters])

medium

A. A list of cluster names in the 'us-central1' region for 'my-project'

B. A syntax error due to missing import

C. An empty list if no clusters exist

D. A runtime error because 'list_clusters' requires more parameters

Solution

1. Step 1: Understand the code's purpose

   The code uses Google Cloud Dataproc client to list clusters in a project and region.

2. Step 2: Analyze the output

   The print statement extracts cluster names into a list. If clusters exist, it prints their names; otherwise, an empty list.

3. Final Answer:

   A list of cluster names in the 'us-central1' region for 'my-project' -> Option A

4. Quick Check:

   Dataproc list_clusters returns cluster objects with names [OK]

Hint: List clusters returns objects; extract names with list comprehension [OK]

Common Mistakes:

• Assuming list_clusters returns strings directly
• Expecting errors without proper context
• Confusing parameters needed for list_clusters

4. You try to submit a Hadoop job on Azure HDInsight using this command:

hadoop jar example.jar com.example.WordCount input output

But get an error: 'command not found'. What is the likely cause?

medium

A. The input directory does not exist

B. The jar file 'example.jar' is missing

C. Hadoop is not installed or not in PATH on the HDInsight cluster

D. The output directory already exists

Solution

1. Step 1: Understand the error message

   'command not found' means the shell cannot find the 'hadoop' command.

2. Step 2: Identify possible reasons

   This usually happens if Hadoop is not installed or its binary path is not set in the environment PATH variable.

3. Final Answer:

   Hadoop is not installed or not in PATH on the HDInsight cluster -> Option C

4. Quick Check:

   'command not found' means missing executable in PATH [OK]

Hint: Check if 'hadoop' command runs before job submission [OK]

Common Mistakes:

• Assuming jar or input/output issues cause 'command not found'
• Ignoring environment PATH settings
• Confusing runtime errors with command errors

5. You want to run a Hadoop job on AWS EMR that reads data from S3, processes it, and writes output back to S3. Which of these steps is NOT required?

hard

A. Manually install Hadoop on each EMR node after cluster creation

B. Configure the EMR cluster with IAM roles to access S3 buckets

C. Create an EMR cluster with Hadoop installed

D. Submit the Hadoop job referencing S3 paths for input and output

Solution

1. Step 1: Understand EMR cluster setup

   EMR clusters come with Hadoop pre-installed; manual installation is not needed.

2. Step 2: Review required steps for S3 integration

   IAM roles must allow S3 access, and job submission must use S3 paths.

3. Final Answer:

   Manually install Hadoop on each EMR node after cluster creation -> Option A

4. Quick Check:

   EMR auto-installs Hadoop; manual install unnecessary [OK]

Hint: EMR clusters have Hadoop ready; no manual install needed [OK]

Common Mistakes:

• Thinking Hadoop must be installed manually on EMR nodes
• Forgetting to set IAM roles for S3 access
• Using local paths instead of S3 paths in job submission