Shuffle and sort phase in Hadoop - Mini Project: Build & Apply

Understanding the Shuffle and Sort Phase in Hadoop MapReduce

📖 Scenario: Imagine you are working with a large dataset of sales records from different stores. You want to count how many sales each product has across all stores. Hadoop MapReduce helps by splitting this task into smaller parts. The Shuffle and Sort phase is a key step that organizes the data between the map and reduce steps.

🎯 Goal: You will simulate the Shuffle and Sort phase by grouping and sorting intermediate key-value pairs produced by the map step. This will prepare the data for the reduce step to count sales per product.

📋 What You'll Learn

Create a list of intermediate key-value pairs from the map output

Create a configuration variable to specify sorting order

Group and sort the intermediate data by product name

Print the grouped and sorted data to show the shuffle and sort result

💡 Why This Matters

🌍 Real World

The shuffle and sort phase is essential in big data processing to organize data between map and reduce steps for aggregation.

💼 Career

Understanding shuffle and sort helps data engineers optimize Hadoop jobs and troubleshoot performance issues in distributed data processing.

1

Create the map output data

Create a list called map_output with these exact tuples representing product sales: ("apple", 1), ("banana", 1), ("apple", 1), ("orange", 1), ("banana", 1).

Hadoop

# Create the list map_output with the given tuples
# Your code here

Need a hint?

Use a Python list with tuples exactly as shown.

2

Set sorting order configuration

Create a variable called sort_ascending and set it to True to specify that sorting should be in ascending order.

Hadoop

map_output = [("apple", 1), ("banana", 1), ("apple", 1), ("orange", 1), ("banana", 1)]
# Create sort_ascending variable and set to True
# Your code here

Need a hint?

This variable controls if sorting is ascending or descending.

3

Group and sort the map output

Create a dictionary called shuffled_sorted that groups values by product name from map_output. Sort the keys in ascending order if sort_ascending is True. Use a for loop with variables product and count to iterate over map_output.

Hadoop

map_output = [("apple", 1), ("banana", 1), ("apple", 1), ("orange", 1), ("banana", 1)]
sort_ascending = True
# Group and sort map_output into shuffled_sorted
# Your code here

Need a hint?

Use a dictionary to group counts by product. Then sort the dictionary keys.

4

Print the shuffle and sort result

Print the variable shuffled_sorted to display the grouped and sorted intermediate data.

Hadoop

map_output = [("apple", 1), ("banana", 1), ("apple", 1), ("orange", 1), ("banana", 1)]
sort_ascending = True

shuffled_sorted = {}
for product, count in map_output:
    if product not in shuffled_sorted:
        shuffled_sorted[product] = []
    shuffled_sorted[product].append(count)

shuffled_sorted = dict(sorted(shuffled_sorted.items(), reverse=not sort_ascending))
# Print the shuffled_sorted dictionary
# Your code here

Need a hint?

The output shows each product with a list of counts grouped and sorted by product name.