Reduce phase explained in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main role of the Reduce phase in Hadoop MapReduce?

In Hadoop MapReduce, after the Map phase processes input data, the Reduce phase takes over. What is the main role of the Reduce phase?

A It sorts the input data before mapping.

B It splits the input data into smaller chunks for processing.

C It aggregates and summarizes the intermediate data produced by the Map phase.

D It reads raw input data from the source files.

💡 Hint

Think about what happens after the Map phase outputs key-value pairs.

The Reduce phase collects all intermediate values associated with the same key and combines them to produce a summarized output. This aggregation is the main purpose of the Reduce phase.

❓ Predict Output

intermediate

Output of Reduce function aggregation

Consider the following simplified Reduce function in Hadoop MapReduce that sums values for each key:

def reduce(key, values):
    total = 0
    for v in values:
        total += v
    print(f"{key}: {total}")

What will be the output if the input to reduce is key = 'apple' and values = [2, 3, 5]?

Hadoop

def reduce(key, values):
    total = 0
    for v in values:
        total += v
    print(f"{key}: {total}")

reduce('apple', [2, 3, 5])

A apple: 10

B apple: 235

C apple: 0

D apple: 5

💡 Hint

Sum all numbers in the list.

The Reduce function sums all values in the list [2, 3, 5], which equals 10, and prints it with the key.

❓ data_output

advanced

Resulting key-value pairs after Reduce phase

Given the following intermediate key-value pairs from the Map phase:

{'cat': [1, 1, 1], 'dog': [1, 1], 'bird': [1]}

If the Reduce phase sums the values for each key, what is the resulting output?

A {'cat': 1, 'dog': 1, 'bird': 1}

B {'cat': 3, 'dog': 2, 'bird': 1}

C {'cat': [3], 'dog': [2], 'bird': [1]}

D {'cat': 6, 'dog': 4, 'bird': 2}

💡 Hint

Sum the list of values for each key.

Each key's values are summed: cat (1+1+1=3), dog (1+1=2), bird (1).

🔧 Debug

advanced

Identify the error in this Reduce function

Look at this Reduce function code snippet:

def reduce(key, values):
    total = 0
    for v in values
        total += v
    print(f"{key}: {total}")

What error will this code produce when run?

A NameError because total is not defined

B IndentationError due to wrong indentation

C TypeError because values is not iterable

D SyntaxError due to missing colon after for loop

💡 Hint

Check the for loop syntax carefully.

The for loop is missing a colon at the end of the line, causing a SyntaxError.

🚀 Application

expert

Choosing the correct Reduce phase output for word count

In a word count MapReduce job, the Map phase outputs key-value pairs where the key is a word and the value is 1 for each occurrence. The Reduce phase sums these counts. Given the following Map output for the word 'data':

[('data', 1), ('data', 1), ('data', 1), ('data', 1)]

Which of the following is the correct Reduce phase output for the key 'data'?

A ('data', 4)

B ('data', [1, 1, 1, 1])

C ('data', 1)

D ('data', 0)

💡 Hint

Sum all the counts for the word.

The Reduce phase sums all the 1s for 'data', resulting in 4.