Reduce phase explained in Hadoop - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to define the reduce function that sums values for each key.

Hadoop

def reduce(key, values):
    result = 0
    for value in values:
        result += [1]
    return (key, result)

Drag options to blanks, or click blank then click option'

Akey

Bvalues

Cvalue

Dresult

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'key' instead of 'value' inside the loop.

Adding 'values' instead of individual 'value'.

The reduce function sums each value in the list of values for a given key, so we add 'value' to the result.

2fill in blank

medium

Complete the code to emit the final key-value pair from the reduce function.

Hadoop

def reduce(key, values):
    total = sum(values)
    [1](key, total)

Drag options to blanks, or click blank then click option'

Aprint

Boutput

Creturn

Demit

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'print' instead of 'emit'.

Using 'return' which does not output to Hadoop.

In Hadoop's reduce phase, the 'emit' function outputs the final key and its aggregated value.

3fill in blank

hard

Fix the error in the reduce function to correctly sum values.

Hadoop

def reduce(key, values):
    total = 0
    for val in values:
        total = total [1] val
    emit(key, total)

Drag options to blanks, or click blank then click option'

A+

B/

C*

D-

Attempts:

3 left

💡 Hint

Common Mistakes

Using '-' which subtracts values.

Using '*' or '/' which multiply or divide.

To sum values, use the '+' operator to add each val to total.

4fill in blank

hard

Fill both blanks to create a dictionary of word counts using reduce logic.

Hadoop

word_counts = {word: [1] for word in words if len(word) [2] 3}

Drag options to blanks, or click blank then click option'

Awords.count(word)

Blen(word)

C>

D<=

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'len(word)' as count instead of 'words.count(word)'.

Using '<=' instead of '>' for filtering.

We count occurrences of each word with 'words.count(word)' and filter words longer than 3 with '>'.

5fill in blank

hard

Fill all three blanks to create a reduce function that filters and sums values.

Hadoop

def reduce(key, values):
    filtered = [v for v in values if v [1] 10]
    total = sum(filtered)
    if total [2] 0:
        [3](key, total)

Drag options to blanks, or click blank then click option'

A>

B>=

Cemit

Dprint

Attempts:

3 left

💡 Hint

Common Mistakes

Using '<' instead of '>' for filtering.

Using 'print' instead of 'emit' to output.

Filter values greater than 10 with '>', check if total is >= 0, then emit the result.

Practice

1.

What is the main purpose of the Reduce phase in Hadoop MapReduce?

easy

A. To combine and summarize data from the Map outputs by key

B. To split the input data into smaller chunks

C. To read data from the file system

D. To sort the input data before mapping

Solution

1. Step 1: Understand the role of Reduce phase

   The Reduce phase takes the output from the Map phase and groups data by keys.

2. Step 2: Identify the main function of Reduce

   It combines or aggregates the values for each key to produce summarized results.

3. Final Answer:

   To combine and summarize data from the Map outputs by key -> Option A

4. Quick Check:

   Reduce phase = combine by key [OK]

Hint: Reduce phase aggregates data by keys from Map outputs [OK]

Common Mistakes:

• Confusing Reduce with Map phase
• Thinking Reduce splits data
• Assuming Reduce reads input files

2.

Which of the following is the correct signature of a Reduce function in Hadoop MapReduce?

public void reduce(KEY key, Iterable<VALUE> values, Context context) throws IOException, InterruptedException

easy

A. public void reduce(KEY key, VALUE value, Context context) throws IOException, InterruptedException

B. public void map(KEY key, VALUE value, Context context) throws IOException, InterruptedException

C. public void reduce(KEY key, Iterable<VALUE> values, Context context) throws IOException, InterruptedException

D. public void map(KEY key, Iterable<VALUE> values, Context context) throws IOException, InterruptedException

Solution

1. Step 1: Recall Reduce function signature

   The Reduce function receives a key and an iterable list of values for that key.

2. Step 2: Match the correct method signature

   public void reduce(KEY key, Iterable<VALUE> values, Context context) throws IOException, InterruptedException correctly shows the reduce method with key, Iterable values, and context parameters.

3. Final Answer:

   public void reduce(KEY key, Iterable<VALUE> values, Context context) throws IOException, InterruptedException -> Option C

4. Quick Check:

   Reduce signature = key + Iterable values + context [OK]

Hint: Reduce gets key and list of values, not single value [OK]

Common Mistakes:

• Confusing map and reduce method signatures
• Using single VALUE instead of Iterable<VALUE> in reduce
• Mixing map parameters with reduce function

3.

Given the following Reduce function snippet, what will be the output for key = "apple" and values = [2, 3, 5]?

public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
    int sum = 0;
    for (IntWritable val : values) {
        sum += val.get();
    }
    context.write(key, new IntWritable(sum));
}

medium

A. (apple, 2)

B. (apple, 10)

C. (apple, 0)

D. (apple, 5)

Solution

1. Step 1: Sum the values list

   Values are 2, 3, and 5. Summing gives 2 + 3 + 5 = 10.

2. Step 2: Output key with summed value

   The context writes the key "apple" with the sum 10 as IntWritable.

3. Final Answer:

   (apple, 10) -> Option B

4. Quick Check:

   Sum of values = 10 [OK]

Hint: Add all values for key to get output [OK]

Common Mistakes:

• Returning only first value instead of sum
• Returning count instead of sum
• Ignoring values and outputting zero

4.

Identify the error in this Reduce function code snippet:

public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
    int sum = 0;
    for (IntWritable val : values) {
        sum = val.get();
    }
    context.write(key, new IntWritable(sum));
}

medium

A. The Iterable<IntWritable> should be a List<IntWritable>

B. The key parameter type is incorrect

C. The context.write method is missing

D. The sum variable is overwritten instead of accumulated

Solution

1. Step 1: Analyze the for loop logic

   The code assigns sum = val.get() inside the loop, overwriting sum each time.

2. Step 2: Identify correct accumulation

   Sum should accumulate values using sum += val.get() to add all values.

3. Final Answer:

   The sum variable is overwritten instead of accumulated -> Option D

4. Quick Check:

   Use += to accumulate sum, not = [OK]

Hint: Use += to add values, not = which overwrites [OK]

Common Mistakes:

• Using = instead of += in accumulation
• Confusing key type with error
• Thinking context.write is missing

5.

You want to count the total occurrences of each word in a large text dataset using Hadoop MapReduce. Which of the following best describes what the Reduce phase should do?

hard

A. Receive each word as key and list of counts as values, then sum counts to get total occurrences

B. Split the input text into words and assign counts

C. Sort all words alphabetically before counting

D. Write the raw text data to output without changes

Solution

1. Step 1: Understand MapReduce word count flow

   Map phase outputs (word, 1) pairs for each word occurrence.

2. Step 2: Role of Reduce phase in counting

   Reduce receives each word as key and list of counts (1s), sums them to get total count.

3. Final Answer:

   Receive each word as key and list of counts as values, then sum counts to get total occurrences -> Option A

4. Quick Check:

   Reduce sums counts per word [OK]

Hint: Reduce sums counts for each word key [OK]

Common Mistakes:

• Confusing Map and Reduce roles
• Thinking Reduce sorts data
• Assuming Reduce outputs raw data