Word count as MapReduce example in Hadoop - Step-by-Step Execution

Concept Flow - Word count as MapReduce example

Input Text

↓

Map Function

↓

Emit (word, 1)

↓

Shuffle & Sort

↓

Reduce Function

↓

Sum counts per word

↓

Output (word, total count)

The input text is split and processed by the Map function to emit word counts. Then, the framework groups counts by word and the Reduce function sums them up to produce final counts.

Execution Sample

Hadoop

map(String key, String value):
  for word in value.split():
    emit(word, 1)

reduce(String word, Iterator counts):
  sum = 0
  for c in counts:
    sum += c
  emit(word, sum)

This code counts how many times each word appears in the input text using Map and Reduce functions.

Execution Table

Step	Input	Action	Output
1	"hello world hello"	Map splits text and emits (word,1)	(hello,1), (world,1), (hello,1)
2	(hello,1), (world,1), (hello,1)	Shuffle groups by word	(hello: [1,1]), (world: [1])
3	(hello: [1,1])	Reduce sums counts	(hello, 2)
4	(world: [1])	Reduce sums counts	(world, 1)
5	All words processed	Output final counts	(hello, 2), (world, 1)

💡 All words processed and counts summed, MapReduce job completes.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
map_output	empty	(hello,1),(world,1),(hello,1)	(hello,1),(world,1),(hello,1)	(hello,1),(world,1),(hello,1)	N/A
shuffle_output	empty	N/A	(hello:[1,1]),(world:[1])	(hello:[1,1]),(world:[1])	N/A
reduce_output	empty	N/A	N/A	(hello,2)	(hello,2),(world,1)

Key Moments - 3 Insights

Why does the Map function emit (word, 1) instead of just the word?

Because Map emits a count of 1 for each word occurrence, so Reduce can sum these counts to get total occurrences. See execution_table step 1 where each word is paired with 1.

What happens during the Shuffle & Sort phase?

Shuffle groups all (word, 1) pairs by word so Reduce gets all counts for a word together. This is shown in execution_table step 2 where words are grouped with their counts.

How does the Reduce function calculate the total count?

Reduce sums all counts for a word from the grouped list. For example, in step 3, it sums [1,1] for 'hello' to get 2.

Visual Quiz - 3 Questions

Look at the execution table, what is the output of the Map function at step 1?

A (hello,1), (world,1), (hello,1)

B (hello,2), (world,1)

C (hello:[1,1]), (world:[1])

D (hello,0), (world,0)

💡 Hint

Check the 'Output' column at step 1 in the execution_table.

At which step does the Reduce function output the total count for 'hello'?

A Step 1

B Step 2

C Step 3

D Step 5

💡 Hint

Look at the 'Action' and 'Output' columns in execution_table rows for Reduce function.

If the input text had one more 'world', how would the Reduce output for 'world' change?

A It would remain (world,1)

B It would become (world,2)

C It would become (world,3)

D It would be removed

💡 Hint

Refer to variable_tracker and how counts are summed in Reduce.

Concept Snapshot

MapReduce Word Count:
- Map splits text, emits (word,1) pairs
- Shuffle groups pairs by word
- Reduce sums counts per word
- Output is (word, total count)
- Used for counting words in large data sets

Full Transcript

This example shows how MapReduce counts words in text. The Map function reads input text and emits each word with a count of 1. Then, the framework groups all counts by word in the Shuffle phase. The Reduce function sums these counts to get total occurrences per word. Finally, the output lists each word with its total count. This process allows counting words efficiently in big data by splitting work across many machines.

Practice

1. What is the main purpose of the Map step in the Word Count MapReduce example?

easy

A. To split input text into pairs of (word, 1)

B. To sum the counts of each word

C. To sort the words alphabetically

D. To output the final word counts

Solution

1. Step 1: Understand the Map step role

   The Map step processes input text and breaks it into pairs where each word is paired with the number 1. The Reduce step sums these counts, so the Map step does not sum or sort but only creates (word, 1) pairs.

2. Final Answer:

   To split input text into pairs of (word, 1) -> Option A

3. Quick Check:

   Map step = (word, 1) pairs [OK]

Hint: Map creates (word, 1) pairs, Reduce sums counts [OK]

Common Mistakes:

• Confusing Map with Reduce step
• Thinking Map sorts words
• Assuming Map sums counts

2. Which of the following is the correct syntax for the Mapper output in a Hadoop Word Count example?

easy

A. context.write(word, 1);

B. context.write(1, word);

C. write.context(word, 1);

D. context.write(word);

Solution

1. Step 1: Recall Mapper output syntax

   In Hadoop MapReduce, the Mapper outputs key-value pairs using context.write(key, value);. For word count, the key is the word and the value is 1, so context.write(word, 1); is correct.

2. Final Answer:

   context.write(word, 1); -> Option A

3. Quick Check:

   Mapper output = context.write(word, 1) [OK]

Hint: Mapper outputs (word, 1) using context.write(word, 1) [OK]

Common Mistakes:

• Swapping key and value order
• Using incorrect method name
• Omitting the value in output

3. Given the following input text: "cat dog cat bird dog cat", what will be the output of the Reduce step in the Word Count MapReduce example?

medium

A. [('cat', 3), ('dog', 3), ('bird', 1)]

B. [('cat', 3), ('dog', 2), ('bird', 1)]

C. [('cat', 2), ('dog', 3), ('bird', 1)]

D. [('cat', 1), ('dog', 1), ('bird', 1)]

Solution

1. Step 1: Count each word occurrences

   Input words: cat (3 times), dog (2 times), bird (1 time). The Reduce step sums all counts for each word, resulting in ('cat', 3), ('dog', 2), ('bird', 1).

2. Final Answer:

   [('cat', 3), ('dog', 2), ('bird', 1)] -> Option B

3. Quick Check:

   Word counts = [('cat', 3), ('dog', 2), ('bird', 1)] [OK]

Hint: Count each word's frequency, sum in Reduce step [OK]

Common Mistakes:

• Mixing counts of dog and cat
• Counting only unique words once
• Adding counts incorrectly

4. Identify the error in this Mapper code snippet for Word Count:

public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
  String[] words = value.toString().split(" ");
  for (String word : words) {
    context.write(word, new IntWritable(1));
  }
}

medium

A. The split method should use "," instead of space

B. The map method should return a value

C. IntWritable should be replaced with Integer

D. The key type in context.write should be Text, not String

Solution

1. Step 1: Check context.write parameter types

   In Hadoop, context.write expects key and value types matching Mapper output types, usually Text and IntWritable. The variable word is a String, but context.write requires a Text object as key, so it should be new Text(word).

2. Final Answer:

   The key type in context.write should be Text, not String -> Option D

3. Quick Check:

   Use Text for keys in context.write [OK]

Hint: Use Hadoop types (Text, IntWritable) in context.write [OK]

Common Mistakes:

• Passing String instead of Text as key
• Using wrong split delimiter
• Expecting map method to return a value

5. You want to modify the Word Count MapReduce job to ignore case (treat 'Cat' and 'cat' as the same word). Which change should you apply in the Mapper?

hard

A. Sort words alphabetically in the Mapper

B. Convert words to uppercase in the Reducer

C. Convert each word to lowercase before writing: context.write(new Text(word.toLowerCase()), new IntWritable(1));

D. Filter out words shorter than 3 characters in the Mapper

Solution

1. Step 1: Understand case normalization

   To treat 'Cat' and 'cat' as the same, convert all words to the same case before counting. Changing words to lowercase in the Mapper ensures all keys are uniform before Reduce sums counts.

2. Final Answer:

   Convert each word to lowercase before writing: context.write(new Text(word.toLowerCase()), new IntWritable(1)); -> Option C

3. Quick Check:

   Normalize case in Mapper for consistent keys [OK]

Hint: Lowercase words in Mapper to unify counts [OK]

Common Mistakes:

• Changing case in Reducer instead of Mapper
• Sorting words instead of normalizing case
• Filtering words unrelated to case