Word count as MapReduce example in Hadoop - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main purpose of the Map function in a Word Count MapReduce job?

The Map function processes input data line by line, splits each line into words, and outputs each word paired with the number 1, indicating a single occurrence.

beginner

What does the Reduce function do in the Word Count MapReduce example?

The Reduce function takes all the values associated with the same word (key) and sums them up to get the total count of that word in the input data.

intermediate

Why is MapReduce useful for word counting in large datasets?

MapReduce allows processing large datasets by splitting the work across many machines, mapping words in parallel, and then reducing counts efficiently, making it scalable and fast.

intermediate

In the Word Count MapReduce example, what data type is typically used for the output key and value?

The output key is usually a word (Text type), and the output value is an integer count (IntWritable type in Hadoop).

advanced

What is the role of the Combiner in the Word Count MapReduce job?

The Combiner acts like a mini-Reducer that runs after the Map phase to locally sum word counts before sending data to the Reducer, reducing data transfer and improving efficiency.

What does the Mapper output in a Word Count MapReduce job?

A Pairs of (word, 1)

B Pairs of (line, word count)

C Pairs of (word, total count)

D Pairs of (file name, word)

The Mapper outputs each word paired with the number 1 to indicate one occurrence.

What is the main task of the Reducer in the Word Count example?

A Sum counts for each word

B Sort words alphabetically

C Split lines into words

D Filter out common words

The Reducer sums all counts for each word to get the total occurrences.

Which Hadoop data type is commonly used for the word key in Word Count?

A LongWritable

B IntWritable

C Text

D FloatWritable

Text is used for string data like words in Hadoop.

Why use a Combiner in Word Count MapReduce?

A To increase the number of mappers

B To reduce data sent to reducers

C To sort the output

D To split input files

The Combiner reduces the amount of data transferred by locally summing counts.

What is the input to the Mapper in Word Count?

A Individual words

B Sorted words

C Word counts

D Lines of text

The Mapper receives lines of text as input to process.

Explain the flow of data in a Word Count MapReduce job from input to output.

Think about how data moves and changes from start to finish.

Describe the purpose and benefit of using a Combiner in the Word Count example.

Consider how data transfer affects speed.

Practice

1. What is the main purpose of the Map step in the Word Count MapReduce example?

easy

A. To split input text into pairs of (word, 1)

B. To sum the counts of each word

C. To sort the words alphabetically

D. To output the final word counts

Solution

1. Step 1: Understand the Map step role

   The Map step processes input text and breaks it into pairs where each word is paired with the number 1. The Reduce step sums these counts, so the Map step does not sum or sort but only creates (word, 1) pairs.

2. Final Answer:

   To split input text into pairs of (word, 1) -> Option A

3. Quick Check:

   Map step = (word, 1) pairs [OK]

Hint: Map creates (word, 1) pairs, Reduce sums counts [OK]

Common Mistakes:

• Confusing Map with Reduce step
• Thinking Map sorts words
• Assuming Map sums counts

2. Which of the following is the correct syntax for the Mapper output in a Hadoop Word Count example?

easy

A. context.write(word, 1);

B. context.write(1, word);

C. write.context(word, 1);

D. context.write(word);

Solution

1. Step 1: Recall Mapper output syntax

   In Hadoop MapReduce, the Mapper outputs key-value pairs using context.write(key, value);. For word count, the key is the word and the value is 1, so context.write(word, 1); is correct.

2. Final Answer:

   context.write(word, 1); -> Option A

3. Quick Check:

   Mapper output = context.write(word, 1) [OK]

Hint: Mapper outputs (word, 1) using context.write(word, 1) [OK]

Common Mistakes:

• Swapping key and value order
• Using incorrect method name
• Omitting the value in output

3. Given the following input text: "cat dog cat bird dog cat", what will be the output of the Reduce step in the Word Count MapReduce example?

medium

A. [('cat', 3), ('dog', 3), ('bird', 1)]

B. [('cat', 3), ('dog', 2), ('bird', 1)]

C. [('cat', 2), ('dog', 3), ('bird', 1)]

D. [('cat', 1), ('dog', 1), ('bird', 1)]

Solution

1. Step 1: Count each word occurrences

   Input words: cat (3 times), dog (2 times), bird (1 time). The Reduce step sums all counts for each word, resulting in ('cat', 3), ('dog', 2), ('bird', 1).

2. Final Answer:

   [('cat', 3), ('dog', 2), ('bird', 1)] -> Option B

3. Quick Check:

   Word counts = [('cat', 3), ('dog', 2), ('bird', 1)] [OK]

Hint: Count each word's frequency, sum in Reduce step [OK]

Common Mistakes:

• Mixing counts of dog and cat
• Counting only unique words once
• Adding counts incorrectly

4. Identify the error in this Mapper code snippet for Word Count:

public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
  String[] words = value.toString().split(" ");
  for (String word : words) {
    context.write(word, new IntWritable(1));
  }
}

medium

A. The split method should use "," instead of space

B. The map method should return a value

C. IntWritable should be replaced with Integer

D. The key type in context.write should be Text, not String

Solution

1. Step 1: Check context.write parameter types

   In Hadoop, context.write expects key and value types matching Mapper output types, usually Text and IntWritable. The variable word is a String, but context.write requires a Text object as key, so it should be new Text(word).

2. Final Answer:

   The key type in context.write should be Text, not String -> Option D

3. Quick Check:

   Use Text for keys in context.write [OK]

Hint: Use Hadoop types (Text, IntWritable) in context.write [OK]

Common Mistakes:

• Passing String instead of Text as key
• Using wrong split delimiter
• Expecting map method to return a value

5. You want to modify the Word Count MapReduce job to ignore case (treat 'Cat' and 'cat' as the same word). Which change should you apply in the Mapper?

hard

A. Sort words alphabetically in the Mapper

B. Convert words to uppercase in the Reducer

C. Convert each word to lowercase before writing: context.write(new Text(word.toLowerCase()), new IntWritable(1));

D. Filter out words shorter than 3 characters in the Mapper

Solution

1. Step 1: Understand case normalization

   To treat 'Cat' and 'cat' as the same, convert all words to the same case before counting. Changing words to lowercase in the Mapper ensures all keys are uniform before Reduce sums counts.

2. Final Answer:

   Convert each word to lowercase before writing: context.write(new Text(word.toLowerCase()), new IntWritable(1)); -> Option C

3. Quick Check:

   Normalize case in Mapper for consistent keys [OK]

Hint: Lowercase words in Mapper to unify counts [OK]

Common Mistakes:

• Changing case in Reducer instead of Mapper
• Sorting words instead of normalizing case
• Filtering words unrelated to case