Word count as MapReduce example in Hadoop - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main purpose of the Map function in a Word Count MapReduce job?

The Map function processes input data line by line, splits each line into words, and outputs each word paired with the number 1, indicating a single occurrence.

beginner

What does the Reduce function do in the Word Count MapReduce example?

The Reduce function takes all the values associated with the same word (key) and sums them up to get the total count of that word in the input data.

intermediate

Why is MapReduce useful for word counting in large datasets?

MapReduce allows processing large datasets by splitting the work across many machines, mapping words in parallel, and then reducing counts efficiently, making it scalable and fast.

intermediate

In the Word Count MapReduce example, what data type is typically used for the output key and value?

The output key is usually a word (Text type), and the output value is an integer count (IntWritable type in Hadoop).

advanced

What is the role of the Combiner in the Word Count MapReduce job?

The Combiner acts like a mini-Reducer that runs after the Map phase to locally sum word counts before sending data to the Reducer, reducing data transfer and improving efficiency.

What does the Mapper output in a Word Count MapReduce job?

A Pairs of (word, 1)

B Pairs of (line, word count)

C Pairs of (word, total count)

D Pairs of (file name, word)

The Mapper outputs each word paired with the number 1 to indicate one occurrence.

What is the main task of the Reducer in the Word Count example?

A Sum counts for each word

B Sort words alphabetically

C Split lines into words

D Filter out common words

The Reducer sums all counts for each word to get the total occurrences.

Which Hadoop data type is commonly used for the word key in Word Count?

A LongWritable

B IntWritable

C Text

D FloatWritable

Text is used for string data like words in Hadoop.

Why use a Combiner in Word Count MapReduce?

A To increase the number of mappers

B To reduce data sent to reducers

C To sort the output

D To split input files

The Combiner reduces the amount of data transferred by locally summing counts.

What is the input to the Mapper in Word Count?

A Individual words

B Sorted words

C Word counts

D Lines of text

The Mapper receives lines of text as input to process.

Explain the flow of data in a Word Count MapReduce job from input to output.

Think about how data moves and changes from start to finish.

Describe the purpose and benefit of using a Combiner in the Word Count example.

Consider how data transfer affects speed.