Map phase explained in Hadoop - Step-by-Step Execution

Concept Flow - Map phase explained

Input Data Split

↓

Map Function Applied

↓

Process Each Record

↓

Emit Key-Value Pairs

↓

Shuffle and Sort (Prepare for Reduce)

The Map phase takes input data splits, applies the map function to each record, and emits key-value pairs for the next phase.

Execution Sample

Hadoop

def map(record):
    words = record.split()
    for word in words:
        emit(word, 1)

This map function splits a text record into words and emits each word with count 1.

Execution Table

Step	Input Record	Action	Output Key-Value Pairs
1	"hello world"	Split into words ['hello', 'world']	None
2	First word 'hello'	Emit ('hello', 1)	('hello', 1)
3	Second word 'world'	Emit ('world', 1)	('world', 1)
4	Next record 'hello hadoop'	Split into words ['hello', 'hadoop']	None
5	First word 'hello'	Emit ('hello', 1)	('hello', 1)
6	Second word 'hadoop'	Emit ('hadoop', 1)	('hadoop', 1)
7	No more records	Map phase ends	End of map output

💡 All input records processed, map phase completes emitting key-value pairs.

Variable Tracker

Variable	Start	After 1	After 2	After 3	After 4	After 5	After 6	Final
record	None	"hello world"	"hello world"	"hello world"	"hello hadoop"	"hello hadoop"	"hello hadoop"	None
words	None	['hello', 'world']	['hello', 'world']	['hello', 'world']	['hello', 'hadoop']	['hello', 'hadoop']	['hello', 'hadoop']	None
word	None	None	'hello'	'world'	None	'hello'	'hadoop'	None
emitted_pairs	[]	[]	[('hello', 1)]	[('hello', 1), ('world', 1)]	[('hello', 1), ('world', 1)]	[('hello', 1), ('world', 1), ('hello', 1)]	[('hello', 1), ('world', 1), ('hello', 1), ('hadoop', 1)]	[('hello', 1), ('world', 1), ('hello', 1), ('hadoop', 1)]

Key Moments - 2 Insights

Why does the map function emit multiple key-value pairs for one input record?

Because the input record is split into multiple words, and the map function emits one key-value pair per word as shown in execution_table rows 2, 3, 5, and 6.

What happens after the map function emits key-value pairs?

After emitting, the pairs are shuffled and sorted to prepare for the reduce phase, as indicated in the concept_flow after the 'Emit Key-Value Pairs' step.

Visual Quiz - 3 Questions

Look at the execution table, what key-value pair is emitted at step 3?

A ('hello', 1)

B ('hadoop', 1)

C ('world', 1)

D No pair emitted

💡 Hint

Check the 'Output Key-Value Pairs' column at step 3 in the execution_table.

At which step does the map function process the record 'hello hadoop'?

A Step 4

B Step 1

C Step 6

D Step 7

💡 Hint

Look at the 'Input Record' column in the execution_table to find when 'hello hadoop' is processed.

If the input record was empty, what would the map function emit?

A ('null', 0)

B No key-value pairs

C ('empty', 1)

D One pair with empty string key

💡 Hint

Refer to how the map function emits pairs only for words found in the record, as shown in the execution_table.

Concept Snapshot

Map phase takes input data split
Processes each record with map function
Splits record into parts (e.g., words)
Emits key-value pairs for each part
Prepares data for shuffle and reduce phases

Full Transcript

The Map phase in Hadoop starts by taking a split of input data. Each record in this split is processed by the map function. For example, a text record is split into words. For each word, the map function emits a key-value pair, typically the word and the count 1. These emitted pairs are collected and later shuffled and sorted to prepare for the reduce phase. The execution table shows step-by-step how records are split and pairs emitted. Variables like 'record', 'words', and 'emitted_pairs' change as the map function runs. Beginners often wonder why multiple pairs come from one record; this is because each word is processed separately. After map, the data is ready for the next phase in Hadoop's data processing pipeline.

Practice

1. What is the main role of the Map phase in Hadoop's data processing?

easy

A. To store data permanently in HDFS

B. To combine all results into a final output

C. To process input data and produce intermediate key-value pairs

D. To clean and validate input data before processing

Solution

1. Step 1: Understand the Map phase purpose

   The Map phase reads input data and processes it piece by piece.

2. Step 2: Identify output of Map phase

   It produces intermediate key-value pairs for the next phase (Reduce) to process.

3. Final Answer:

   To process input data and produce intermediate key-value pairs -> Option C

4. Quick Check:

   Map phase = intermediate key-value pairs [OK]

Hint: Map phase outputs key-value pairs from input data [OK]

Common Mistakes:

• Confusing Map phase with Reduce phase
• Thinking Map phase stores final results
• Assuming Map phase cleans data

2. Which of the following is the correct syntax to define a Map function in Hadoop's MapReduce framework (Java)?

easy

A. public void map(Context context, LongWritable key, Text value)

B. public void reduce(Text key, Iterable values, Context context)

C. public void map(String key, String value)

D. public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException

Solution

1. Step 1: Recall Map function signature in Hadoop Java

   The Map method signature includes input key, input value, and context for output.

2. Step 2: Match correct parameters and exceptions

   public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException matches the correct signature with LongWritable key, Text value, Context context, and throws exceptions.

3. Final Answer:

   public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException -> Option D

4. Quick Check:

   Map method signature = public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException [OK]

Hint: Map method has key, value, context parameters [OK]

Common Mistakes:

• Mixing Map and Reduce method signatures
• Wrong parameter order or types
• Omitting exception declarations

3. Given the following Map function in Hadoop, what will be the intermediate output for input line "apple orange apple"?

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
  String[] words = value.toString().split(" ");
  for (String word : words) {
    context.write(new Text(word), new IntWritable(1));
  }
}

medium

A. [('apple', 1), ('orange', 1), ('apple', 1)]

B. [('apple orange apple', 1)]

C. [('apple', 3), ('orange', 1)]

D. [('apple', 1), ('orange', 1)]

Solution

1. Step 1: Split input line into words

   The input "apple orange apple" splits into ["apple", "orange", "apple"].

2. Step 2: Emit each word with count 1

   The Map function outputs each word paired with 1: ('apple',1), ('orange',1), ('apple',1).

3. Final Answer:

   [('apple', 1), ('orange', 1), ('apple', 1)] -> Option A

4. Quick Check:

   Map outputs word counts as pairs [OK]

Hint: Map outputs each word with count 1 separately [OK]

Common Mistakes:

• Summing counts in Map phase (done in Reduce)
• Outputting entire line as one key
• Missing duplicate words in output

4. Identify the error in this Map function snippet:

public void map(LongWritable key, Text value, Context context) {
  String[] words = value.toString().split(" ");
  for (String word : words) {
    context.write(new Text(word), new IntWritable(1));
  }
}

medium

A. Missing throws IOException, InterruptedException in method signature

B. Using wrong data type for key parameter

C. Incorrect split delimiter in split method

D. Context object cannot be used inside map method

Solution

1. Step 1: Check method signature requirements

   Hadoop Map method must declare throwing IOException and InterruptedException.

2. Step 2: Verify method signature correctness

   The given method lacks "throws IOException, InterruptedException", causing a compile error.

3. Final Answer:

   Missing throws IOException, InterruptedException in method signature -> Option A

4. Quick Check:

   Map method must declare exceptions [OK]

Hint: Map method must declare IOException, InterruptedException [OK]

Common Mistakes:

• Ignoring required exceptions in method signature
• Assuming context cannot be used
• Thinking split delimiter is wrong

5. You want to count the frequency of words but only output words longer than 3 characters during the Map phase. Which modification to the Map function below achieves this?

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
  String[] words = value.toString().split(" ");
  for (String word : words) {
    context.write(new Text(word), new IntWritable(1));
  }
}

hard

A. Use context.write only once after the loop with all words

B. Add an if condition: if (word.length() > 3) before context.write

C. Change split to split(".{3}") to split only words longer than 3

D. Filter words in the Reduce phase instead of Map phase

Solution

1. Step 1: Identify filtering requirement in Map phase

   We want to output only words longer than 3 characters during Map.

2. Step 2: Add condition to output only qualifying words

   Adding if (word.length() > 3) before context.write filters words correctly.

3. Final Answer:

   Add an if condition: if (word.length() > 3) before context.write -> Option B

4. Quick Check:

   Filter words by length in Map phase [OK]

Hint: Use if condition inside Map loop to filter words [OK]

Common Mistakes:

• Trying to filter by regex in split (wrong approach)
• Filtering only in Reduce phase (too late)
• Writing all words at once outside loop