Map phase explained in Hadoop - Mini Project: Build & Apply

Map Phase Explained in Hadoop MapReduce

📖 Scenario: Imagine you work at a company that wants to count how many times each word appears in a large collection of documents. Hadoop MapReduce helps process this big data by breaking the job into smaller parts. The first part is called the Map phase, where each document is split into words and each word is paired with the number 1.

🎯 Goal: You will create a simple Python program that simulates the Map phase of Hadoop MapReduce. You will start by setting up the data, then configure a helper variable, apply the map logic to split text into words and pair each with 1, and finally print the mapped output.

📋 What You'll Learn

Create a dictionary called documents with three entries: 'doc1', 'doc2', and 'doc3', each containing a short sentence.

Create a variable called separator and set it to a space character.

Use a for loop with variables doc_id and text to iterate over documents.items().

Inside the loop, split the text using separator and create a list of tuples called mapped_words where each tuple is (word, 1).

Print the doc_id and the mapped_words list for each document.

💡 Why This Matters

🌍 Real World

The Map phase is the first step in processing large text data in Hadoop. It breaks down big files into smaller pieces and prepares data for counting or analysis.

💼 Career

Understanding the Map phase helps you work with big data tools like Hadoop and Spark, which are widely used in data engineering and data science jobs.

1

Set up the documents dictionary

Create a dictionary called documents with these exact entries: 'doc1': 'hello world', 'doc2': 'hello hadoop', and 'doc3': 'map reduce example'.

Hadoop

# Create the documents dictionary with exact entries
# Your code here

Hint

Use curly braces {} to create a dictionary. Each key is a document ID string, and each value is a sentence string.

2

Create the separator variable

Create a variable called separator and set it to a single space character ' '.

Hadoop

documents = {
    'doc1': 'hello world',
    'doc2': 'hello hadoop',
    'doc3': 'map reduce example'
}
# Create the separator variable as a space character
# Your code here

Hint

The separator is the character used to split sentences into words. Use a space ' '.

3

Apply the map logic to split and pair words

Use a for loop with variables doc_id and text to iterate over documents.items(). Inside the loop, split the text using separator and create a list called mapped_words where each element is a tuple (word, 1).

Hadoop

documents = {
    'doc1': 'hello world',
    'doc2': 'hello hadoop',
    'doc3': 'map reduce example'
}
separator = ' '

# Use a for loop to iterate over documents and create mapped_words list
# Your code here

Hint

Use text.split(separator) to get words. Then use a list comprehension to pair each word with 1.

4

Print the mapped output for each document

Inside the for loop, print the doc_id and the mapped_words list for each document.

Hadoop

documents = {
    'doc1': 'hello world',
    'doc2': 'hello hadoop',
    'doc3': 'map reduce example'
}
separator = ' '

for doc_id, text in documents.items():
    words = text.split(separator)
    mapped_words = [(word, 1) for word in words]
    # Print the doc_id and mapped_words
    # Your code here

Hint

Use print(doc_id, mapped_words) inside the loop to show the output.

Practice

1. What is the main role of the Map phase in Hadoop's data processing?

easy

A. To store data permanently in HDFS

B. To combine all results into a final output

C. To process input data and produce intermediate key-value pairs

D. To clean and validate input data before processing

Solution

1. Step 1: Understand the Map phase purpose

   The Map phase reads input data and processes it piece by piece.

2. Step 2: Identify output of Map phase

   It produces intermediate key-value pairs for the next phase (Reduce) to process.

3. Final Answer:

   To process input data and produce intermediate key-value pairs -> Option C

4. Quick Check:

   Map phase = intermediate key-value pairs [OK]

Hint: Map phase outputs key-value pairs from input data [OK]

Common Mistakes:

• Confusing Map phase with Reduce phase
• Thinking Map phase stores final results
• Assuming Map phase cleans data

2. Which of the following is the correct syntax to define a Map function in Hadoop's MapReduce framework (Java)?

easy

A. public void map(Context context, LongWritable key, Text value)

B. public void reduce(Text key, Iterable values, Context context)

C. public void map(String key, String value)

D. public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException

Solution

1. Step 1: Recall Map function signature in Hadoop Java

   The Map method signature includes input key, input value, and context for output.

2. Step 2: Match correct parameters and exceptions

   public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException matches the correct signature with LongWritable key, Text value, Context context, and throws exceptions.

3. Final Answer:

   public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException -> Option D

4. Quick Check:

   Map method signature = public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException [OK]

Hint: Map method has key, value, context parameters [OK]

Common Mistakes:

• Mixing Map and Reduce method signatures
• Wrong parameter order or types
• Omitting exception declarations

3. Given the following Map function in Hadoop, what will be the intermediate output for input line "apple orange apple"?

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
  String[] words = value.toString().split(" ");
  for (String word : words) {
    context.write(new Text(word), new IntWritable(1));
  }
}

medium

A. [('apple', 1), ('orange', 1), ('apple', 1)]

B. [('apple orange apple', 1)]

C. [('apple', 3), ('orange', 1)]

D. [('apple', 1), ('orange', 1)]

Solution

1. Step 1: Split input line into words

   The input "apple orange apple" splits into ["apple", "orange", "apple"].

2. Step 2: Emit each word with count 1

   The Map function outputs each word paired with 1: ('apple',1), ('orange',1), ('apple',1).

3. Final Answer:

   [('apple', 1), ('orange', 1), ('apple', 1)] -> Option A

4. Quick Check:

   Map outputs word counts as pairs [OK]

Hint: Map outputs each word with count 1 separately [OK]

Common Mistakes:

• Summing counts in Map phase (done in Reduce)
• Outputting entire line as one key
• Missing duplicate words in output

4. Identify the error in this Map function snippet:

public void map(LongWritable key, Text value, Context context) {
  String[] words = value.toString().split(" ");
  for (String word : words) {
    context.write(new Text(word), new IntWritable(1));
  }
}

medium

A. Missing throws IOException, InterruptedException in method signature

B. Using wrong data type for key parameter

C. Incorrect split delimiter in split method

D. Context object cannot be used inside map method

Solution

1. Step 1: Check method signature requirements

   Hadoop Map method must declare throwing IOException and InterruptedException.

2. Step 2: Verify method signature correctness

   The given method lacks "throws IOException, InterruptedException", causing a compile error.

3. Final Answer:

   Missing throws IOException, InterruptedException in method signature -> Option A

4. Quick Check:

   Map method must declare exceptions [OK]

Hint: Map method must declare IOException, InterruptedException [OK]

Common Mistakes:

• Ignoring required exceptions in method signature
• Assuming context cannot be used
• Thinking split delimiter is wrong

5. You want to count the frequency of words but only output words longer than 3 characters during the Map phase. Which modification to the Map function below achieves this?

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
  String[] words = value.toString().split(" ");
  for (String word : words) {
    context.write(new Text(word), new IntWritable(1));
  }
}

hard

A. Use context.write only once after the loop with all words

B. Add an if condition: if (word.length() > 3) before context.write

C. Change split to split(".{3}") to split only words longer than 3

D. Filter words in the Reduce phase instead of Map phase

Solution

1. Step 1: Identify filtering requirement in Map phase

   We want to output only words longer than 3 characters during Map.

2. Step 2: Add condition to output only qualifying words

   Adding if (word.length() > 3) before context.write filters words correctly.

3. Final Answer:

   Add an if condition: if (word.length() > 3) before context.write -> Option B

4. Quick Check:

   Filter words by length in Map phase [OK]

Hint: Use if condition inside Map loop to filter words [OK]

Common Mistakes:

• Trying to filter by regex in split (wrong approach)
• Filtering only in Reduce phase (too late)
• Writing all words at once outside loop