MapReduce job execution flow in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the first step in a MapReduce job execution?

In the MapReduce job execution flow, which step happens first?

A Splitting the input data into chunks

B Shuffling and sorting the map output

C Reducing the intermediate data

D Writing the final output to HDFS

💡 Hint

Think about how the data is prepared before processing.

The first step is splitting the input data into chunks so that each map task can process a part of the data in parallel.

🧠 Conceptual

intermediate

What happens during the shuffle phase in MapReduce?

During the MapReduce job execution, what is the main purpose of the shuffle phase?

A Executing the map function on input data

B Splitting the input data into smaller chunks

C Writing the final output to the distributed file system

D Sorting and transferring map outputs to reducers

💡 Hint

Think about how data moves from mappers to reducers.

The shuffle phase sorts and transfers the output of map tasks to the appropriate reduce tasks based on keys.

❓ data_output

advanced

Identify the output of the map phase given this input

Given the input data ["apple", "banana", "apple"] and a map function that outputs (word, 1) for each word, what is the map phase output?

Hadoop

input_data = ["apple", "banana", "apple"]
map_output = [(word, 1) for word in input_data]
print(map_output)

A [('banana', 1), ('apple', 1), ('apple', 1)]

B [('apple', 1), ('banana', 1), ('apple', 1)]

C [('apple', 1), ('banana', 1)]

D [('apple', 2), ('banana', 1)]

💡 Hint

The map function emits one pair per input word.

The map phase outputs a list of key-value pairs, one for each input element, without aggregation.

❓ data_output

advanced

What is the reducer output for this intermediate data?

Given the intermediate data {'apple': [1, 1], 'banana': [1]}, what is the output of the reducer that sums the values?

Hadoop

intermediate_data = {'apple': [1, 1], 'banana': [1]}
reducer_output = {k: sum(v) for k, v in intermediate_data.items()}
print(reducer_output)

A {'apple': 2, 'banana': 1}

B {'apple': 1, 'banana': 1}

C {'apple': [2], 'banana': [1]}

D {'apple': 3, 'banana': 1}

💡 Hint

The reducer sums all values for each key.

The reducer adds the list of values for each key to produce the final count.

🧠 Conceptual

expert

Which component manages the overall MapReduce job execution?

In Hadoop's MapReduce architecture, which component is responsible for managing the entire job execution, including resource allocation and task scheduling?

A NameNode

B TaskTracker

C JobTracker

D DataNode

💡 Hint

Think about the component that coordinates tasks across the cluster.

The JobTracker manages job execution, resource allocation, and task scheduling in Hadoop MapReduce.

Practice

1. What is the main role of the Map task in a MapReduce job?

easy

A. Process input data and create key-value pairs

B. Aggregate results from all nodes

C. Store the final output in HDFS

D. Schedule tasks on cluster nodes

Solution

1. Step 1: Understand the Map task function

   The Map task reads input data and transforms it into intermediate key-value pairs.

2. Step 2: Differentiate from Reduce task

   The Reduce task aggregates these key-value pairs, so it is not responsible for initial processing.

3. Final Answer:

   Process input data and create key-value pairs -> Option A

4. Quick Check:

   Map task = create key-value pairs [OK]

Hint: Map creates key-value pairs from input data [OK]

Common Mistakes:

• Confusing Map with Reduce task role
• Thinking Map stores final output
• Assuming Map schedules tasks

2. Which of the following is the correct sequence of steps in a MapReduce job execution?

easy

A. Map -> Shuffle -> Reduce

B. Shuffle -> Map -> Reduce

C. Reduce -> Map -> Shuffle

D. Reduce -> Shuffle -> Map

Solution

1. Step 1: Recall MapReduce execution order

   The job starts with Map tasks processing input, then Shuffle sorts and transfers data, followed by Reduce tasks aggregating results.

2. Step 2: Match sequence to options

   Only Map -> Shuffle -> Reduce correctly shows Map first, then Shuffle, then Reduce.

3. Final Answer:

   Map -> Shuffle -> Reduce -> Option A

4. Quick Check:

   Execution order = Map, Shuffle, Reduce [OK]

Hint: Remember: Map first, then Shuffle, then Reduce [OK]

Common Mistakes:

• Mixing up Shuffle and Map order
• Placing Reduce before Map
• Ignoring Shuffle step

3. Consider this simplified MapReduce code snippet:

map(String key, String value) {
  emit(value, 1);
}

reduce(String key, Iterator values) {
  int sum = 0;
  while (values.hasNext()) {
    sum += values.next();
  }
  emit(key, sum);
}

What will be the output for input: ["a", "apple"], ["b", "apple"], ["c", "banana"]?

medium

A. [('apple', 1), ('banana', 1)]

B. [('a', 1), ('b', 1), ('c', 1)]

C. [('apple', 2), ('banana', 1)]

D. [('apple', 3), ('banana', 1)]

Solution

1. Step 1: Analyze Map output

   Map emits (value, 1) pairs: ('apple',1), ('apple',1), ('banana',1).

2. Step 2: Analyze Reduce aggregation

   Reduce sums counts for each key: 'apple' = 1+1=2, 'banana' = 1.

3. Final Answer:

   [('apple', 2), ('banana', 1)] -> Option C

4. Quick Check:

   Count of 'apple' = 2, 'banana' = 1 [OK]

Hint: Map emits value with 1, Reduce sums counts by key [OK]

Common Mistakes:

• Counting keys instead of values
• Confusing input keys with values
• Adding counts incorrectly

4. Identify the error in this MapReduce job execution flow:

Map task reads input and emits key-value pairs.
Reduce task tries to read input directly from HDFS instead of shuffle output.
Reduce task aggregates values by key.

medium

A. Reduce task should emit key-value pairs before Map

B. Map task should aggregate values, not Reduce task

C. Map task should read from shuffle output

D. Reduce task should read from shuffle output, not directly from HDFS

Solution

1. Step 1: Understand Reduce input source

   Reduce tasks receive data from the shuffle phase, not directly from HDFS input files.

2. Step 2: Identify incorrect flow

   Reading directly from HDFS in Reduce breaks the flow and causes errors.

3. Final Answer:

   Reduce task should read from shuffle output, not directly from HDFS -> Option D

4. Quick Check:

   Reduce reads shuffle output [OK]

Hint: Reduce reads shuffle output, never direct HDFS input [OK]

Common Mistakes:

• Thinking Reduce reads input files
• Swapping Map and Reduce roles
• Ignoring shuffle phase

5. You want to optimize a MapReduce job that processes large logs by filtering out entries with empty user IDs before counting occurrences. Where should you apply this filter for best performance?

hard

A. In the Reduce task after aggregation

B. In the Map task before emitting key-value pairs

C. During the shuffle phase

D. After the job completes on the output files

Solution

1. Step 1: Understand filtering impact

   Filtering early reduces data volume, saving network and processing time.

2. Step 2: Apply filter in Map task

   Filtering in Map avoids sending unnecessary data to shuffle and Reduce phases.

3. Final Answer:

   In the Map task before emitting key-value pairs -> Option B

4. Quick Check:

   Filter early in Map for best performance [OK]

Hint: Filter data early in Map to reduce workload [OK]

Common Mistakes:

• Filtering only in Reduce wastes resources
• Trying to filter during shuffle (not possible)
• Filtering after job completion is too late