MapReduce job execution flow in Hadoop - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the first step in the MapReduce job execution flow?

The first step is Job Submission, where the client submits the job to the Hadoop cluster, including the code and input data location.

intermediate

What role does the JobTracker play in MapReduce execution?

The JobTracker manages the job by splitting it into tasks, assigning them to TaskTrackers, and monitoring their progress.

beginner

Explain the function of the Mapper in MapReduce.

The Mapper processes input data splits and produces intermediate key-value pairs for further processing.

intermediate

What happens during the Shuffle and Sort phase?

Intermediate data from Mappers is shuffled and sorted by keys to prepare for the Reducer, ensuring all values for a key are grouped together.

beginner

Describe the final step in the MapReduce job execution flow.

The final step is Job Completion, where the Reducers write output to HDFS, and the JobTracker reports job status back to the client.

What component assigns tasks to nodes in a Hadoop cluster?

A NameNode

B JobTracker

C DataNode

D ResourceManager

The JobTracker is responsible for assigning MapReduce tasks to TaskTrackers (nodes).

During which phase are intermediate key-value pairs grouped by key?

A Job submission

B Map phase

C Reduce phase

D Shuffle and Sort phase

The Shuffle and Sort phase groups intermediate data by key before reducing.

What does the Mapper output in MapReduce?

A Intermediate key-value pairs

B Input splits

C Job status

D Final results

Mappers output intermediate key-value pairs for the Reducers.

Where is the final output of a MapReduce job stored?

A HDFS

B Local disk

C JobTracker memory

D Mapper output buffer

The final output is stored in the Hadoop Distributed File System (HDFS).

Which phase immediately follows the Map phase?

A Job submission

B Reduce phase

C Shuffle and Sort phase

D Job completion

After mapping, data is shuffled and sorted before reducing.

Describe the complete flow of a MapReduce job from submission to completion.

Think about how data moves and changes from start to finish in the cluster.

Explain the purpose of the Shuffle and Sort phase in MapReduce.

Consider why Reducers need data organized before processing.

Practice

1. What is the main role of the Map task in a MapReduce job?

easy

A. Process input data and create key-value pairs

B. Aggregate results from all nodes

C. Store the final output in HDFS

D. Schedule tasks on cluster nodes

Solution

1. Step 1: Understand the Map task function

   The Map task reads input data and transforms it into intermediate key-value pairs.

2. Step 2: Differentiate from Reduce task

   The Reduce task aggregates these key-value pairs, so it is not responsible for initial processing.

3. Final Answer:

   Process input data and create key-value pairs -> Option A

4. Quick Check:

   Map task = create key-value pairs [OK]

Hint: Map creates key-value pairs from input data [OK]

Common Mistakes:

• Confusing Map with Reduce task role
• Thinking Map stores final output
• Assuming Map schedules tasks

2. Which of the following is the correct sequence of steps in a MapReduce job execution?

easy

A. Map -> Shuffle -> Reduce

B. Shuffle -> Map -> Reduce

C. Reduce -> Map -> Shuffle

D. Reduce -> Shuffle -> Map

Solution

1. Step 1: Recall MapReduce execution order

   The job starts with Map tasks processing input, then Shuffle sorts and transfers data, followed by Reduce tasks aggregating results.

2. Step 2: Match sequence to options

   Only Map -> Shuffle -> Reduce correctly shows Map first, then Shuffle, then Reduce.

3. Final Answer:

   Map -> Shuffle -> Reduce -> Option A

4. Quick Check:

   Execution order = Map, Shuffle, Reduce [OK]

Hint: Remember: Map first, then Shuffle, then Reduce [OK]

Common Mistakes:

• Mixing up Shuffle and Map order
• Placing Reduce before Map
• Ignoring Shuffle step

3. Consider this simplified MapReduce code snippet:

map(String key, String value) {
  emit(value, 1);
}

reduce(String key, Iterator values) {
  int sum = 0;
  while (values.hasNext()) {
    sum += values.next();
  }
  emit(key, sum);
}

What will be the output for input: ["a", "apple"], ["b", "apple"], ["c", "banana"]?

medium

A. [('apple', 1), ('banana', 1)]

B. [('a', 1), ('b', 1), ('c', 1)]

C. [('apple', 2), ('banana', 1)]

D. [('apple', 3), ('banana', 1)]

Solution

1. Step 1: Analyze Map output

   Map emits (value, 1) pairs: ('apple',1), ('apple',1), ('banana',1).

2. Step 2: Analyze Reduce aggregation

   Reduce sums counts for each key: 'apple' = 1+1=2, 'banana' = 1.

3. Final Answer:

   [('apple', 2), ('banana', 1)] -> Option C

4. Quick Check:

   Count of 'apple' = 2, 'banana' = 1 [OK]

Hint: Map emits value with 1, Reduce sums counts by key [OK]

Common Mistakes:

• Counting keys instead of values
• Confusing input keys with values
• Adding counts incorrectly

4. Identify the error in this MapReduce job execution flow:

Map task reads input and emits key-value pairs.
Reduce task tries to read input directly from HDFS instead of shuffle output.
Reduce task aggregates values by key.

medium

A. Reduce task should emit key-value pairs before Map

B. Map task should aggregate values, not Reduce task

C. Map task should read from shuffle output

D. Reduce task should read from shuffle output, not directly from HDFS

Solution

1. Step 1: Understand Reduce input source

   Reduce tasks receive data from the shuffle phase, not directly from HDFS input files.

2. Step 2: Identify incorrect flow

   Reading directly from HDFS in Reduce breaks the flow and causes errors.

3. Final Answer:

   Reduce task should read from shuffle output, not directly from HDFS -> Option D

4. Quick Check:

   Reduce reads shuffle output [OK]

Hint: Reduce reads shuffle output, never direct HDFS input [OK]

Common Mistakes:

• Thinking Reduce reads input files
• Swapping Map and Reduce roles
• Ignoring shuffle phase

5. You want to optimize a MapReduce job that processes large logs by filtering out entries with empty user IDs before counting occurrences. Where should you apply this filter for best performance?

hard

A. In the Reduce task after aggregation

B. In the Map task before emitting key-value pairs

C. During the shuffle phase

D. After the job completes on the output files

Solution

1. Step 1: Understand filtering impact

   Filtering early reduces data volume, saving network and processing time.

2. Step 2: Apply filter in Map task

   Filtering in Map avoids sending unnecessary data to shuffle and Reduce phases.

3. Final Answer:

   In the Map task before emitting key-value pairs -> Option B

4. Quick Check:

   Filter early in Map for best performance [OK]

Hint: Filter data early in Map to reduce workload [OK]

Common Mistakes:

• Filtering only in Reduce wastes resources
• Trying to filter during shuffle (not possible)
• Filtering after job completion is too late