MapReduce job execution flow in Hadoop - Step-by-Step Execution

Concept Flow - MapReduce job execution flow

Submit Job

↓

Split Input Data

↓

Map Tasks Start

↓

Map Tasks Produce (key,value)

↓

Shuffle and Sort

↓

Reduce Tasks Start

↓

Reduce Tasks Aggregate Results

↓

Write Output

↓

Job Complete

The flow starts with job submission, then input data is split and processed by map tasks. Map outputs are shuffled and sorted, then reduce tasks aggregate results and write output, ending the job.

Execution Sample

Hadoop

1. Submit job
2. Split input
3. Run map tasks
4. Shuffle and sort
5. Run reduce tasks
6. Write output

This sequence shows the main steps of a MapReduce job from start to finish.

Execution Table

Step	Action	Input	Output	Notes
1	Submit Job	User program	Job configuration	Job is sent to Hadoop cluster
2	Split Input Data	Large input file	Input splits	Data divided for parallel processing
3	Run Map Tasks	Input splits	(key,value) pairs	Map function processes each split
4	Shuffle and Sort	(key,value) pairs	Sorted (key, list(values))	Groups values by key across all maps
5	Run Reduce Tasks	Sorted (key, list(values))	Aggregated results	Reduce function processes grouped data
6	Write Output	Aggregated results	Output files	Results saved to distributed storage
7	Job Complete	Output files	Success status	Job finishes successfully

💡 Job completes after output is written and success status is returned

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	Final
Input Data	Large file	Split into chunks	Chunks processed by map	Mapped pairs ready	Reduced results ready	Output files
Map Output	N/A	N/A	(key,value) pairs	Grouped by key	Aggregated results	Saved output
Reduce Output	N/A	N/A	N/A	N/A	Aggregated results	Saved output

Key Moments - 3 Insights

Why does the input data get split before mapping?

Splitting input allows parallel processing by multiple map tasks, speeding up the job as shown in execution_table step 2.

What happens during the shuffle and sort phase?

Shuffle and sort groups all map outputs by key so reduce tasks can process all values for each key together, as seen in step 4.

When is the job considered complete?

After reduce tasks write the output files and the job returns a success status, shown in steps 6 and 7.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output after step 3 (Run Map Tasks)?

A Input splits

B Aggregated results

C (key,value) pairs

D Output files

💡 Hint

Check the 'Output' column for step 3 in the execution_table.

At which step does the job split the input data for parallel processing?

A Step 2

B Step 1

C Step 4

D Step 5

💡 Hint

Look at the 'Action' column and find where input data is divided.

If the shuffle and sort phase was skipped, what would likely happen?

A Map tasks would fail to run

B Reduce tasks would receive unsorted data and might miss some keys

C Input data would not be split

D Output files would be written earlier

💡 Hint

Refer to the purpose of shuffle and sort in grouping keys before reduce tasks.

Concept Snapshot

MapReduce job flow:
1. Submit job
2. Split input data
3. Map tasks process splits
4. Shuffle and sort map outputs
5. Reduce tasks aggregate data
6. Write output files
Splitting enables parallelism; shuffle groups keys for reduce.

Full Transcript

A MapReduce job starts when a user submits it to the Hadoop cluster. The input data is split into smaller chunks so multiple map tasks can run in parallel. Each map task processes its input split and produces key-value pairs. These pairs are shuffled and sorted to group all values by their keys. Reduce tasks then process each key and its list of values to aggregate results. Finally, the reduce outputs are written to output files in distributed storage. The job completes successfully after writing output. This flow enables processing large data efficiently by dividing work and combining results.