MapReduce job execution flow in Hadoop - Time & Space Complexity

Time Complexity: MapReduce job execution flow

O(n)

Understanding Time Complexity

We want to understand how the time to run a MapReduce job changes as the input data grows.

How does the job's execution time grow when we add more data to process?

Scenario Under Consideration

Analyze the time complexity of the following MapReduce job execution flow.

// Simplified MapReduce job flow
Job job = new Job(conf, "word count");
job.setMapperClass(WordCountMapper.class);
job.setReducerClass(WordCountReducer.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(inputPath));
FileOutputFormat.setOutputPath(job, new Path(outputPath));
job.waitForCompletion(true);

This code sets up and runs a MapReduce job that counts words in input files.

Identify Repeating Operations

• Primary operation: The Mapper processes each input record once.
• How many times: Once per input record, so as many times as there are records.
• Secondary operation: The Reducer processes each unique key and its values.
• How many times: Once per unique key, which depends on data.

How Execution Grows With Input

As input data grows, the Mapper runs more times, once per record.

Input Size (n)	Approx. Operations
10	About 10 map operations
100	About 100 map operations
1000	About 1000 map operations

Pattern observation: The number of map operations grows directly with input size.

Final Time Complexity

Time Complexity: O(n)

This means the job's running time grows roughly in direct proportion to the amount of input data.

Common Mistake

[X] Wrong: "The Reducer runs once regardless of input size."

[OK] Correct: The Reducer runs once per unique key, so if input grows and produces more unique keys, the Reducer work also grows.

Interview Connect

Understanding how MapReduce scales with data size helps you explain how big data jobs handle growth efficiently.

Self-Check

"What if we added a Combiner step before the Reducer? How would the time complexity change?"

Practice

1. What is the main role of the Map task in a MapReduce job?

easy

A. Process input data and create key-value pairs

B. Aggregate results from all nodes

C. Store the final output in HDFS

D. Schedule tasks on cluster nodes

Solution

1. Step 1: Understand the Map task function

   The Map task reads input data and transforms it into intermediate key-value pairs.

2. Step 2: Differentiate from Reduce task

   The Reduce task aggregates these key-value pairs, so it is not responsible for initial processing.

3. Final Answer:

   Process input data and create key-value pairs -> Option A

4. Quick Check:

   Map task = create key-value pairs [OK]

Hint: Map creates key-value pairs from input data [OK]

Common Mistakes:

• Confusing Map with Reduce task role
• Thinking Map stores final output
• Assuming Map schedules tasks

2. Which of the following is the correct sequence of steps in a MapReduce job execution?

easy

A. Map -> Shuffle -> Reduce

B. Shuffle -> Map -> Reduce

C. Reduce -> Map -> Shuffle

D. Reduce -> Shuffle -> Map

Solution

1. Step 1: Recall MapReduce execution order

   The job starts with Map tasks processing input, then Shuffle sorts and transfers data, followed by Reduce tasks aggregating results.

2. Step 2: Match sequence to options

   Only Map -> Shuffle -> Reduce correctly shows Map first, then Shuffle, then Reduce.

3. Final Answer:

   Map -> Shuffle -> Reduce -> Option A

4. Quick Check:

   Execution order = Map, Shuffle, Reduce [OK]

Hint: Remember: Map first, then Shuffle, then Reduce [OK]

Common Mistakes:

• Mixing up Shuffle and Map order
• Placing Reduce before Map
• Ignoring Shuffle step

3. Consider this simplified MapReduce code snippet:

map(String key, String value) {
  emit(value, 1);
}

reduce(String key, Iterator values) {
  int sum = 0;
  while (values.hasNext()) {
    sum += values.next();
  }
  emit(key, sum);
}

What will be the output for input: ["a", "apple"], ["b", "apple"], ["c", "banana"]?

medium

A. [('apple', 1), ('banana', 1)]

B. [('a', 1), ('b', 1), ('c', 1)]

C. [('apple', 2), ('banana', 1)]

D. [('apple', 3), ('banana', 1)]

Solution

1. Step 1: Analyze Map output

   Map emits (value, 1) pairs: ('apple',1), ('apple',1), ('banana',1).

2. Step 2: Analyze Reduce aggregation

   Reduce sums counts for each key: 'apple' = 1+1=2, 'banana' = 1.

3. Final Answer:

   [('apple', 2), ('banana', 1)] -> Option C

4. Quick Check:

   Count of 'apple' = 2, 'banana' = 1 [OK]

Hint: Map emits value with 1, Reduce sums counts by key [OK]

Common Mistakes:

• Counting keys instead of values
• Confusing input keys with values
• Adding counts incorrectly

4. Identify the error in this MapReduce job execution flow:

Map task reads input and emits key-value pairs.
Reduce task tries to read input directly from HDFS instead of shuffle output.
Reduce task aggregates values by key.

medium

A. Reduce task should emit key-value pairs before Map

B. Map task should aggregate values, not Reduce task

C. Map task should read from shuffle output

D. Reduce task should read from shuffle output, not directly from HDFS

Solution

1. Step 1: Understand Reduce input source

   Reduce tasks receive data from the shuffle phase, not directly from HDFS input files.

2. Step 2: Identify incorrect flow

   Reading directly from HDFS in Reduce breaks the flow and causes errors.

3. Final Answer:

   Reduce task should read from shuffle output, not directly from HDFS -> Option D

4. Quick Check:

   Reduce reads shuffle output [OK]

Hint: Reduce reads shuffle output, never direct HDFS input [OK]

Common Mistakes:

• Thinking Reduce reads input files
• Swapping Map and Reduce roles
• Ignoring shuffle phase

5. You want to optimize a MapReduce job that processes large logs by filtering out entries with empty user IDs before counting occurrences. Where should you apply this filter for best performance?

hard

A. In the Reduce task after aggregation

B. In the Map task before emitting key-value pairs

C. During the shuffle phase

D. After the job completes on the output files

Solution

1. Step 1: Understand filtering impact

   Filtering early reduces data volume, saving network and processing time.

2. Step 2: Apply filter in Map task

   Filtering in Map avoids sending unnecessary data to shuffle and Reduce phases.

3. Final Answer:

   In the Map task before emitting key-value pairs -> Option B

4. Quick Check:

   Filter early in Map for best performance [OK]

Hint: Filter data early in Map to reduce workload [OK]

Common Mistakes:

• Filtering only in Reduce wastes resources
• Trying to filter during shuffle (not possible)
• Filtering after job completion is too late