MapReduce job execution flow in Hadoop - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to specify the input format class in a MapReduce job configuration.

Hadoop

job.setInputFormatClass([1]);

Drag options to blanks, or click blank then click option'

ATextInputFormat.class

BFileOutputFormat.class

CMapper.class

DReducer.class

Attempts:

3 left

💡 Hint

Common Mistakes

Confusing input format with output format.

Using Mapper or Reducer classes instead of input format.

The input format class defines how input files are split and read. TextInputFormat is commonly used for text files.

2fill in blank

medium

Complete the code to set the number of reduce tasks in a MapReduce job.

Hadoop

job.setNumReduceTasks([1]);

Drag options to blanks, or click blank then click option'

A0

B-1

C10

D1

Attempts:

3 left

💡 Hint

Common Mistakes

Setting reduce tasks to zero when reduce phase is needed.

Using negative numbers which are invalid.

Setting the number of reduce tasks to 1 means the job will have one reducer. Zero means no reduce phase.

3fill in blank

hard

Fix the error in the code to correctly set the output key class for a MapReduce job.

Hadoop

job.setOutputKeyClass([1]);

Drag options to blanks, or click blank then click option'

AText.class

BIntWritable.class

CTextInputFormat.class

DMapper.class

Attempts:

3 left

💡 Hint

Common Mistakes

Using input format or Mapper classes instead of data types.

Choosing a value class instead of key class.

The output key class defines the data type of the output keys. Text.class is commonly used for string keys.

4fill in blank

hard

Fill both blanks to set the output key class and output value class in a MapReduce job.

Hadoop

job.setOutputKeyClass([1]); job.setOutputValueClass([2]);

Drag options to blanks, or click blank then click option'

AText.class

BIntWritable.class

CTextInputFormat.class

DMapper.class

Attempts:

3 left

💡 Hint

Common Mistakes

Selecting mapper or format classes.

Wrong type for key/value (e.g., IntWritable for key).

The output key class is typically Text.class for strings (e.g., words), and output value class IntWritable.class for integers (e.g., counts) in jobs like word count.

5fill in blank

hard

Fill all three blanks to configure map output key class, value class, and combiner class in a MapReduce job.

Hadoop

job.setMapOutputKeyClass([1]); job.setMapOutputValueClass([2]); job.setCombinerClass([3]);

Drag options to blanks, or click blank then click option'

AText.class

BIntWritable.class

CReducer.class

DMapper.class

Attempts:

3 left

💡 Hint

Common Mistakes

Confusing map output with final output classes.

Using Mapper.class for combiner.

Map output key Text.class, value IntWritable.class for intermediate data, combiner Reducer.class for local aggregation on map nodes.

Practice

1. What is the main role of the Map task in a MapReduce job?

easy

A. Process input data and create key-value pairs

B. Aggregate results from all nodes

C. Store the final output in HDFS

D. Schedule tasks on cluster nodes

Solution

1. Step 1: Understand the Map task function

   The Map task reads input data and transforms it into intermediate key-value pairs.

2. Step 2: Differentiate from Reduce task

   The Reduce task aggregates these key-value pairs, so it is not responsible for initial processing.

3. Final Answer:

   Process input data and create key-value pairs -> Option A

4. Quick Check:

   Map task = create key-value pairs [OK]

Hint: Map creates key-value pairs from input data [OK]

Common Mistakes:

• Confusing Map with Reduce task role
• Thinking Map stores final output
• Assuming Map schedules tasks

2. Which of the following is the correct sequence of steps in a MapReduce job execution?

easy

A. Map -> Shuffle -> Reduce

B. Shuffle -> Map -> Reduce

C. Reduce -> Map -> Shuffle

D. Reduce -> Shuffle -> Map

Solution

1. Step 1: Recall MapReduce execution order

   The job starts with Map tasks processing input, then Shuffle sorts and transfers data, followed by Reduce tasks aggregating results.

2. Step 2: Match sequence to options

   Only Map -> Shuffle -> Reduce correctly shows Map first, then Shuffle, then Reduce.

3. Final Answer:

   Map -> Shuffle -> Reduce -> Option A

4. Quick Check:

   Execution order = Map, Shuffle, Reduce [OK]

Hint: Remember: Map first, then Shuffle, then Reduce [OK]

Common Mistakes:

• Mixing up Shuffle and Map order
• Placing Reduce before Map
• Ignoring Shuffle step

3. Consider this simplified MapReduce code snippet:

map(String key, String value) {
  emit(value, 1);
}

reduce(String key, Iterator values) {
  int sum = 0;
  while (values.hasNext()) {
    sum += values.next();
  }
  emit(key, sum);
}

What will be the output for input: ["a", "apple"], ["b", "apple"], ["c", "banana"]?

medium

A. [('apple', 1), ('banana', 1)]

B. [('a', 1), ('b', 1), ('c', 1)]

C. [('apple', 2), ('banana', 1)]

D. [('apple', 3), ('banana', 1)]

Solution

1. Step 1: Analyze Map output

   Map emits (value, 1) pairs: ('apple',1), ('apple',1), ('banana',1).

2. Step 2: Analyze Reduce aggregation

   Reduce sums counts for each key: 'apple' = 1+1=2, 'banana' = 1.

3. Final Answer:

   [('apple', 2), ('banana', 1)] -> Option C

4. Quick Check:

   Count of 'apple' = 2, 'banana' = 1 [OK]

Hint: Map emits value with 1, Reduce sums counts by key [OK]

Common Mistakes:

• Counting keys instead of values
• Confusing input keys with values
• Adding counts incorrectly

4. Identify the error in this MapReduce job execution flow:

Map task reads input and emits key-value pairs.
Reduce task tries to read input directly from HDFS instead of shuffle output.
Reduce task aggregates values by key.

medium

A. Reduce task should emit key-value pairs before Map

B. Map task should aggregate values, not Reduce task

C. Map task should read from shuffle output

D. Reduce task should read from shuffle output, not directly from HDFS

Solution

1. Step 1: Understand Reduce input source

   Reduce tasks receive data from the shuffle phase, not directly from HDFS input files.

2. Step 2: Identify incorrect flow

   Reading directly from HDFS in Reduce breaks the flow and causes errors.

3. Final Answer:

   Reduce task should read from shuffle output, not directly from HDFS -> Option D

4. Quick Check:

   Reduce reads shuffle output [OK]

Hint: Reduce reads shuffle output, never direct HDFS input [OK]

Common Mistakes:

• Thinking Reduce reads input files
• Swapping Map and Reduce roles
• Ignoring shuffle phase

5. You want to optimize a MapReduce job that processes large logs by filtering out entries with empty user IDs before counting occurrences. Where should you apply this filter for best performance?

hard

A. In the Reduce task after aggregation

B. In the Map task before emitting key-value pairs

C. During the shuffle phase

D. After the job completes on the output files

Solution

1. Step 1: Understand filtering impact

   Filtering early reduces data volume, saving network and processing time.

2. Step 2: Apply filter in Map task

   Filtering in Map avoids sending unnecessary data to shuffle and Reduce phases.

3. Final Answer:

   In the Map task before emitting key-value pairs -> Option B

4. Quick Check:

   Filter early in Map for best performance [OK]

Hint: Filter data early in Map to reduce workload [OK]

Common Mistakes:

• Filtering only in Reduce wastes resources
• Trying to filter during shuffle (not possible)
• Filtering after job completion is too late