MapReduce job tuning parameters in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding the impact of map task memory allocation

Which of the following tuning parameters directly controls the maximum memory allocated to each map task in a Hadoop MapReduce job?

A mapreduce.reduce.memory.mb

B mapreduce.job.reduces

C mapreduce.task.io.sort.mb

D mapreduce.map.memory.mb

💡 Hint

Think about which parameter sets memory limits specifically for map tasks.

The parameter mapreduce.map.memory.mb sets the maximum memory allocated to each map task. Other options control reduce task memory, sort buffer size, or number of reducers.

❓ data_output

intermediate

Effect of changing number of reducers on output files

Given a MapReduce job with 10 reducers, what will be the number of output files generated in HDFS after job completion?

A 10

B Depends on input splits

C Number of map tasks

D 1

💡 Hint

Each reducer writes one output file.

The number of output files equals the number of reducers because each reducer writes its own output file.

❓ Predict Output

advanced

Analyzing the effect of io.sort.mb on map task output

Consider the following MapReduce configuration snippet:

mapreduce.task.io.sort.mb=100

What is the effect of increasing mapreduce.task.io.sort.mb from 100 to 300?

A Increases the buffer size for sorting map outputs, reducing spills and improving performance

B Increases the memory allocated to reduce tasks, allowing more parallelism

C Limits the maximum size of input splits processed by map tasks

D Controls the number of map tasks launched per node

💡 Hint

Focus on what io.sort.mb controls in the map phase.

The mapreduce.task.io.sort.mb parameter sets the size of the buffer used to sort map outputs before spilling to disk. Increasing it reduces the number of spills and can improve performance.

🔧 Debug

advanced

Identifying cause of slow reduce tasks

A MapReduce job has many reduce tasks running slowly. Which tuning parameter adjustment is most likely to improve reduce task performance?

A Lower <code>mapreduce.map.memory.mb</code> to free cluster memory

B Decrease <code>mapreduce.task.io.sort.mb</code> to reduce map output buffer size

C Increase <code>mapreduce.reduce.shuffle.parallelcopies</code> to allow more parallel fetches

D Set <code>mapreduce.job.reduces</code> to 1 to reduce overhead

💡 Hint

Consider how reduce tasks fetch map outputs.

Increasing mapreduce.reduce.shuffle.parallelcopies allows reduce tasks to fetch map outputs in parallel, speeding up the shuffle phase and improving reduce task performance.

🚀 Application

expert

Optimizing a MapReduce job for large input data

You have a MapReduce job processing very large input files. The job is slow due to frequent spills during the map phase. Which combination of tuning parameters is best to reduce spills and improve performance?

A Decrease <code>mapreduce.task.io.sort.mb</code> and increase <code>mapreduce.reduce.memory.mb</code>

B Increase <code>mapreduce.task.io.sort.mb</code> and increase <code>mapreduce.map.memory.mb</code>

C Increase <code>mapreduce.job.reduces</code> and decrease <code>mapreduce.map.memory.mb</code>

D Decrease <code>mapreduce.reduce.shuffle.parallelcopies</code> and increase <code>mapreduce.task.io.sort.mb</code>

💡 Hint

Think about memory available for sorting map outputs and map task memory limits.

Increasing mapreduce.task.io.sort.mb enlarges the buffer for sorting map outputs, reducing spills. Increasing mapreduce.map.memory.mb ensures map tasks have enough memory to handle the larger buffer.