MapReduce job tuning parameters in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding the impact of map task memory allocation

Which of the following tuning parameters directly controls the maximum memory allocated to each map task in a Hadoop MapReduce job?

A mapreduce.reduce.memory.mb

B mapreduce.job.reduces

C mapreduce.task.io.sort.mb

D mapreduce.map.memory.mb

💡 Hint

Think about which parameter sets memory limits specifically for map tasks.

The parameter mapreduce.map.memory.mb sets the maximum memory allocated to each map task. Other options control reduce task memory, sort buffer size, or number of reducers.

❓ data_output

intermediate

Effect of changing number of reducers on output files

Given a MapReduce job with 10 reducers, what will be the number of output files generated in HDFS after job completion?

A 10

B Depends on input splits

C Number of map tasks

D 1

💡 Hint

Each reducer writes one output file.

The number of output files equals the number of reducers because each reducer writes its own output file.

❓ Predict Output

advanced

Analyzing the effect of io.sort.mb on map task output

Consider the following MapReduce configuration snippet:

mapreduce.task.io.sort.mb=100

What is the effect of increasing mapreduce.task.io.sort.mb from 100 to 300?

A Increases the buffer size for sorting map outputs, reducing spills and improving performance

B Increases the memory allocated to reduce tasks, allowing more parallelism

C Limits the maximum size of input splits processed by map tasks

D Controls the number of map tasks launched per node

💡 Hint

Focus on what io.sort.mb controls in the map phase.

The mapreduce.task.io.sort.mb parameter sets the size of the buffer used to sort map outputs before spilling to disk. Increasing it reduces the number of spills and can improve performance.

🔧 Debug

advanced

Identifying cause of slow reduce tasks

A MapReduce job has many reduce tasks running slowly. Which tuning parameter adjustment is most likely to improve reduce task performance?

A Lower <code>mapreduce.map.memory.mb</code> to free cluster memory

B Decrease <code>mapreduce.task.io.sort.mb</code> to reduce map output buffer size

C Increase <code>mapreduce.reduce.shuffle.parallelcopies</code> to allow more parallel fetches

D Set <code>mapreduce.job.reduces</code> to 1 to reduce overhead

💡 Hint

Consider how reduce tasks fetch map outputs.

Increasing mapreduce.reduce.shuffle.parallelcopies allows reduce tasks to fetch map outputs in parallel, speeding up the shuffle phase and improving reduce task performance.

🚀 Application

expert

Optimizing a MapReduce job for large input data

You have a MapReduce job processing very large input files. The job is slow due to frequent spills during the map phase. Which combination of tuning parameters is best to reduce spills and improve performance?

A Decrease <code>mapreduce.task.io.sort.mb</code> and increase <code>mapreduce.reduce.memory.mb</code>

B Increase <code>mapreduce.task.io.sort.mb</code> and increase <code>mapreduce.map.memory.mb</code>

C Increase <code>mapreduce.job.reduces</code> and decrease <code>mapreduce.map.memory.mb</code>

D Decrease <code>mapreduce.reduce.shuffle.parallelcopies</code> and increase <code>mapreduce.task.io.sort.mb</code>

💡 Hint

Think about memory available for sorting map outputs and map task memory limits.

Increasing mapreduce.task.io.sort.mb enlarges the buffer for sorting map outputs, reducing spills. Increasing mapreduce.map.memory.mb ensures map tasks have enough memory to handle the larger buffer.

Practice

1. Which MapReduce parameter controls the number of reduce tasks in a job?

easy

A. mapreduce.job.reduces

B. mapreduce.job.maps

C. mapreduce.task.io.sort.mb

D. mapreduce.reduce.shuffle.parallelcopies

Solution

1. Step 1: Understand the role of reduce tasks

   Reduce tasks process the output from map tasks and aggregate results.

2. Step 2: Identify the parameter controlling reduce tasks

   mapreduce.job.reduces sets the number of reduce tasks in a MapReduce job.

3. Final Answer:

   mapreduce.job.reduces -> Option A

4. Quick Check:

   Reduce tasks count = mapreduce.job.reduces [OK]

Hint: Reducers controlled by mapreduce.job.reduces parameter [OK]

Common Mistakes:

• Confusing mapreduce.job.maps with reducers
• Thinking shuffle parameters control reduce count
• Mixing memory parameters with task counts

2. Which of the following is the correct syntax to set the memory allocated to each map task in a Hadoop job configuration?

easy

A. mapreduce.map.memory.mb=2048

B. mapreduce.task.memory.map=2048

C. mapreduce.memory.map.task=2048

D. mapreduce.mapreduce.memory.mb=2048

Solution

1. Step 1: Recall the standard memory parameter for map tasks

   The correct parameter to set map task memory is mapreduce.map.memory.mb.

2. Step 2: Verify syntax correctness

   mapreduce.map.memory.mb=2048 matches the exact syntax used in Hadoop configurations.

3. Final Answer:

   mapreduce.map.memory.mb=2048 -> Option A

4. Quick Check:

   Map task memory = mapreduce.map.memory.mb [OK]

Hint: Map memory uses mapreduce.map.memory.mb [OK]

Common Mistakes:

• Swapping map and reduce memory parameters
• Using incorrect parameter names
• Adding extra words like 'task' incorrectly

3. Given the following configuration snippet in a MapReduce job:

mapreduce.job.reduces=3
mapreduce.map.memory.mb=1024
mapreduce.reduce.memory.mb=2048

What will be the number of reduce tasks and memory allocated to each reduce task?

medium

A. 1 reduce task, 2048 MB memory each

B. 3 reduce tasks, 1024 MB memory each

C. 3 reduce tasks, 2048 MB memory each

D. 1 reduce task, 1024 MB memory each

Solution

1. Step 1: Identify number of reduce tasks

   The parameter mapreduce.job.reduces=3 sets 3 reduce tasks.

2. Step 2: Identify memory per reduce task

   mapreduce.reduce.memory.mb=2048 sets 2048 MB memory for each reduce task.

3. Final Answer:

   3 reduce tasks, 2048 MB memory each -> Option C

4. Quick Check:

   Reducers=3, Reduce memory=2048 MB [OK]

Hint: Reducers = mapreduce.job.reduces; reduce memory = mapreduce.reduce.memory.mb [OK]

Common Mistakes:

• Confusing map memory with reduce memory
• Assuming default reduce task count is 1 always
• Ignoring memory parameters

4. You have set mapreduce.job.reduces=0 in your job configuration, but your job still runs reduce tasks. What is the likely cause?

medium

A. The job uses a custom partitioner forcing reduce tasks

B. The parameter is overridden by command line or code

C. Reduce tasks run by default regardless of setting

D. Map tasks are mistakenly counted as reduce tasks

Solution

1. Step 1: Understand effect of mapreduce.job.reduces=0

   This setting means no reduce tasks should run.

2. Step 2: Identify why reduce tasks still run

   If reduce tasks run, likely the setting is overridden by command line options or programmatic code.

3. Final Answer:

   The parameter is overridden by command line or code -> Option B

4. Quick Check:

   Overrides cause unexpected reduce tasks [OK]

Hint: Check for overrides if reduce count ignored [OK]

Common Mistakes:

• Assuming reduce tasks always run regardless
• Confusing map tasks with reduce tasks
• Ignoring programmatic overrides

5. You want to optimize a MapReduce job that processes large input data with many small files. Which tuning parameter adjustment can help reduce overhead and improve performance?

hard

A. Increase mapreduce.reduce.shuffle.parallelcopies to 1 to limit network usage

B. Decrease mapreduce.job.reduces to zero to skip reduce phase

C. Set mapreduce.task.io.sort.mb to a very low value to reduce memory use

D. Increase mapreduce.input.fileinputformat.split.maxsize to combine small files into fewer splits

Solution

1. Step 1: Understand problem with many small files

   Many small files cause many small input splits, increasing overhead.

2. Step 2: Identify tuning to reduce splits

   Increasing mapreduce.input.fileinputformat.split.maxsize combines small files into fewer splits, reducing overhead.

3. Step 3: Evaluate other options

   Setting reducers to zero skips reduce phase but may lose aggregation. Low sort memory hurts performance. Limiting shuffle copies to 1 reduces parallelism.

4. Final Answer:

   Increase mapreduce.input.fileinputformat.split.maxsize to combine small files into fewer splits -> Option D

5. Quick Check:

   Combine small files by increasing split max size [OK]

Hint: Combine small files by increasing split max size [OK]

Common Mistakes:

• Setting reducers to zero losing reduce benefits
• Reducing sort memory hurting map output sorting
• Limiting shuffle copies reducing network throughput