MapReduce job tuning parameters in Hadoop - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of the mapreduce.map.memory.mb parameter?

It sets the amount of memory (in MB) allocated to each map task. Increasing this can help map tasks handle larger data or complex processing.

intermediate

How does mapreduce.reduce.shuffle.parallelcopies affect job performance?

It controls the number of parallel transfers during the shuffle phase for reduce tasks. More parallel copies can speed up data transfer but may increase network load.

beginner

What does the mapreduce.task.io.sort.mb parameter control?

It sets the buffer size (in MB) for sorting map output before it is written to disk. Larger buffers reduce disk I/O but use more memory.

beginner

Why would you adjust mapreduce.reduce.memory.mb in a MapReduce job?

To allocate more memory to reduce tasks, which can help when reduce tasks process large amounts of data or perform complex computations.

intermediate

What is the effect of increasing mapreduce.job.reduces?

It increases the number of reduce tasks. This can speed up the reduce phase by parallelizing work but may increase overhead if set too high.

Which parameter controls the memory allocated to map tasks?

A mapreduce.job.reduces

B mapreduce.reduce.memory.mb

C mapreduce.task.io.sort.mb

D mapreduce.map.memory.mb

mapreduce.map.memory.mb sets memory for map tasks.

What does increasing mapreduce.job.reduces do?

A Increases map tasks

B Increases memory for map tasks

C Increases reduce tasks

D Increases shuffle parallel copies

mapreduce.job.reduces sets the number of reduce tasks.

Which parameter affects the buffer size for sorting map output?

A mapreduce.task.io.sort.mb

B mapreduce.reduce.shuffle.parallelcopies

C mapreduce.map.memory.mb

D mapreduce.reduce.memory.mb

mapreduce.task.io.sort.mb controls the sort buffer size.

What is the role of mapreduce.reduce.shuffle.parallelcopies?

A Memory for reduce tasks

B Parallel data transfers during shuffle

C Number of map tasks

D Buffer size for map output

It controls parallel transfers in the shuffle phase.

Why increase mapreduce.reduce.memory.mb?

A To allocate more memory to reduce tasks

B To increase map tasks

C To increase shuffle copies

D To increase sort buffer size

It allocates more memory to reduce tasks for heavy processing.

Explain how tuning mapreduce.task.io.sort.mb and mapreduce.map.memory.mb can improve map task performance.

Think about memory and sorting buffers in map tasks.

Describe the impact of increasing mapreduce.job.reduces and mapreduce.reduce.shuffle.parallelcopies on reduce phase performance.

Consider parallelism and data transfer in reduce tasks.