MapReduce job tuning parameters in Hadoop

Introduction

MapReduce job tuning parameters help make your data processing faster and use resources better.

When your MapReduce job runs too slow and you want to speed it up.

When your job uses too much memory or CPU and crashes.

When you want to balance work between many computers evenly.

When you want to control how much data each step processes.

When you want to avoid wasting resources on small tasks.

Syntax

Hadoop

mapreduce.job.reduces = number_of_reducers
mapreduce.map.memory.mb = memory_for_map_tasks
mapreduce.reduce.memory.mb = memory_for_reduce_tasks
mapreduce.task.io.sort.mb = memory_for_sorting
mapreduce.map.cpu.vcores = cpu_cores_for_map
mapreduce.reduce.cpu.vcores = cpu_cores_for_reduce

These parameters are set in the job configuration before running the job.

Adjusting these values changes how the job uses memory, CPU, and parallelism.

Examples

Set the number of reduce tasks to 10 to control parallel reduce work.

Hadoop

mapreduce.job.reduces = 10

Assign 2GB memory for each map task and 4GB for each reduce task.

Hadoop

mapreduce.map.memory.mb = 2048
mapreduce.reduce.memory.mb = 4096

Set 512MB memory for sorting data before sending to reducers.

Hadoop

mapreduce.task.io.sort.mb = 512

Sample Program

This example shows how to set tuning parameters in a MapReduce job using Pydoop. It sets 5 reducers, map tasks with 2GB memory, reduce tasks with 4GB memory, and 256MB for sorting.

Hadoop

from pydoop import hdfs
from pydoop.mapreduce.api import Mapper, Reducer
from pydoop.mapreduce.pipes import run_task

class MyMapper(Mapper):
    def map(self, context):
        for word in context.value.split():
            context.emit(word.lower(), 1)

class MyReducer(Reducer):
    def reduce(self, context):
        total = sum(context.values)
        context.emit(context.key, total)

if __name__ == '__main__':
    import pydoop.mapreduce.pipes as pipes
    import pydoop.mapreduce.api as api
    import pydoop.mapreduce.job as job

    conf = job.JobConf()
    conf.set('mapreduce.job.reduces', '5')
    conf.set('mapreduce.map.memory.mb', '2048')
    conf.set('mapreduce.reduce.memory.mb', '4096')
    conf.set('mapreduce.task.io.sort.mb', '256')

    run_task(pipes.Factory(MyMapper, MyReducer), job_conf=conf)

print('MapReduce job configured with 5 reducers, 2GB map memory, 4GB reduce memory, 256MB sort memory.')

Important Notes

Setting too many reducers can cause overhead and slow down the job.

Memory settings must match the cluster's available resources to avoid failures.

Sorting memory affects how fast data is shuffled between map and reduce tasks.

Summary

MapReduce tuning parameters control memory, CPU, and task numbers.

Proper tuning improves job speed and resource use.

Always test changes on small data before big runs.