Memory and container sizing in Hadoop - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a container in Hadoop YARN?

A container is a resource allocation unit in Hadoop YARN that bundles CPU, memory, and other resources to run a task.

beginner

Why is memory sizing important for containers in Hadoop?

Proper memory sizing ensures tasks run efficiently without crashing or wasting resources, balancing performance and cluster utilization.

intermediate

What happens if a container is assigned less memory than needed?

The container may fail or the task may crash due to out-of-memory errors, causing job failures or retries.

intermediate

How does YARN decide container size?

YARN uses configuration settings like yarn.scheduler.minimum-allocation-mb and yarn.scheduler.maximum-allocation-mb to control container sizes.

intermediate

What is the impact of setting container memory too high?

It can lead to inefficient resource use, fewer containers running simultaneously, and lower cluster throughput.

What does a container in Hadoop YARN include?

A CPU and memory resources

B Only CPU resources

C Only memory resources

D Disk space only

Containers bundle CPU, memory, and sometimes other resources to run tasks.

Which configuration controls the minimum container memory in YARN?

A yarn.nodemanager.resource.memory-mb

B yarn.scheduler.minimum-allocation-mb

C mapreduce.task.io.sort.mb

D yarn.resourcemanager.scheduler.class

yarn.scheduler.minimum-allocation-mb sets the smallest container size allowed.

What is a risk of assigning too little memory to a container?

A Running too many containers

B Wasting cluster resources

C Task failure due to out-of-memory

D Improved performance

Insufficient memory causes tasks to crash or fail.

If container memory is set too high, what happens?

A No effect on cluster

B More containers run at once

C Tasks run faster always

D Fewer containers run at once

High memory per container reduces how many containers fit on nodes.

Which is NOT a resource managed by YARN containers?

A Network bandwidth

B Memory

C CPU

D Disk space

YARN containers manage CPU, memory, and disk but not network bandwidth directly.

Explain why correct memory and container sizing is important in Hadoop YARN.

Think about what happens if memory is too low or too high.

Describe how YARN uses configuration to control container sizes.

Focus on yarn.scheduler.minimum-allocation-mb and yarn.scheduler.maximum-allocation-mb.