Why YARN manages cluster resources in Hadoop - Visual Breakdown

Concept Flow - Why YARN manages cluster resources

User submits job

↓

YARN ResourceManager receives job

↓

ResourceManager checks cluster resources

↓

ResourceManager allocates resources

↓

NodeManagers launch containers

↓

Application runs using allocated resources

↓

ResourceManager monitors and manages resources

↓

Job completes and resources are freed

YARN manages cluster resources by receiving jobs, checking available resources, allocating them, and monitoring usage until job completion.

Execution Sample

Hadoop

1. Submit job to YARN
2. ResourceManager checks resources
3. Allocate containers on nodes
4. Run job in containers
5. Monitor and free resources

This sequence shows how YARN manages resources from job submission to completion.

Execution Table

Step	Action	ResourceManager State	NodeManager State	Result
1	Job submitted	Waiting for resource check	Idle	Job queued
2	Check cluster resources	Resources available: 10GB RAM, 8 cores	Idle	Ready to allocate
3	Allocate containers	Allocating 4GB RAM, 2 cores	Preparing containers	Containers allocated
4	Launch containers	Tracking allocated containers	Containers running job	Job started
5	Monitor job	Updating resource usage	Running containers	Job progressing
6	Job completes	Releasing resources	Containers stopped	Resources freed
7	Idle state	Resources available: 10GB RAM, 8 cores	Idle	Ready for next job

💡 Job completes and resources are freed for new jobs

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 6	Final
Available RAM	10GB	10GB	6GB	10GB	10GB
Available CPU cores	8	8	6	8	8
Job State	Not submitted	Queued	Running	Completed	Idle

Key Moments - 3 Insights

Why does ResourceManager check resources before allocating?

ResourceManager must ensure enough resources exist to run the job without overloading nodes, as shown in step 2 of the execution_table.

What happens if resources are not available?

The job stays queued until resources free up, so ResourceManager waits before allocating containers (not shown in this trace but implied in step 1 and 2).

Why does YARN monitor resource usage during job execution?

To track progress and free resources promptly after job completion, as seen in steps 5 and 6.

Visual Quiz - 3 Questions

Look at the execution_table, what is the ResourceManager state after allocating containers?

A Tracking allocated containers

B Allocating 4GB RAM, 2 cores

C Resources available: 10GB RAM, 8 cores

D Releasing resources

💡 Hint

Check the ResourceManager State column at step 4 in the execution_table

At which step does the job start running on NodeManagers?

A Step 3

B Step 5

C Step 4

D Step 6

💡 Hint

Look at the NodeManager State column to see when containers run the job

If the available RAM was only 2GB at step 2, what would happen?

A Job runs with less resources

B Job stays queued until enough RAM is free

C ResourceManager allocates containers anyway

D Job is canceled immediately

💡 Hint

ResourceManager checks resources before allocation as shown in step 2

Concept Snapshot

YARN manages cluster resources by:
- Receiving job requests
- Checking available resources
- Allocating containers on nodes
- Running jobs inside containers
- Monitoring and freeing resources after job completion

Full Transcript

YARN manages cluster resources by first receiving a job submission. The ResourceManager checks the cluster's available resources to ensure there is enough memory and CPU to run the job. It then allocates containers on NodeManagers, which launch the job. During execution, YARN monitors resource usage to track progress and manage resources efficiently. Once the job finishes, resources are freed for new jobs. This process helps keep the cluster balanced and jobs running smoothly.

Practice

1. Why does YARN manage cluster resources in a Hadoop environment?

easy

A. To replace the need for MapReduce

B. To store data permanently on the cluster

C. To share computer power fairly among all running jobs

D. To speed up network communication between nodes

Solution

1. Step 1: Understand YARN's role in resource management

   YARN is designed to allocate and manage resources like CPU and memory fairly across all jobs running on the cluster.

2. Step 2: Differentiate from other Hadoop components

   Unlike storage or network components, YARN focuses on resource sharing, not data storage or network speed.

3. Final Answer:

   To share computer power fairly among all running jobs -> Option C

4. Quick Check:

   YARN manages resources = fair sharing [OK]

Hint: YARN controls resources, not storage or network [OK]

Common Mistakes:

• Confusing YARN with HDFS storage
• Thinking YARN replaces MapReduce
• Assuming YARN speeds up network

2. Which of the following is the correct syntax to start the YARN ResourceManager service in Hadoop?

easy

A. start-mapreduce.sh

B. start-hdfs.sh

C. start-cluster.sh

D. start-yarn.sh

Solution

1. Step 1: Identify the script for YARN services

   The script start-yarn.sh is used to start YARN ResourceManager and NodeManager services.

2. Step 2: Differentiate from other Hadoop scripts

   start-hdfs.sh starts HDFS services, start-mapreduce.sh is not a standard script, and start-cluster.sh is not a valid command.

3. Final Answer:

   start-yarn.sh -> Option D

4. Quick Check:

   YARN start script = start-yarn.sh [OK]

Hint: YARN starts with start-yarn.sh script [OK]

Common Mistakes:

• Using start-hdfs.sh for YARN
• Confusing MapReduce with YARN start scripts
• Assuming start-cluster.sh exists

3. Consider this simplified YARN resource allocation code snippet in Python pseudocode:

resources = {'CPU': 16, 'Memory': 64}  # total cluster resources
jobs = [{'CPU': 4, 'Memory': 16}, {'CPU': 8, 'Memory': 32}, {'CPU': 6, 'Memory': 20}]
allocated = []
for job in jobs:
    if job['CPU'] <= resources['CPU'] and job['Memory'] <= resources['Memory']:
        allocated.append(job)
        resources['CPU'] -= job['CPU']
        resources['Memory'] -= job['Memory']
print(len(allocated))

What is the output of this code?

medium

A. 2

B. 0

C. 1

D. 3

Solution

1. Step 1: Check resource allocation for each job

   Start with CPU=16, Memory=64. Job1 needs CPU=4, Memory=16, fits, allocate and update resources to CPU=12, Memory=48. Job2 needs CPU=8, Memory=32, fits, allocate and update to CPU=4, Memory=16. Job3 needs CPU=6, Memory=20, does not fit CPU=4 < 6, so skip.

2. Step 2: Count allocated jobs

   Only Job1 and Job2 allocated, so length of allocated list is 2.

3. Final Answer:

   2 -> Option A

4. Quick Check:

   Allocated jobs = 2 [OK]

Hint: Subtract resources stepwise, check fit for each job [OK]

Common Mistakes:

• Assuming all jobs fit without checking resources
• Not updating resources after allocation
• Counting skipped jobs

4. The following YARN configuration snippet has an error:

yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
yarn.nodemanager.resource.memory-mb=8192
yarn.scheduler.maximum-allocation-mb=4096

What is the main problem with this configuration?

medium

A. Maximum allocation memory is less than node manager memory, causing job failures

B. Scheduler class name is misspelled

C. Node manager memory is too high for YARN

D. Missing configuration for CPU resources

Solution

1. Step 1: Compare node manager memory and max allocation

   Node manager memory is 8192 MB but max allocation is only 4096 MB, which means YARN cannot allocate more than 4096 MB per container, limiting resource usage.

2. Step 2: Understand impact on job execution

   This mismatch can cause jobs requesting more memory than max allocation to fail or be rejected.

3. Final Answer:

   Maximum allocation memory is less than node manager memory, causing job failures -> Option A

4. Quick Check:

   Max allocation < node manager memory = problem [OK]

Hint: Max allocation must be ≤ node manager memory [OK]

Common Mistakes:

• Ignoring memory mismatch
• Assuming scheduler class typo
• Overlooking missing CPU config

5. You have a Hadoop cluster with 100 CPU cores and 400 GB memory. YARN is configured to allocate resources fairly. If three jobs request resources as follows:

• Job A: 40 CPU cores, 150 GB memory
• Job B: 30 CPU cores, 100 GB memory
• Job C: 50 CPU cores, 200 GB memory

Which resource allocation strategy should YARN use to ensure all jobs get some resources without starving any job?

hard

A. Allocate full requested resources to jobs in order of submission until resources run out

B. Allocate resources proportionally based on requests, limiting each to available cluster capacity

C. Allocate resources only to the largest job to maximize throughput

D. Allocate resources randomly to jobs without checking requests

Solution

1. Step 1: Understand fair resource sharing in YARN

   YARN aims to share cluster resources fairly, so no job is starved and all get a portion of requested resources.

2. Step 2: Analyze allocation strategies

   Proportional allocation limits each job to a fair share based on total cluster capacity, unlike full allocation or random assignment which can cause starvation or inefficiency.

3. Final Answer:

   Allocate resources proportionally based on requests, limiting each to available cluster capacity -> Option B

4. Quick Check:

   Fair sharing = proportional allocation [OK]

Hint: Fair sharing means proportional resource allocation [OK]

Common Mistakes:

• Allocating full resources to first jobs only
• Ignoring fairness and starving jobs
• Random allocation without resource checks