Why YARN manages cluster resources in Hadoop - Visual Breakdown

Concept Flow - Why YARN manages cluster resources

User submits job

↓

YARN ResourceManager receives job

↓

ResourceManager checks cluster resources

↓

ResourceManager allocates resources

↓

NodeManagers launch containers

↓

Application runs using allocated resources

↓

ResourceManager monitors and manages resources

↓

Job completes and resources are freed

YARN manages cluster resources by receiving jobs, checking available resources, allocating them, and monitoring usage until job completion.

Execution Sample

Hadoop

1. Submit job to YARN
2. ResourceManager checks resources
3. Allocate containers on nodes
4. Run job in containers
5. Monitor and free resources

This sequence shows how YARN manages resources from job submission to completion.

Execution Table

Step	Action	ResourceManager State	NodeManager State	Result
1	Job submitted	Waiting for resource check	Idle	Job queued
2	Check cluster resources	Resources available: 10GB RAM, 8 cores	Idle	Ready to allocate
3	Allocate containers	Allocating 4GB RAM, 2 cores	Preparing containers	Containers allocated
4	Launch containers	Tracking allocated containers	Containers running job	Job started
5	Monitor job	Updating resource usage	Running containers	Job progressing
6	Job completes	Releasing resources	Containers stopped	Resources freed
7	Idle state	Resources available: 10GB RAM, 8 cores	Idle	Ready for next job

💡 Job completes and resources are freed for new jobs

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 6	Final
Available RAM	10GB	10GB	6GB	10GB	10GB
Available CPU cores	8	8	6	8	8
Job State	Not submitted	Queued	Running	Completed	Idle

Key Moments - 3 Insights

Why does ResourceManager check resources before allocating?

ResourceManager must ensure enough resources exist to run the job without overloading nodes, as shown in step 2 of the execution_table.

What happens if resources are not available?

The job stays queued until resources free up, so ResourceManager waits before allocating containers (not shown in this trace but implied in step 1 and 2).

Why does YARN monitor resource usage during job execution?

To track progress and free resources promptly after job completion, as seen in steps 5 and 6.

Visual Quiz - 3 Questions

Look at the execution_table, what is the ResourceManager state after allocating containers?

A Tracking allocated containers

B Allocating 4GB RAM, 2 cores

C Resources available: 10GB RAM, 8 cores

D Releasing resources

💡 Hint

Check the ResourceManager State column at step 4 in the execution_table

At which step does the job start running on NodeManagers?

A Step 3

B Step 5

C Step 4

D Step 6

💡 Hint

Look at the NodeManager State column to see when containers run the job

If the available RAM was only 2GB at step 2, what would happen?

A Job runs with less resources

B Job stays queued until enough RAM is free

C ResourceManager allocates containers anyway

D Job is canceled immediately

💡 Hint

ResourceManager checks resources before allocation as shown in step 2

Concept Snapshot

YARN manages cluster resources by:
- Receiving job requests
- Checking available resources
- Allocating containers on nodes
- Running jobs inside containers
- Monitoring and freeing resources after job completion

Full Transcript

YARN manages cluster resources by first receiving a job submission. The ResourceManager checks the cluster's available resources to ensure there is enough memory and CPU to run the job. It then allocates containers on NodeManagers, which launch the job. During execution, YARN monitors resource usage to track progress and manage resources efficiently. Once the job finishes, resources are freed for new jobs. This process helps keep the cluster balanced and jobs running smoothly.