YARN scheduling policies in Hadoop - Step-by-Step Execution

Concept Flow - YARN scheduling policies

Job arrives in YARN

↓

Scheduler selects policy

↓

FIFO: Jobs run in order

↓

Capacity: Resources split by queues

↓

Fair: Resources shared fairly

↓

Allocate resources to containers

↓

Run tasks

↓

Job completes or waits

YARN receives jobs and chooses a scheduling policy (FIFO, Capacity, or Fair) to allocate resources and run tasks.

Execution Sample

Hadoop

jobs = [Job1, Job2, Job3]
scheduler = 'FIFO'
for job in jobs:
    allocate_resources(job)
    run(job)

This code runs jobs one by one in the order they arrive using FIFO scheduling.

Execution Table

Step	Job	Scheduler Policy	Action	Resources Allocated	Status
1	Job1	FIFO	Allocate resources	4 containers	Running
2	Job1	FIFO	Run tasks	4 containers	Running
3	Job1	FIFO	Complete	0 containers	Finished
4	Job2	FIFO	Allocate resources	3 containers	Running
5	Job2	FIFO	Run tasks	3 containers	Running
6	Job2	FIFO	Complete	0 containers	Finished
7	Job3	FIFO	Allocate resources	2 containers	Running
8	Job3	FIFO	Run tasks	2 containers	Running
9	Job3	FIFO	Complete	0 containers	Finished

💡 All jobs completed in FIFO order, no more jobs to schedule.

Variable Tracker

Variable	Start	After Step 1	After Step 4	After Step 7	Final
Job1 Status	Waiting	Running	Finished	Finished	Finished
Job2 Status	Waiting	Waiting	Running	Finished	Finished
Job3 Status	Waiting	Waiting	Waiting	Running	Finished
Resources Allocated	0	4	3	2	0

Key Moments - 2 Insights

Why does Job2 wait until Job1 finishes in FIFO scheduling?

Because FIFO runs jobs strictly in arrival order, Job2 cannot start until Job1 completes, as shown in steps 1-6 in the execution_table.

How does resource allocation differ between Capacity and Fair schedulers?

Capacity divides resources by fixed queues, while Fair shares resources dynamically to balance usage. This differs from FIFO's one-job-at-a-time approach.

Visual Quiz - 3 Questions

Look at the execution_table, what is the status of Job2 at Step 3?

A Waiting

B Running

C Finished

D Not started

💡 Hint

Check the 'Status' column for Job2 at Step 3 in the execution_table.

At which step does Job3 start running in FIFO scheduling?

A Step 4

B Step 7

C Step 6

D Step 9

💡 Hint

Look at the 'Action' and 'Status' columns for Job3 in the execution_table.

If we switch to Fair scheduler, how would resource allocation change?

A Jobs run one by one with full resources

B Only the first job gets resources

C Resources are shared among jobs simultaneously

D Jobs run in reverse order

💡 Hint

Fair scheduler shares resources fairly among running jobs, unlike FIFO.

Concept Snapshot

YARN Scheduling Policies:
- FIFO: Jobs run one after another in arrival order.
- Capacity: Resources split by queues with fixed capacity.
- Fair: Resources shared dynamically for fairness.
- Scheduler allocates containers to run tasks.
- Choice affects job wait and resource use.

Full Transcript

YARN scheduling policies decide how jobs get resources to run. FIFO runs jobs one by one in order. Capacity scheduler splits resources by queues with set limits. Fair scheduler shares resources evenly among jobs. This example shows FIFO running three jobs in order, allocating containers, running tasks, and finishing each job before starting the next. Variables track job status and resource use. Key points include why jobs wait in FIFO and how resource sharing differs in other policies. Quizzes check understanding of job status at steps and scheduler behavior.