Application lifecycle in YARN in Hadoop - Step-by-Step Execution

Concept Flow - Application lifecycle in YARN

Client submits application

↓

ResourceManager receives request

↓

ResourceManager allocates ApplicationMaster container

↓

ApplicationMaster starts and registers with ResourceManager

↓

ApplicationMaster requests resources for tasks

↓

ResourceManager allocates containers

↓

ApplicationMaster launches tasks in containers

↓

Tasks run and report progress

↓

ApplicationMaster monitors tasks

↓

On completion, ApplicationMaster unregisters

↓

ResourceManager cleans up application

↓

Application ends

This flow shows how a YARN application moves from submission to completion through ResourceManager and ApplicationMaster coordination.

Execution Sample

Hadoop

1. Client submits application to ResourceManager
2. ResourceManager allocates ApplicationMaster container
3. ApplicationMaster registers and requests task containers
4. ResourceManager allocates containers
5. ApplicationMaster launches tasks
6. Tasks run and report progress
7. ApplicationMaster unregisters on completion

This sequence shows the main steps in the YARN application lifecycle from submission to completion.

Execution Table

Step	Component	Action	Result	Next Step
1	Client	Submit application	Application request sent to ResourceManager	ResourceManager receives request
2	ResourceManager	Allocate ApplicationMaster container	Container allocated for ApplicationMaster	ApplicationMaster starts
3	ApplicationMaster	Register with ResourceManager	Registration successful	Request task containers
4	ApplicationMaster	Request containers for tasks	ResourceManager receives requests	ResourceManager allocates containers
5	ResourceManager	Allocate containers for tasks	Containers allocated	ApplicationMaster launches tasks
6	ApplicationMaster	Launch tasks in containers	Tasks started	Tasks run and report progress
7	Tasks	Run and report progress	Progress updates sent	ApplicationMaster monitors tasks
8	ApplicationMaster	Monitor tasks	Tasks complete	Unregister from ResourceManager
9	ApplicationMaster	Unregister from ResourceManager	Unregistration successful	ResourceManager cleans up
10	ResourceManager	Clean up application	Resources freed	Application ends
11	-	Application ends	Application lifecycle complete	-

💡 Application ends after ResourceManager cleans up resources and ApplicationMaster unregisters.

Variable Tracker

Variable	Start	After Step 2	After Step 5	After Step 8	Final
Application State	Not submitted	AM container allocated	Tasks launched	Tasks completed	Application ended
ResourceManager State	Idle	AM container allocated	Task containers allocated	Monitoring tasks	Cleaning up
ApplicationMaster State	Not started	Started	Launching tasks	Monitoring tasks	Unregistered

Key Moments - 3 Insights

Why does the ApplicationMaster need to register with the ResourceManager?

The ApplicationMaster registers to inform the ResourceManager it is ready to manage the application. This is shown in step 3 of the execution_table where registration enables resource requests.

What happens if the ResourceManager does not allocate containers for tasks?

Without container allocation (step 5), the ApplicationMaster cannot launch tasks, so the application cannot proceed. This blocks progress until containers are allocated.

Why does the ApplicationMaster unregister at the end?

Unregistering (step 9) signals the ResourceManager that the application is complete and resources can be freed, as shown in the cleanup step 10.

Visual Quiz - 3 Questions

Look at the execution_table, what is the ApplicationMaster state after step 3?

A Started and registered

B Not started

C Unregistered

D Tasks launched

💡 Hint

Check the execution_table rows for steps 2 and 3.

At which step does the ResourceManager allocate containers for tasks?

A Step 4

B Step 5

C Step 2

D Step 7

💡 Hint

Look at the execution_table row where ResourceManager allocates containers.

If the ApplicationMaster does not unregister, what is the likely impact?

A ResourceManager cleans up resources anyway

B Tasks restart automatically

C Application remains active and resources are not freed

D Client resubmits application

💡 Hint

Refer to key_moments explanation about unregistering and resource cleanup.

Concept Snapshot

YARN Application Lifecycle:
1. Client submits application to ResourceManager.
2. ResourceManager allocates ApplicationMaster container.
3. ApplicationMaster registers and requests task containers.
4. ResourceManager allocates containers.
5. ApplicationMaster launches tasks.
6. Tasks run and report progress.
7. ApplicationMaster unregisters and ResourceManager cleans up.
Key: ApplicationMaster manages tasks and communicates with ResourceManager.

Full Transcript

The YARN application lifecycle starts when the client submits an application to the ResourceManager. The ResourceManager allocates a container for the ApplicationMaster, which then registers itself with the ResourceManager. The ApplicationMaster requests containers for running tasks. The ResourceManager allocates these containers, and the ApplicationMaster launches the tasks inside them. Tasks run and report progress back to the ApplicationMaster, which monitors them. When all tasks complete, the ApplicationMaster unregisters from the ResourceManager. Finally, the ResourceManager cleans up resources, ending the application lifecycle.