Application lifecycle in YARN in Hadoop - Step-by-Step Execution

Concept Flow - Application lifecycle in YARN

Client submits application

↓

ResourceManager receives request

↓

ResourceManager allocates ApplicationMaster container

↓

ApplicationMaster starts and registers with ResourceManager

↓

ApplicationMaster requests resources for tasks

↓

ResourceManager allocates containers

↓

ApplicationMaster launches tasks in containers

↓

Tasks run and report progress

↓

ApplicationMaster monitors tasks

↓

On completion, ApplicationMaster unregisters

↓

ResourceManager cleans up application

↓

Application ends

This flow shows how a YARN application moves from submission to completion through ResourceManager and ApplicationMaster coordination.

Execution Sample

Hadoop

1. Client submits application to ResourceManager
2. ResourceManager allocates ApplicationMaster container
3. ApplicationMaster registers and requests task containers
4. ResourceManager allocates containers
5. ApplicationMaster launches tasks
6. Tasks run and report progress
7. ApplicationMaster unregisters on completion

This sequence shows the main steps in the YARN application lifecycle from submission to completion.

Execution Table

Step	Component	Action	Result	Next Step
1	Client	Submit application	Application request sent to ResourceManager	ResourceManager receives request
2	ResourceManager	Allocate ApplicationMaster container	Container allocated for ApplicationMaster	ApplicationMaster starts
3	ApplicationMaster	Register with ResourceManager	Registration successful	Request task containers
4	ApplicationMaster	Request containers for tasks	ResourceManager receives requests	ResourceManager allocates containers
5	ResourceManager	Allocate containers for tasks	Containers allocated	ApplicationMaster launches tasks
6	ApplicationMaster	Launch tasks in containers	Tasks started	Tasks run and report progress
7	Tasks	Run and report progress	Progress updates sent	ApplicationMaster monitors tasks
8	ApplicationMaster	Monitor tasks	Tasks complete	Unregister from ResourceManager
9	ApplicationMaster	Unregister from ResourceManager	Unregistration successful	ResourceManager cleans up
10	ResourceManager	Clean up application	Resources freed	Application ends
11	-	Application ends	Application lifecycle complete	-

💡 Application ends after ResourceManager cleans up resources and ApplicationMaster unregisters.

Variable Tracker

Variable	Start	After Step 2	After Step 5	After Step 8	Final
Application State	Not submitted	AM container allocated	Tasks launched	Tasks completed	Application ended
ResourceManager State	Idle	AM container allocated	Task containers allocated	Monitoring tasks	Cleaning up
ApplicationMaster State	Not started	Started	Launching tasks	Monitoring tasks	Unregistered

Key Moments - 3 Insights

Why does the ApplicationMaster need to register with the ResourceManager?

The ApplicationMaster registers to inform the ResourceManager it is ready to manage the application. This is shown in step 3 of the execution_table where registration enables resource requests.

What happens if the ResourceManager does not allocate containers for tasks?

Without container allocation (step 5), the ApplicationMaster cannot launch tasks, so the application cannot proceed. This blocks progress until containers are allocated.

Why does the ApplicationMaster unregister at the end?

Unregistering (step 9) signals the ResourceManager that the application is complete and resources can be freed, as shown in the cleanup step 10.

Visual Quiz - 3 Questions

Look at the execution_table, what is the ApplicationMaster state after step 3?

A Started and registered

B Not started

C Unregistered

D Tasks launched

💡 Hint

Check the execution_table rows for steps 2 and 3.

At which step does the ResourceManager allocate containers for tasks?

A Step 4

B Step 5

C Step 2

D Step 7

💡 Hint

Look at the execution_table row where ResourceManager allocates containers.

If the ApplicationMaster does not unregister, what is the likely impact?

A ResourceManager cleans up resources anyway

B Tasks restart automatically

C Application remains active and resources are not freed

D Client resubmits application

💡 Hint

Refer to key_moments explanation about unregistering and resource cleanup.

Concept Snapshot

YARN Application Lifecycle:
1. Client submits application to ResourceManager.
2. ResourceManager allocates ApplicationMaster container.
3. ApplicationMaster registers and requests task containers.
4. ResourceManager allocates containers.
5. ApplicationMaster launches tasks.
6. Tasks run and report progress.
7. ApplicationMaster unregisters and ResourceManager cleans up.
Key: ApplicationMaster manages tasks and communicates with ResourceManager.

Full Transcript

The YARN application lifecycle starts when the client submits an application to the ResourceManager. The ResourceManager allocates a container for the ApplicationMaster, which then registers itself with the ResourceManager. The ApplicationMaster requests containers for running tasks. The ResourceManager allocates these containers, and the ApplicationMaster launches the tasks inside them. Tasks run and report progress back to the ApplicationMaster, which monitors them. When all tasks complete, the ApplicationMaster unregisters from the ResourceManager. Finally, the ResourceManager cleans up resources, ending the application lifecycle.

Practice

1. Which component in YARN is responsible for managing the overall lifecycle of an application from submission to completion?

easy

A. ApplicationMaster

B. Container

C. ResourceManager

D. NodeManager

Solution

1. Step 1: Understand YARN components roles

   The ResourceManager is the master that manages resources and schedules applications.

2. Step 2: Identify lifecycle management responsibility

   ResourceManager coordinates application submission, resource allocation, and monitors progress.

3. Final Answer:

   ResourceManager -> Option C

4. Quick Check:

   ResourceManager = lifecycle manager [OK]

Hint: ResourceManager controls app lifecycle, not NodeManager or containers [OK]

Common Mistakes:

• Confusing ApplicationMaster with ResourceManager
• Thinking NodeManager manages entire lifecycle
• Assuming containers manage lifecycle

2. Which of the following is the correct sequence of steps in the YARN application lifecycle?

easy

A. Container launch -> Application submission -> Resource allocation -> Application completion

B. Application submission -> Resource allocation -> Container launch -> Application completion

C. Resource allocation -> Application submission -> Container launch -> Application completion

D. Application completion -> Container launch -> Resource allocation -> Application submission

Solution

1. Step 1: Recall YARN lifecycle order

   First, the application is submitted, then resources are allocated.

2. Step 2: Follow container launch and completion

   After allocation, containers launch tasks, then application completes.

3. Final Answer:

   Application submission -> Resource allocation -> Container launch -> Application completion -> Option B

4. Quick Check:

   Submission -> Allocation -> Launch -> Completion [OK]

Hint: Lifecycle starts with submission, ends with completion [OK]

Common Mistakes:

• Mixing order of resource allocation and submission
• Thinking container launch happens before submission
• Reversing completion step

3. Consider this simplified YARN application lifecycle code snippet in pseudocode:

submitApplication()
allocateResources()
launchContainers()
monitorApplication()
cleanup()

Which step is responsible for requesting containers from NodeManagers?

medium

A. submitApplication()

B. launchContainers()

C. cleanup()

D. allocateResources()

Solution

1. Step 1: Understand resource allocation role

   allocateResources() requests containers from NodeManagers via ResourceManager.

2. Step 2: Differentiate from other steps

   submitApplication() submits job, launchContainers() starts tasks, cleanup() frees resources.

3. Final Answer:

   allocateResources() -> Option D

4. Quick Check:

   Resource allocation = container requests [OK]

Hint: Resource allocation step requests containers from NodeManagers [OK]

Common Mistakes:

• Confusing launchContainers() with resource requests
• Thinking submitApplication() requests containers
• Assuming cleanup() handles resource allocation

4. You see a YARN application stuck in the ACCEPTED state for a long time. What is the most likely cause?

medium

A. ResourceManager has no available resources to allocate

B. ApplicationMaster failed to launch

C. NodeManager crashed after container launch

D. Application has already completed

Solution

1. Step 1: Understand ACCEPTED state meaning

   ACCEPTED means application is waiting for resources to be allocated.

2. Step 2: Identify cause of delay in allocation

   If ResourceManager has no free resources, allocation is delayed, causing ACCEPTED state to persist.

3. Final Answer:

   ResourceManager has no available resources to allocate -> Option A

4. Quick Check:

   ACCEPTED = waiting for resources [OK]

Hint: Long ACCEPTED means no resources available [OK]

Common Mistakes:

• Assuming ApplicationMaster failed immediately
• Thinking NodeManager crash causes ACCEPTED state
• Confusing ACCEPTED with COMPLETED state

5. You want to optimize a YARN application that spends too much time in the RUNNING state but finishes slowly. Which approach best improves the application lifecycle?

hard

A. Increase container memory and CPU to speed up task execution

B. Reduce the number of containers to limit resource usage

C. Submit the application multiple times in parallel

D. Disable ApplicationMaster monitoring to reduce overhead

Solution

1. Step 1: Analyze RUNNING state delay cause

   Long RUNNING time means tasks are slow or resource-starved.

2. Step 2: Choose optimization method

   Increasing container memory and CPU helps tasks run faster, reducing RUNNING time.

3. Step 3: Evaluate other options

   Reducing containers slows progress, parallel submissions cause contention, disabling monitoring risks errors.

4. Final Answer:

   Increase container memory and CPU to speed up task execution -> Option A

5. Quick Check:

   More resources = faster RUNNING tasks [OK]

Hint: Boost container resources to speed RUNNING tasks [OK]

Common Mistakes:

• Reducing containers slows application
• Submitting multiple apps causes resource conflicts
• Disabling monitoring hides problems