Application lifecycle in YARN in Hadoop - Mini Project: Build & Apply

Understanding Application Lifecycle in YARN

📖 Scenario: You are working with Hadoop YARN to manage distributed applications. Understanding how an application moves through its lifecycle in YARN helps you monitor and troubleshoot your big data jobs effectively.

🎯 Goal: Build a simple Python dictionary that models the stages of an application lifecycle in YARN, then filter and display the stages that represent active states.

📋 What You'll Learn

Create a dictionary with exact application lifecycle stages and their descriptions

Add a list variable to hold active states

Use a dictionary comprehension to filter only active states from the lifecycle dictionary

Print the filtered dictionary showing active lifecycle stages

💡 Why This Matters

🌍 Real World

Understanding the application lifecycle in YARN helps data engineers monitor and manage big data jobs efficiently, ensuring resources are used well and failures are caught early.

💼 Career

Knowledge of YARN application states is essential for roles like Hadoop administrator, data engineer, and big data developer to troubleshoot and optimize distributed applications.

1

Create the YARN application lifecycle dictionary

Create a dictionary called app_lifecycle with these exact entries: 'NEW': 'Application is created', 'SUBMITTED': 'Application is submitted to the ResourceManager', 'ACCEPTED': 'Application is accepted by the ResourceManager', 'RUNNING': 'Application is running', 'FINISHED': 'Application has finished successfully', 'FAILED': 'Application has failed', 'KILLED': 'Application was killed'.

Hadoop

# Your code here

Need a hint?

Use curly braces {} to create the dictionary with the exact keys and values.

2

Define active states list

Create a list called active_states containing these exact strings: 'NEW', 'SUBMITTED', 'ACCEPTED', 'RUNNING'.

Hadoop

app_lifecycle = {
    'NEW': 'Application is created',
    'SUBMITTED': 'Application is submitted to the ResourceManager',
    'ACCEPTED': 'Application is accepted by the ResourceManager',
    'RUNNING': 'Application is running',
    'FINISHED': 'Application has finished successfully',
    'FAILED': 'Application has failed',
    'KILLED': 'Application was killed'
}
# Your code here

Need a hint?

Use square brackets [] to create the list with the exact strings.

3

Filter active lifecycle stages

Use a dictionary comprehension to create a new dictionary called active_lifecycle that includes only the entries from app_lifecycle where the key is in the active_states list.

Hadoop

app_lifecycle = {
    'NEW': 'Application is created',
    'SUBMITTED': 'Application is submitted to the ResourceManager',
    'ACCEPTED': 'Application is accepted by the ResourceManager',
    'RUNNING': 'Application is running',
    'FINISHED': 'Application has finished successfully',
    'FAILED': 'Application has failed',
    'KILLED': 'Application was killed'
}

active_states = ['NEW', 'SUBMITTED', 'ACCEPTED', 'RUNNING']
# Your code here

Need a hint?

Use {key: value for key, value in dict.items() if key in list} syntax for dictionary comprehension.

4

Display the active lifecycle stages

Write a print statement to display the active_lifecycle dictionary.

Hadoop

app_lifecycle = {
    'NEW': 'Application is created',
    'SUBMITTED': 'Application is submitted to the ResourceManager',
    'ACCEPTED': 'Application is accepted by the ResourceManager',
    'RUNNING': 'Application is running',
    'FINISHED': 'Application has finished successfully',
    'FAILED': 'Application has failed',
    'KILLED': 'Application was killed'
}

active_states = ['NEW', 'SUBMITTED', 'ACCEPTED', 'RUNNING']

active_lifecycle = {state: desc for state, desc in app_lifecycle.items() if state in active_states}
# Your code here

Need a hint?

Use print(active_lifecycle) to show the filtered dictionary.