YARN vs MapReduce v1 in Hadoop - Visual Side-by-Side Comparison

Concept Flow - YARN vs MapReduce v1

Job Submission

↓

MapReduce v1

↓

JobTracker manages resources & tasks

↓

TaskTrackers run map/reduce tasks

↓

Job completes

↓

Job Submission

↓

YARN Architecture

↓

ResourceManager manages cluster resources

↓

NodeManagers run containers

↓

ApplicationMaster manages job tasks

↓

Job completes

Shows the flow of job execution in MapReduce v1 with JobTracker and TaskTrackers versus YARN with ResourceManager, NodeManagers, and ApplicationMaster.

Execution Sample

Hadoop

# Pseudocode for job flow
submit_job()
if system == 'MapReduce v1':
  JobTracker.assign_tasks()
  TaskTrackers.run_tasks()
else:
  ResourceManager.allocate_resources()
  ApplicationMaster.manage_tasks()
  NodeManagers.run_containers()

This pseudocode shows how job submission differs between MapReduce v1 and YARN.

Execution Table

Step	Component	Action	Result
1	User	Submit job	Job request sent
2	MapReduce v1: JobTracker	Receive job	Schedules tasks and manages cluster
3	MapReduce v1: TaskTrackers	Run map/reduce tasks	Execute tasks and report status
4	MapReduce v1: JobTracker	Monitor tasks	Track progress and handle failures
5	MapReduce v1	Job completes	Output data ready
6	User	Submit job	Job request sent
7	YARN: ResourceManager	Receive job	Allocates cluster resources
8	YARN: ApplicationMaster	Manage job tasks	Coordinates task execution
9	YARN: NodeManagers	Run containers	Execute tasks in containers
10	YARN: ApplicationMaster	Monitor tasks	Track progress and handle failures
11	YARN	Job completes	Output data ready

💡 Job completes after all tasks finish successfully in both systems

Variable Tracker

Component	Initial State	After Job Submission	During Execution	Final State
JobTracker	Idle	Receives job	Schedules and monitors tasks	Idle
TaskTrackers	Idle	Waiting for tasks	Running map/reduce tasks	Idle
ResourceManager	Idle	Receives job	Allocates resources	Idle
ApplicationMaster	Not running	Starts for job	Manages tasks	Stops after job
NodeManagers	Idle	Ready	Run containers	Idle

Key Moments - 3 Insights

Why does MapReduce v1 have a single JobTracker managing everything?

In MapReduce v1, the JobTracker is the central point for resource management and task scheduling, which can cause bottlenecks as shown in steps 2-4 of the execution_table.

How does YARN improve resource management compared to MapReduce v1?

YARN separates resource management (ResourceManager) from job management (ApplicationMaster), distributing responsibilities and improving scalability, as seen in steps 7-10.

What role do NodeManagers play in YARN that TaskTrackers played in MapReduce v1?

NodeManagers in YARN run containers to execute tasks, similar to TaskTrackers running map/reduce tasks, but with more flexibility and better resource isolation (steps 9 vs 3).

Visual Quiz - 3 Questions

Look at the execution_table, which component in MapReduce v1 schedules tasks?

A JobTracker

B ResourceManager

C TaskTrackers

D ApplicationMaster

💡 Hint

Check step 2 in the execution_table where scheduling is mentioned.

At which step does YARN allocate cluster resources?

A Step 9

B Step 8

C Step 7

D Step 10

💡 Hint

Look at the action 'Allocates cluster resources' in the execution_table.

If the ApplicationMaster fails during execution, which step in YARN is affected?

A Step 7

B Step 10

C Step 9

D Step 8

💡 Hint

Refer to the monitoring and failure handling step for ApplicationMaster in the execution_table.

Concept Snapshot

YARN vs MapReduce v1:
- MapReduce v1 uses JobTracker (single manager) and TaskTrackers (workers).
- YARN splits roles: ResourceManager (resources), ApplicationMaster (job), NodeManagers (containers).
- YARN improves scalability and resource use.
- JobTracker is a bottleneck in v1; YARN distributes control.
- Both systems run map and reduce tasks but differ in architecture.

Full Transcript

This visual execution compares YARN and MapReduce v1 job flows. In MapReduce v1, the JobTracker manages both resources and tasks, while TaskTrackers run the actual map and reduce tasks. This centralization can cause bottlenecks. YARN improves this by splitting responsibilities: the ResourceManager handles cluster resources, the ApplicationMaster manages job tasks, and NodeManagers run containers to execute tasks. The execution table shows each step from job submission to completion in both systems. Variable tracking highlights how components change state during execution. Key moments clarify common confusions about the roles of JobTracker and ApplicationMaster. The quiz tests understanding of which components perform key actions at specific steps. Overall, YARN offers better scalability and resource management than MapReduce v1.

Practice

1. What is the main difference between YARN and MapReduce v1 in Hadoop?

easy

A. MapReduce v1 supports multiple application types, YARN supports only MapReduce.

B. YARN separates resource management and job scheduling, while MapReduce v1 combines them.

C. YARN uses JobTracker for all tasks, MapReduce v1 uses ResourceManager.

D. MapReduce v1 is more scalable than YARN.

Solution

1. Step 1: Understand MapReduce v1 architecture

   MapReduce v1 uses a single JobTracker to manage both resources and job scheduling.

2. Step 2: Understand YARN architecture

   YARN splits these roles: ResourceManager handles resources, ApplicationMaster handles job scheduling, improving scalability.

3. Final Answer:

   YARN separates resource management and job scheduling, while MapReduce v1 combines them. -> Option B

4. Quick Check:

   YARN splits roles = A [OK]

Hint: Remember: YARN splits roles, MapReduce v1 combines [OK]

Common Mistakes:

• Confusing JobTracker with ResourceManager
• Thinking MapReduce v1 supports multiple app types
• Assuming YARN uses JobTracker

2. Which of the following is the correct component responsible for resource management in YARN?

easy

A. ResourceManager

B. JobTracker

C. TaskTracker

D. ApplicationMaster

Solution

1. Step 1: Identify resource management in YARN

   YARN uses ResourceManager to manage cluster resources.

2. Step 2: Differentiate other components

   JobTracker and TaskTracker belong to MapReduce v1; ApplicationMaster manages job lifecycle, not resources.

3. Final Answer:

   ResourceManager -> Option A

4. Quick Check:

   ResourceManager manages resources = B [OK]

Hint: ResourceManager manages resources in YARN [OK]

Common Mistakes:

• Choosing JobTracker which is from MapReduce v1
• Confusing ApplicationMaster with resource manager
• Selecting TaskTracker which handles tasks, not resources

3. Consider this code snippet representing YARN components interaction:

class YARN:
    def __init__(self):
        self.rm = 'ResourceManager'
        self.am = 'ApplicationMaster'

    def get_components(self):
        return [self.rm, self.am]

yarn = YARN()
print(yarn.get_components())

What will be the output?

medium

A. ['ResourceManager', 'ApplicationMaster']

B. Error: AttributeError

C. ['ResourceManager']

D. ['JobTracker', 'TaskTracker']

Solution

1. Step 1: Analyze class initialization

   The YARN class initializes two attributes: 'ResourceManager' and 'ApplicationMaster'.

2. Step 2: Check method output

   The get_components method returns a list containing both attributes.

3. Final Answer:

   ['ResourceManager', 'ApplicationMaster'] -> Option A

4. Quick Check:

   List with RM and AM = D [OK]

Hint: Check returned list matches initialized attributes [OK]

Common Mistakes:

• Confusing with MapReduce v1 components
• Expecting only one component returned
• Assuming code throws error

4. The following code tries to simulate MapReduce v1 job submission but has an error:

class MapReduceV1:
    def __init__(self):
        self.jobtracker = 'JobTracker'

    def submit_job(self):
        print(self.tasktracker)

What is the error and how to fix it?

medium

A. TypeError because print expects a string argument.

B. SyntaxError due to missing colon after method definition.

C. AttributeError because 'tasktracker' is not defined; define self.tasktracker in __init__.

D. No error; code runs fine.

Solution

1. Step 1: Identify the error cause

   The method submit_job tries to print self.tasktracker, but this attribute is not defined in __init__.

2. Step 2: Fix the error

   Define self.tasktracker in __init__, e.g., self.tasktracker = 'TaskTracker', to avoid AttributeError.

3. Final Answer:

   AttributeError because 'tasktracker' is not defined; define self.tasktracker in __init__. -> Option C

4. Quick Check:

   Undefined attribute causes AttributeError = C [OK]

Hint: Check all used attributes are defined in __init__ [OK]

Common Mistakes:

• Thinking it's a syntax error
• Assuming print causes TypeError
• Ignoring missing attribute definition

5. You want to run a new type of application on Hadoop that is not MapReduce. Which feature of YARN makes this possible compared to MapReduce v1?

hard

A. MapReduce v1 separates resource management and scheduling.

B. MapReduce v1 supports multiple application types by default.

C. YARN uses JobTracker to schedule all applications.

D. YARN's ApplicationMaster can manage different application types beyond MapReduce.

Solution

1. Step 1: Understand application support in MapReduce v1

   MapReduce v1 is limited to MapReduce jobs only, managed by JobTracker.

2. Step 2: Understand YARN's flexibility

   YARN's ApplicationMaster can be customized to manage various application types, enabling flexibility.

3. Final Answer:

   YARN's ApplicationMaster can manage different application types beyond MapReduce. -> Option D

4. Quick Check:

   YARN supports multiple app types = A [OK]

Hint: ApplicationMaster enables multi-app support in YARN [OK]

Common Mistakes:

• Thinking MapReduce v1 supports multiple apps
• Confusing JobTracker role in YARN
• Assuming MapReduce v1 separates resource management