YARN scheduling policies in Hadoop - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to set the YARN scheduler to Capacity Scheduler.

Hadoop

yarn.scheduler.class=[1]

Drag options to blanks, or click blank then click option'

Aorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

Borg.apache.hadoop.yarn.server.resourcemanager.scheduler.RoundRobinScheduler

Corg.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

Dorg.apache.hadoop.yarn.server.resourcemanager.scheduler.PriorityScheduler

Attempts:

3 left

💡 Hint

Common Mistakes

Choosing FairScheduler instead of CapacityScheduler

Using a non-existent scheduler class

Misspelling the class name

The Capacity Scheduler class is org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler. This sets YARN to use Capacity Scheduler.

2fill in blank

medium

Complete the code to configure the Fair Scheduler policy in YARN.

Hadoop

yarn.scheduler.class=[1]

Drag options to blanks, or click blank then click option'

Aorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

Borg.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

Corg.apache.hadoop.yarn.server.resourcemanager.scheduler.PriorityScheduler

Dorg.apache.hadoop.yarn.server.resourcemanager.scheduler.RoundRobinScheduler

Attempts:

3 left

💡 Hint

Common Mistakes

Confusing CapacityScheduler with FairScheduler

Using incorrect class names

Misspelling the scheduler class

The Fair Scheduler class is org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler. This sets YARN to use Fair Scheduler.

3fill in blank

hard

Fix the error in the code to set the YARN scheduler to Priority Scheduler.

Hadoop

yarn.scheduler.class=[1]

Drag options to blanks, or click blank then click option'

Aorg.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.PriorityScheduler

Borg.apache.hadoop.yarn.server.resourcemanager.scheduler.priority.PriorityScheduler

Corg.apache.hadoop.yarn.server.resourcemanager.scheduler.PriorityScheduler

Dorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.PriorityScheduler

Attempts:

3 left

💡 Hint

Common Mistakes

Using wrong package paths for PriorityScheduler

Misspelling the class name

Confusing PriorityScheduler with other schedulers

The correct class for Priority Scheduler is org.apache.hadoop.yarn.server.resourcemanager.scheduler.PriorityScheduler. Other options are incorrect package paths.

4fill in blank

hard

Fill both blanks to create a dictionary mapping scheduler names to their class paths.

Hadoop

scheduler_classes = {'capacity': '[1]', 'fair': '[2]'}

Drag options to blanks, or click blank then click option'

Aorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

Borg.apache.hadoop.yarn.server.resourcemanager.scheduler.PriorityScheduler

Corg.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

Dorg.apache.hadoop.yarn.server.resourcemanager.scheduler.RoundRobinScheduler

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing up scheduler names and class paths

Using incorrect class paths

Swapping the values for 'capacity' and 'fair'

The dictionary maps 'capacity' to CapacityScheduler class and 'fair' to FairScheduler class.

5fill in blank

hard

Fill all three blanks to create a dictionary with scheduler names as keys, their class paths as values, and a filter for schedulers starting with 'org.apache.hadoop.yarn'.

Hadoop

schedulers = {k: v for k, v in {'capacity': '[1]', 'fair': '[2]', 'priority': '[3]'}.items() if v.startswith('org.apache.hadoop.yarn')}

Drag options to blanks, or click blank then click option'

Aorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

Borg.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

Corg.apache.hadoop.yarn.server.resourcemanager.scheduler.PriorityScheduler

Dcom.example.custom.scheduler.CustomScheduler

Attempts:

3 left

💡 Hint

Common Mistakes

Including custom or invalid scheduler class paths

Not matching the exact class paths

Incorrectly filtering the dictionary

The dictionary includes CapacityScheduler, FairScheduler, and PriorityScheduler class paths. The filter keeps only those starting with 'org.apache.hadoop.yarn'.

Practice

1. Which YARN scheduling policy assigns resources to jobs in the order they arrive without considering job size or priority?

easy

A. Capacity Scheduler

B. Fair Scheduler

C. FIFO Scheduler

D. Priority Scheduler

Solution

1. Step 1: Understand FIFO Scheduler behavior

   FIFO (First In First Out) scheduler assigns resources to jobs strictly by arrival time, ignoring job size or priority.

2. Step 2: Compare with other schedulers

   Capacity Scheduler and Fair Scheduler consider resource sharing and fairness, while Priority Scheduler uses job priority.

3. Final Answer:

   FIFO Scheduler -> Option C

4. Quick Check:

   FIFO = order by arrival [OK]

Hint: FIFO means first come, first served order [OK]

Common Mistakes:

• Confusing FIFO with Fair Scheduler
• Thinking Capacity Scheduler ignores job order
• Assuming Priority Scheduler is default

2. Which of the following is the correct syntax to configure the Capacity Scheduler in YARN's capacity-scheduler.xml file?

easy

A. yarn.scheduler.capacity.root.queuesdefault,production

B. yarn.scheduler.fifo.root.queuesdefault,production

C. yarn.scheduler.fair.root.queuesdefault,production

D. yarn.scheduler.priority.root.queuesdefault,production

Solution

1. Step 1: Identify correct property prefix for Capacity Scheduler

   The Capacity Scheduler uses properties starting with yarn.scheduler.capacity.

2. Step 2: Verify queue configuration syntax

   The property yarn.scheduler.capacity.root.queues defines queues like default and production.

3. Final Answer:

   <property><name>yarn.scheduler.capacity.root.queues</name><value>default,production</value></property> -> Option A

4. Quick Check:

   Capacity Scheduler uses yarn.scheduler.capacity prefix [OK]

Hint: Capacity Scheduler uses 'yarn.scheduler.capacity' prefix [OK]

Common Mistakes:

• Using fifo or fair prefix for Capacity Scheduler
• Misspelling property names
• Confusing queue names with scheduler types

3. Given the following YARN Fair Scheduler configuration snippet:
<pre>queueA: minResources=2GB, maxResources=10GB queueB: minResources=4GB, maxResources=8GB Currently, queueA is using 3GB and queueB is using 5GB.
Which queue can get more resources if a new job requests 3GB?

medium

A. queueA only

B. Both queueA and queueB

C. queueB only

D. Neither queueA nor queueB

Solution

1. Step 1: Check current usage vs maxResources

   queueA uses 3GB, max is 10GB, so it can get up to 7GB more. queueB uses 5GB, max is 8GB, so it can get 3GB more.

2. Step 2: Assess if requested 3GB fits

   queueA can allocate 3GB (3+3=6GB <= 10GB). queueB can allocate 3GB (5+3=8GB = max), so it can also get 3GB.

3. Step 3: Consider Fair Scheduler policy

   Fair Scheduler tries to balance usage. Since queueA is below queueB usage, it will prioritize queueA for extra resources, but queueB can also allocate the requested resources up to its max.

4. Final Answer:

   Both queueA and queueB -> Option B

5. Quick Check:

   Fair Scheduler favors less used queue but both can allocate requested resources [OK]

Hint: Fair Scheduler favors queues with less current usage [OK]

Common Mistakes:

• Ignoring maxResources limits
• Assuming both queues get resources equally
• Not considering current usage in decision

4. You have configured the Capacity Scheduler but notice that some queues are starved and not getting resources. What is the most likely cause?

medium

A. Queue capacities are not properly set, causing imbalance

B. FIFO Scheduler is enabled instead of Capacity Scheduler

C. The yarn.scheduler.capacity.root.queues property is missing

D. Fair Scheduler is overriding Capacity Scheduler settings

Solution

1. Step 1: Understand Capacity Scheduler behavior

   Capacity Scheduler allocates resources based on configured queue capacities.

2. Step 2: Identify cause of starvation

   If queue capacities are not set correctly, some queues may get more resources, starving others.

3. Step 3: Rule out other options

   Missing property would cause scheduler failure, not starvation. FIFO and Fair Scheduler are different policies, not overriding Capacity Scheduler if configured correctly.

4. Final Answer:

   Queue capacities are not properly set, causing imbalance -> Option A

5. Quick Check:

   Capacity settings control resource distribution [OK]

Hint: Check queue capacity percentages to avoid starvation [OK]

Common Mistakes:

• Assuming missing property causes starvation
• Confusing scheduler types
• Ignoring capacity configuration importance

5. You want to run a multi-tenant Hadoop cluster where each department gets guaranteed minimum resources but can use more if available. Which YARN scheduling policy and configuration best fits this need?

hard

A. Priority Scheduler with fixed job priorities

B. FIFO Scheduler with strict job ordering

C. Fair Scheduler with no minimum resource guarantees

D. Capacity Scheduler with minimum and maximum queue capacities set

Solution

1. Step 1: Understand multi-tenant resource needs

   Departments need guaranteed minimum resources but flexible usage if available.

2. Step 2: Match policy features

   Capacity Scheduler supports minimum and maximum queue capacities, ensuring guarantees and flexible sharing.

3. Step 3: Rule out other policies

   FIFO does not guarantee resources per tenant. Fair Scheduler can share fairly but may lack strict minimum guarantees. Priority Scheduler focuses on job priority, not resource guarantees per tenant.

4. Final Answer:

   Capacity Scheduler with minimum and maximum queue capacities set -> Option D

5. Quick Check:

   Capacity Scheduler = guaranteed min + flexible max [OK]

Hint: Capacity Scheduler supports min and max resource guarantees [OK]

Common Mistakes:

• Choosing FIFO for guaranteed resources
• Ignoring min capacity in Fair Scheduler
• Confusing priority with resource guarantees