YARN scheduling policies in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding YARN Capacity Scheduler Queues

In YARN's Capacity Scheduler, what is the primary purpose of defining multiple queues?

A To allocate cluster resources among different organizations or teams fairly

B To increase the total number of nodes in the cluster automatically

C To reduce the memory usage of the ResourceManager

D To speed up the execution of a single job by splitting it into multiple tasks

💡 Hint

Think about how resources are shared among users or groups in a multi-tenant environment.

Capacity Scheduler uses queues to divide cluster resources fairly among different organizations or teams, ensuring each gets a guaranteed capacity.

❓ Predict Output

intermediate

Output of YARN Fair Scheduler Allocation Calculation

Given the following simplified resource allocation snippet in YARN Fair Scheduler, what is the value of allocatedMemory after execution?

Hadoop

allocatedMemory = 0
queueCapacities = {'queueA': 40, 'queueB': 60}
clusterMemory = 10000
for queue, capacity in queueCapacities.items():
    allocatedMemory += clusterMemory * (capacity / 100)
print(allocatedMemory)

A 4000

B 10000

C 6000

D 0

💡 Hint

Sum the memory allocated to each queue based on their capacity percentages.

The loop sums 40% and 60% of 10000, which equals 10000 total allocated memory.

❓ data_output

advanced

YARN Scheduler Resource Allocation Table

Consider a YARN cluster with 3 queues: Q1, Q2, Q3. The cluster has 120 GB memory. The Capacity Scheduler assigns capacities as 50%, 30%, and 20% respectively. If Q1 uses 40 GB, Q2 uses 20 GB, and Q3 uses 10 GB, what is the remaining available memory per queue?

Hadoop

cluster_memory = 120
queue_capacities = {'Q1': 0.5, 'Q2': 0.3, 'Q3': 0.2}
queue_usage = {'Q1': 40, 'Q2': 20, 'Q3': 10}
remaining_memory = {}
for q in queue_capacities:
    max_alloc = cluster_memory * queue_capacities[q]
    remaining_memory[q] = max_alloc - queue_usage[q]
print(remaining_memory)

A {'Q1': 20.0, 'Q2': 10.0, 'Q3': 10.0}

B {'Q1': 40.0, 'Q2': 10.0, 'Q3': 20.0}

C {'Q1': 20.0, 'Q2': 16.0, 'Q3': 14.0}

D {'Q1': 30.0, 'Q2': 16.0, 'Q3': 14.0}

💡 Hint

Calculate max allocation per queue and subtract current usage.

Q1 max: 60 GB - 40 GB = 20 GB; Q2 max: 36 GB - 20 GB = 16 GB; Q3 max: 24 GB - 10 GB = 14 GB.

🔧 Debug

advanced

Identify the Error in YARN FIFO Scheduler Code

What error will the following YARN FIFO Scheduler code snippet produce?

Hadoop

jobs = ['job1', 'job2', 'job3']
for i in range(len(jobs)):
    print(jobs[i+1])

A IndexError: list index out of range

B SyntaxError: invalid syntax

C TypeError: 'int' object is not iterable

D No error, prints all jobs

💡 Hint

Check the loop index and list access carefully.

The loop tries to access jobs[i+1] when i is at the last index, causing an IndexError.

🚀 Application

expert

Choosing the Best Scheduler for a Mixed Workload

A company runs both long-running batch jobs and short interactive jobs on a YARN cluster. Which scheduler policy is best suited to ensure fair resource sharing and low latency for interactive jobs?

A None, use external resource manager instead

B FIFO Scheduler to process jobs in submission order

C Capacity Scheduler with strict queue capacities

D Fair Scheduler to dynamically share resources among jobs

💡 Hint

Consider which scheduler balances fairness and responsiveness.

The Fair Scheduler dynamically shares resources, giving short jobs quick access while ensuring fairness for long jobs.