Container allocation in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Container Allocation in Hadoop YARN

Which statement best describes the role of container allocation in Hadoop YARN?

A Containers are the physical machines where Hadoop daemons run.

B Containers store the data blocks in HDFS for distributed storage.

C Containers are used to manage network traffic between Hadoop nodes.

D Containers are allocated by the ResourceManager to run application tasks with specific resource requirements.

💡 Hint

Think about how YARN manages resources for running tasks.

In Hadoop YARN, the ResourceManager allocates containers which are units of resources (CPU, memory) to run application tasks. Containers do not store data or manage network traffic directly.

❓ Predict Output

intermediate

Output of Container Allocation Log Snippet

Given the following YARN container allocation log snippet, what is the total memory allocated to containers?

Hadoop

2024-06-01 10:00:00,000 INFO ResourceManager: Allocated container container_01 with 2048 MB
2024-06-01 10:00:01,000 INFO ResourceManager: Allocated container container_02 with 1024 MB
2024-06-01 10:00:02,000 INFO ResourceManager: Allocated container container_03 with 4096 MB

A 7168 MB

B 7164 MB

C 7167 MB

D 7169 MB

💡 Hint

Add the memory sizes of all allocated containers.

The total memory is 2048 + 1024 + 4096 = 7168 MB.

❓ data_output

advanced

Resulting Container Allocation Table

Given the following resource requests and allocations, what is the final container allocation table?

Requests: Task1 needs 2 CPUs and 2048 MB memory, Task2 needs 1 CPU and 1024 MB memory.

Allocations: Task1 got 2 CPUs and 2048 MB, Task2 got 1 CPU and 512 MB.

Hadoop

import pandas as pd

requests = {'Task': ['Task1', 'Task2'], 'CPU_requested': [2, 1], 'Memory_requested_MB': [2048, 1024]}
allocations = {'Task': ['Task1', 'Task2'], 'CPU_allocated': [2, 1], 'Memory_allocated_MB': [2048, 512]}

req_df = pd.DataFrame(requests)
alloc_df = pd.DataFrame(allocations)

result = pd.merge(req_df, alloc_df, on='Task')
result

A [{'Task': 'Task1', 'CPU_requested': 2, 'Memory_requested_MB': 2048, 'CPU_allocated': 1, 'Memory_allocated_MB': 2048}, {'Task': 'Task2', 'CPU_requested': 1, 'Memory_requested_MB': 1024, 'CPU_allocated': 1, 'Memory_allocated_MB': 512}]

B [{'Task': 'Task1', 'CPU_requested': 2, 'Memory_requested_MB': 2048, 'CPU_allocated': 2, 'Memory_allocated_MB': 2048}, {'Task': 'Task2', 'CPU_requested': 1, 'Memory_requested_MB': 1024, 'CPU_allocated': 1, 'Memory_allocated_MB': 512}]

C [{'Task': 'Task1', 'CPU_requested': 2, 'Memory_requested_MB': 2048, 'CPU_allocated': 2, 'Memory_allocated_MB': 1024}, {'Task': 'Task2', 'CPU_requested': 1, 'Memory_requested_MB': 1024, 'CPU_allocated': 1, 'Memory_allocated_MB': 512}]

D [{'Task': 'Task1', 'CPU_requested': 2, 'Memory_requested_MB': 2048, 'CPU_allocated': 2, 'Memory_allocated_MB': 512}, {'Task': 'Task2', 'CPU_requested': 1, 'Memory_requested_MB': 1024, 'CPU_allocated': 1, 'Memory_allocated_MB': 1024}]

💡 Hint

Look carefully at the allocated CPU and memory for each task.

The merged table shows requested and allocated resources side by side. Task1 got full requested resources, Task2 got less memory than requested.

❓ visualization

advanced

Interpreting Container Allocation Visualization

You see a bar chart showing container memory allocation for 3 tasks: Task1 (2048 MB), Task2 (1024 MB), Task3 (4096 MB). Which statement is true about the visualization?

A Task3 has the highest memory allocation, twice that of Task1.

B Task2 has the highest memory allocation, more than Task3.

C All tasks have equal memory allocation.

D Task1 has the lowest memory allocation, less than Task2.

💡 Hint

Compare the bar heights representing memory sizes.

Task3's 4096 MB is twice Task1's 2048 MB. Task2 has the smallest allocation.

🔧 Debug

expert

Identifying Error in Container Allocation Code

What error does the following Hadoop container allocation pseudo-code produce?

containers = []
for i in range(3):
    container = {'id': f'container_{i}', 'memory': 2048 * i}
containers.append(container)
print(containers)

A The list 'containers' will contain three containers with memory 0, 2048, and 4096 MB.

B TypeError because 'containers' is not defined.

C Only one container will be in the list because append is outside the loop.

D SyntaxError due to missing colon after for loop.

💡 Hint

Check the indentation of the append statement.

The append is outside the loop, so only the last container is added once. The list ends with one container with memory 4096 MB.

Practice

1. What is the primary role of a container in Hadoop's resource management?

easy

A. To monitor network traffic

B. To store data permanently

C. To allocate resources for running tasks

D. To manage user authentication

Solution

1. Step 1: Understand container function in Hadoop

   Containers are units of resources like CPU and memory assigned to run tasks.

2. Step 2: Differentiate from other Hadoop components

   Containers do not store data or manage authentication; they only allocate resources.

3. Final Answer:

   To allocate resources for running tasks -> Option C

4. Quick Check:

   Container = resource allocation [OK]

Hint: Containers run tasks by allocating resources [OK]

Common Mistakes:

• Confusing containers with data storage
• Thinking containers handle security
• Assuming containers monitor network

2. Which of the following is the correct syntax to configure the memory size of a container in Hadoop's yarn-site.xml?

easy

A. yarn.nodemanager.memory.mb4096

B. yarn.container.memory.size4096

C. yarn.resource.memory4096

D. yarn.nodemanager.resource.memory-mb4096

Solution

1. Step 1: Identify correct property name

   The correct property to set NodeManager's memory resource is 'yarn.nodemanager.resource.memory-mb'.

2. Step 2: Verify XML syntax

   The syntax uses <property>, <name>, and <value> tags correctly.

3. Final Answer:

   <property><name>yarn.nodemanager.resource.memory-mb</name><value>4096</value></property> -> Option D

4. Quick Check:

   Correct property name and XML format [OK]

Hint: Look for 'nodemanager.resource.memory-mb' in config [OK]

Common Mistakes:

• Using incorrect property names
• Missing XML tags or wrong tag order
• Confusing container memory with NodeManager memory

3. Given the following pseudo-code for container allocation in Hadoop:

available_memory = 8192
container_memory = 2048
containers_allocated = 0
while available_memory >= container_memory:
    available_memory -= container_memory
    containers_allocated += 1
print(containers_allocated)

What is the output of this code?

medium

A. 3

B. 4

C. 5

D. 6

Solution

1. Step 1: Understand the loop condition and variables

   Initial available_memory is 8192, container_memory is 2048. Loop runs while available_memory >= container_memory.

2. Step 2: Calculate iterations

   8192 / 2048 = 4, so loop runs 4 times, subtracting 2048 each time and incrementing containers_allocated.

3. Final Answer:

   4 -> Option B

4. Quick Check:

   8192 ÷ 2048 = 4 containers [OK]

Hint: Divide total memory by container size [OK]

Common Mistakes:

• Off-by-one error in loop count
• Confusing subtraction with division
• Ignoring loop exit condition

4. Identify the error in this container allocation snippet:

available_memory = 4096
container_memory = 1024
containers_allocated = 0
while available_memory > container_memory:
    available_memory -= container_memory
    containers_allocated += 1
print(containers_allocated)

medium

A. The loop condition should be >= instead of >

B. The container_memory value is too large

C. containers_allocated should start at 1

D. available_memory should not be decremented

Solution

1. Step 1: Analyze loop condition impact

   The loop runs while available_memory > container_memory, so it stops when equal.

2. Step 2: Check if last container is allocated

   When available_memory equals container_memory (1024), loop stops, missing one container allocation.

3. Final Answer:

   The loop condition should be >= instead of > -> Option A

4. Quick Check:

   Use >= to include equal memory case [OK]

Hint: Use >= to include exact memory match [OK]

Common Mistakes:

• Using > excludes last container
• Starting count at 1 instead of 0
• Not decrementing available memory

5. You have a Hadoop cluster with 3 nodes. Each node has 16 GB RAM. You want to allocate containers of 4 GB each. Considering some memory overhead, you reserve 2 GB per node for system processes. How many containers can you allocate in total across the cluster?

hard

A. 9

B. 12

C. 36

D. 42

Solution

1. Step 1: Calculate usable memory per node

   Each node has 16 GB RAM, reserve 2 GB for system, so usable memory = 16 - 2 = 14 GB per node.

2. Step 2: Calculate containers per node

   Each container uses 4 GB, so containers per node = 14 / 4 = 3 (integer division).

3. Step 3: Calculate total containers

   Total containers = 3 nodes * 3 containers = 9 containers.

4. Step 4: Re-check calculation

   Wait, 14 / 4 = 3.5, but only full containers count, so 3 per node. Total 3*3=9 containers.

5. Final Answer:

   9 -> Option A

6. Quick Check:

   Usable memory ÷ container size x nodes = total containers [OK]

Hint: Subtract overhead, divide by container size, multiply nodes [OK]

Common Mistakes:

• Not subtracting system reserved memory
• Using float containers instead of integer
• Multiplying before dividing