Memory and container sizing in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Container Memory Allocation in Hadoop

In Hadoop YARN, if you set yarn.scheduler.maximum-allocation-mb to 8192 and yarn.nodemanager.resource.memory-mb to 16384, what is the maximum memory a single container can request?

A 8192 MB

B 16384 MB

C 24576 MB

D 4096 MB

💡 Hint

Think about the maximum allocation limit per container set by the scheduler.

The maximum memory a single container can request is limited by yarn.scheduler.maximum-allocation-mb, which is 8192 MB in this case, even though the node has 16384 MB available.

❓ data_output

intermediate

Calculating Number of Containers per Node

Given a node with 64 GB RAM and yarn.nodemanager.resource.memory-mb set to 61440 MB, if each container requests 4096 MB, how many containers can run simultaneously on this node?

A 16

B 12

C 15

D 14

💡 Hint

Divide the total available memory by the container memory request.

61440 MB / 4096 MB = 15 containers can run simultaneously.

❓ Predict Output

advanced

YARN Container Memory Calculation Output

What is the output of the following Python code simulating container memory allocation?

total_node_memory = 32768  # in MB
container_memory_request = 4096  # in MB
max_containers = total_node_memory // container_memory_request
print(max_containers)

A 6

B 7

C 9

D 8

💡 Hint

Use integer division to find how many containers fit.

32768 divided by 4096 equals 8 containers.

🔧 Debug

advanced

Identify the Error in Container Memory Configuration

Consider this YARN configuration snippet:

yarn.nodemanager.resource.memory-mb=8192
yarn.scheduler.maximum-allocation-mb=16384

What issue will this cause when running containers?

A Containers may request more memory than the node provides, causing allocation failures.

B No issue; containers will run normally.

C YARN will ignore the maximum allocation setting.

D NodeManager will allocate double the memory requested.

💡 Hint

Check if maximum allocation exceeds node memory.

The maximum allocation per container is set higher than the node's available memory, which can cause container allocation failures.

🚀 Application

expert

Optimizing Container Memory for Mixed Workloads

You have a Hadoop cluster with nodes having 128 GB RAM each. You want to run two types of workloads:

• Heavy jobs needing 16 GB per container
• Light jobs needing 4 GB per container

Given yarn.nodemanager.resource.memory-mb is set to 122880 MB, which container memory configuration allows running the maximum number of containers simultaneously without exceeding node memory?

A Set container memory to 16 GB for all jobs, run 7 containers per node

B Set container memory to 4 GB for all jobs, run 30 containers per node

C Set container memory to 8 GB for all jobs, run 15 containers per node

D Mix containers with 16 GB and 4 GB memory, run 10 heavy and 10 light containers per node

💡 Hint

Calculate total memory used by mixed containers and compare with node memory.

10 heavy containers * 16 GB = 160 GB + 10 light containers * 4 GB = 40 GB exceeds node memory. So option B is invalid.

Option B: 30 * 4 GB = 120 GB fits under 122.88 GB.

Option B: 7 * 16 GB = 112 GB fits.

Option B: 15 * 8 GB = 120 GB fits.

Option B: 10 * 16 + 10 * 4 = 200 GB exceeds node memory.

Therefore, option B allows the maximum containers.