Memory and container sizing in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Container Memory Allocation in Hadoop

In Hadoop YARN, if you set yarn.scheduler.maximum-allocation-mb to 8192 and yarn.nodemanager.resource.memory-mb to 16384, what is the maximum memory a single container can request?

A 8192 MB

B 16384 MB

C 24576 MB

D 4096 MB

💡 Hint

Think about the maximum allocation limit per container set by the scheduler.

The maximum memory a single container can request is limited by yarn.scheduler.maximum-allocation-mb, which is 8192 MB in this case, even though the node has 16384 MB available.

❓ data_output

intermediate

Calculating Number of Containers per Node

Given a node with 64 GB RAM and yarn.nodemanager.resource.memory-mb set to 61440 MB, if each container requests 4096 MB, how many containers can run simultaneously on this node?

A 16

B 12

C 15

D 14

💡 Hint

Divide the total available memory by the container memory request.

61440 MB / 4096 MB = 15 containers can run simultaneously.

❓ Predict Output

advanced

YARN Container Memory Calculation Output

What is the output of the following Python code simulating container memory allocation?

total_node_memory = 32768  # in MB
container_memory_request = 4096  # in MB
max_containers = total_node_memory // container_memory_request
print(max_containers)

A 6

B 7

C 9

D 8

💡 Hint

Use integer division to find how many containers fit.

32768 divided by 4096 equals 8 containers.

🔧 Debug

advanced

Identify the Error in Container Memory Configuration

Consider this YARN configuration snippet:

yarn.nodemanager.resource.memory-mb=8192
yarn.scheduler.maximum-allocation-mb=16384

What issue will this cause when running containers?

A Containers may request more memory than the node provides, causing allocation failures.

B No issue; containers will run normally.

C YARN will ignore the maximum allocation setting.

D NodeManager will allocate double the memory requested.

💡 Hint

Check if maximum allocation exceeds node memory.

The maximum allocation per container is set higher than the node's available memory, which can cause container allocation failures.

🚀 Application

expert

Optimizing Container Memory for Mixed Workloads

You have a Hadoop cluster with nodes having 128 GB RAM each. You want to run two types of workloads:

• Heavy jobs needing 16 GB per container
• Light jobs needing 4 GB per container

Given yarn.nodemanager.resource.memory-mb is set to 122880 MB, which container memory configuration allows running the maximum number of containers simultaneously without exceeding node memory?

A Set container memory to 16 GB for all jobs, run 7 containers per node

B Set container memory to 4 GB for all jobs, run 30 containers per node

C Set container memory to 8 GB for all jobs, run 15 containers per node

D Mix containers with 16 GB and 4 GB memory, run 10 heavy and 10 light containers per node

💡 Hint

Calculate total memory used by mixed containers and compare with node memory.

10 heavy containers * 16 GB = 160 GB + 10 light containers * 4 GB = 40 GB exceeds node memory. So option B is invalid.

Option B: 30 * 4 GB = 120 GB fits under 122.88 GB.

Option B: 7 * 16 GB = 112 GB fits.

Option B: 15 * 8 GB = 120 GB fits.

Option B: 10 * 16 + 10 * 4 = 200 GB exceeds node memory.

Therefore, option B allows the maximum containers.

Practice

1. What is the main purpose of setting container memory size in Hadoop?

easy

A. To set the network bandwidth for data transfer

B. To increase the disk space available for Hadoop jobs

C. To control the number of CPU cores assigned to a task

D. To limit the amount of memory a task can use during execution

Solution

1. Step 1: Understand container memory role and compare options

   Container memory size limits how much RAM a Hadoop task can use while running. Only To limit the amount of memory a task can use during execution correctly describes this purpose; others relate to disk, CPU, or network, which are unrelated.

2. Final Answer:

   To limit the amount of memory a task can use during execution -> Option D

3. Quick Check:

   Container memory = task RAM limit [OK]

Hint: Container memory controls RAM usage per task [OK]

Common Mistakes:

• Confusing memory with disk space
• Thinking container memory controls CPU cores
• Assuming it sets network bandwidth

2. Which of the following is the correct way to set the Java heap size for a Hadoop container in the configuration?

easy

A. mapreduce.map.memory.mb=512\nmapreduce.map.java.opts=-Xmx5120m

B. mapreduce.map.memory.mb=2048\nmapreduce.map.java.opts=-Xmx1536m

C. mapreduce.map.memory.mb=1024\nmapreduce.map.java.opts=-Xmx2048m

D. mapreduce.map.memory.mb=4096\nmapreduce.map.java.opts=-Xmx8192m

Solution

1. Step 1: Understand heap size vs container memory and check options

   Java heap size (-Xmx) must be less than container memory to avoid errors. mapreduce.map.memory.mb=2048\nmapreduce.map.java.opts=-Xmx1536m sets container memory to 2048 MB and heap to 1536 MB, which is valid. Others have heap larger than container memory, causing errors.

2. Final Answer:

   mapreduce.map.memory.mb=2048\nmapreduce.map.java.opts=-Xmx1536m -> Option B

3. Quick Check:

   Heap size < container memory [OK]

Hint: Heap size must be smaller than container memory [OK]

Common Mistakes:

• Setting heap size larger than container memory
• Using inconsistent units (MB vs GB)
• Forgetting to set Java opts after memory

3. Given the following configuration:

mapreduce.reduce.memory.mb=4096
mapreduce.reduce.java.opts=-Xmx3072m

What will happen if a reduce task tries to use 3500 MB heap memory?

medium

A. The task will run successfully using 3500 MB heap

B. The task will automatically increase container memory

C. The task will fail due to exceeding Java heap size limit

D. The task will fail due to exceeding container memory limit

Solution

1. Step 1: Compare requested heap with limits

   Heap limit is 3072 MB, but task tries 3500 MB, which exceeds Java heap limit (-Xmx). Container memory is 4096 MB, so 3500 MB fits container but exceeds heap limit, causing failure.

2. Final Answer:

   The task will fail due to exceeding Java heap size limit -> Option C

3. Quick Check:

   Heap > Java heap limit = failure [OK]

Hint: Heap usage must not exceed Java heap size (-Xmx) [OK]

Common Mistakes:

• Confusing container memory with Java heap size
• Assuming container memory limits heap directly
• Thinking container memory auto-adjusts

4. Identify the error in this Hadoop container memory configuration:

mapreduce.map.memory.mb=1024
mapreduce.map.java.opts=-Xmx1536m

medium

A. Java heap size is larger than container memory size

B. Container memory size is too large for the job

C. Java opts syntax is incorrect

D. No error, configuration is valid

Solution

1. Step 1: Compare Java heap size and container memory

   Heap size (-Xmx1536m) is 1536 MB, which is larger than container memory (1024 MB). This causes the task to fail because heap cannot exceed container memory.

2. Final Answer:

   Java heap size is larger than container memory size -> Option A

3. Quick Check:

   Heap > container memory = error [OK]

Hint: Heap size must never exceed container memory [OK]

Common Mistakes:

• Ignoring heap size limits
• Assuming syntax error instead of sizing error
• Thinking container memory can auto-expand

5. You have a Hadoop job with 10 map tasks. Each map task requires 3 GB of Java heap memory. To avoid container failures, what is the minimum container memory size you should set per map task?

hard

A. 4 GB

B. 3 GB

C. 6 GB

D. 10 GB

Solution

1. Step 1: Calculate container memory with overhead

   Container memory must be larger than Java heap size to allow overhead (usually 25-30%). Heap = 3 GB, so container memory should be at least 3 GB + 25% = 3.75 GB, rounded to 4 GB.

2. Final Answer:

   4 GB -> Option A

3. Quick Check:

   Container memory > heap size + overhead [OK]

Hint: Add 25% overhead to heap for container memory [OK]

Common Mistakes:

• Setting container memory equal to heap size
• Ignoring overhead memory needs
• Confusing total job memory with per-task memory