Cluster planning and sizing in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Key factor in determining Hadoop cluster size

Which of the following is the most important factor when deciding the size of a Hadoop cluster?

A The total volume of data to be processed

B The number of users accessing the cluster

C The brand of hardware used

D The color of the server racks

💡 Hint

Think about what directly impacts storage and processing needs.

The total volume of data determines how much storage and processing power is needed, which directly affects cluster size.

❓ data_output

intermediate

Calculate total storage needed for a Hadoop cluster

You have 10 TB of raw data. Hadoop uses replication factor 3 by default. What is the total storage needed in the cluster?

A 10 TB

B 20 TB

C 3 TB

D 30 TB

💡 Hint

Remember Hadoop stores multiple copies of data for fault tolerance.

With replication factor 3, each data block is stored 3 times, so total storage is 10 TB * 3 = 30 TB.

❓ Predict Output

advanced

Estimate number of nodes needed based on CPU cores

Given a Hadoop job requires 200 CPU cores and each node has 16 cores, how many nodes are needed?

Consider only whole nodes.

A 13 nodes

B 12 nodes

C 14 nodes

D 15 nodes

💡 Hint

Divide total cores needed by cores per node and round up.

200 cores / 16 cores per node = 12.5 nodes, so 13 nodes are needed to meet the requirement.

❓ visualization

advanced

Visualize cluster storage distribution

You have a Hadoop cluster with 5 nodes. Each node has 4 TB storage. The replication factor is 3. How much usable storage is available in the cluster?

A 20 TB usable storage

B 60 TB usable storage

C 6.67 TB usable storage

D 15 TB usable storage

💡 Hint

Calculate total raw storage then divide by replication factor.

Total raw storage is 5 nodes * 4 TB = 20 TB. Usable storage = 20 TB / 3 = 6.67 TB.

🚀 Application

expert

Optimize cluster size for mixed workload

You manage a Hadoop cluster running both batch and real-time jobs. Batch jobs need high storage, real-time jobs need low latency and high CPU. Which cluster sizing strategy best balances these needs?

A Use small nodes with minimal storage and CPU, run all workloads on same nodes

B Use large nodes with high storage and many CPU cores, and separate batch and real-time workloads on different nodes

C Use nodes with only high storage, ignore CPU needs

D Use nodes with only high CPU, ignore storage needs

💡 Hint

Think about workload isolation and resource specialization.

Separating workloads on nodes optimized for their needs improves performance and resource use.

Practice

1. What is the main goal of cluster planning in Hadoop?

easy

A. To increase network speed only

B. To write complex MapReduce code

C. To ensure enough resources for big data tasks

D. To reduce the size of data stored

Solution

1. Step 1: Understand cluster planning purpose

   Cluster planning is about preparing resources like CPU, memory, and storage to handle big data jobs efficiently.

2. Step 2: Identify the main goal

   The main goal is to ensure the cluster has enough resources to process and store data reliably.

3. Final Answer:

   To ensure enough resources for big data tasks -> Option C

4. Quick Check:

   Cluster planning = resource readiness [OK]

Hint: Think about what cluster planning prepares for [OK]

Common Mistakes:

• Confusing cluster planning with coding tasks
• Thinking it only improves network speed
• Assuming it reduces data size

2. Which of the following is the correct way to calculate total storage needed for a Hadoop cluster with 10 TB data and replication factor 3?

easy

A. 10 TB * 3 = 30 TB

B. 10 TB / 3 = 3.33 TB

C. 10 TB + 3 = 13 TB

D. 10 TB - 3 = 7 TB

Solution

1. Step 1: Understand replication factor meaning

   Replication factor 3 means each data block is stored 3 times for fault tolerance.

2. Step 2: Calculate total storage

   Total storage = original data size * replication factor = 10 TB * 3 = 30 TB.

3. Final Answer:

   10 TB * 3 = 30 TB -> Option A

4. Quick Check:

   Storage needed = data size x replication factor [OK]

Hint: Multiply data size by replication factor [OK]

Common Mistakes:

• Dividing instead of multiplying
• Adding or subtracting replication factor incorrectly
• Ignoring replication factor

3. Given a Hadoop cluster with 5 nodes, each having 16 CPU cores and 64 GB RAM, what is the total CPU cores and RAM available?

medium

A. 80 CPU cores and 320 GB RAM

B. 16 CPU cores and 64 GB RAM

C. 5 CPU cores and 5 GB RAM

D. 320 CPU cores and 80 GB RAM

Solution

1. Step 1: Calculate total CPU cores

   Total CPU cores = number of nodes * CPU cores per node = 5 * 16 = 80 cores.

2. Step 2: Calculate total RAM

   Total RAM = number of nodes * RAM per node = 5 * 64 GB = 320 GB.

3. Final Answer:

   80 CPU cores and 320 GB RAM -> Option A

4. Quick Check:

   Total CPU = 5x16, Total RAM = 5x64 [OK]

Hint: Multiply nodes by cores and RAM per node [OK]

Common Mistakes:

• Adding CPU cores and RAM instead of multiplying
• Mixing CPU cores with RAM units
• Using per node values as total

4. Identify the error in this cluster sizing calculation: A cluster has 8 nodes, each with 32 GB RAM. The total RAM is calculated as 8 + 32 = 40 GB. What is wrong?

medium

A. Ignoring CPU cores

B. Multiplying nodes and RAM correctly

C. Using wrong units for RAM

D. Adding nodes and RAM instead of multiplying

Solution

1. Step 1: Understand total RAM calculation

   Total RAM should be number of nodes multiplied by RAM per node, not added.

2. Step 2: Identify the mistake

   The calculation 8 + 32 = 40 GB is incorrect; it should be 8 * 32 = 256 GB.

3. Final Answer:

   Adding nodes and RAM instead of multiplying -> Option D

4. Quick Check:

   Total RAM = nodes x RAM per node [OK]

Hint: Multiply nodes by RAM, don't add [OK]

Common Mistakes:

• Adding node count and RAM size
• Confusing units of measurement
• Ignoring multiplication for totals

5. You have 20 TB of raw data with replication factor 3 and want to plan a Hadoop cluster. Each node has 4 TB usable storage. How many nodes are needed at minimum to store the data?

hard

A. 20 nodes

B. 15 nodes

C. 60 nodes

D. 5 nodes

Solution

1. Step 1: Calculate total storage needed with replication

   Total storage = raw data x replication factor = 20 TB x 3 = 60 TB.

2. Step 2: Calculate number of nodes needed

   Nodes needed = total storage / storage per node = 60 TB / 4 TB = 15 nodes.

3. Final Answer:

   15 nodes -> Option B

4. Quick Check:

   Nodes = (data x replication) ÷ node storage [OK]

Hint: Divide total replicated data by node storage [OK]

Common Mistakes:

• Ignoring replication factor
• Dividing raw data by node storage only
• Rounding nodes incorrectly