Block storage and replication in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding HDFS Block Replication Factor

In Hadoop Distributed File System (HDFS), what happens if the replication factor of a file is set to 1?

A The file is stored as a single block on one DataNode without any copies.

B The file is split into multiple blocks, each replicated twice on different DataNodes.

C The file is stored on the NameNode only without any DataNode storage.

D The file is replicated on all DataNodes in the cluster.

💡 Hint

Think about what replication factor means for data copies.

A replication factor of 1 means there is only one copy of each block, stored on a single DataNode. No additional copies exist for fault tolerance.

❓ data_output

intermediate

Calculating Number of Blocks for a File

Given a file size of 450 MB and HDFS block size of 128 MB, how many blocks will HDFS create to store this file?

A 3 blocks

B 5 blocks

C 4 blocks

D 6 blocks

💡 Hint

Divide file size by block size and round up.

450 MB divided by 128 MB equals about 3.52, so HDFS creates 4 blocks to store the entire file.

❓ Predict Output

advanced

HDFS Block Replication Command Output

What is the output of the following HDFS command? hdfs fsck /user/data/file.txt -files -blocks -racks

Hadoop

Sample output snippet:

Status: HEALTHY
 Total size: 2560 B
 Total blocks: 2 (avg. block size 1280 B)
 Minimally replicated blocks: 2
 Over-replicated blocks: 0
 Under-replicated blocks: 0

Block replica on datanode1:50010
Block replica on datanode2:50010

A The file has 2 blocks but some blocks are under-replicated.

B The file is missing blocks and is unhealthy.

C The file has 1 block with over-replication.

D The file has 2 blocks, each fully replicated with no under or over replication.

💡 Hint

Look at the replication status lines carefully.

The output shows 2 blocks with minimal replication and no under or over replication, indicating healthy block storage.

❓ visualization

advanced

Visualizing Block Distribution Across DataNodes

You have a file split into 3 blocks with replication factor 3. Which visualization best represents the block distribution across 3 DataNodes?

A Each block is stored on exactly 3 different DataNodes, with blocks distributed evenly.

B Each block is stored on exactly one DataNode only.

C Each DataNode stores all 3 blocks (full replication on each node).

D Blocks are stored randomly with some blocks missing replicas.

💡 Hint

Replication factor 3 means 3 copies of each block on different nodes.

With replication factor 3, each block has 3 copies on different DataNodes, ensuring fault tolerance and balanced storage.

🔧 Debug

expert

Diagnosing Under-Replication in HDFS

You notice that the HDFS report shows some blocks as under-replicated. Which of the following is the most likely cause?

A The replication factor is set to 1 for the file.

B One or more DataNodes storing replicas are down or unreachable.

C The file size is smaller than the block size.

D The NameNode is running out of memory.

💡 Hint

Think about what causes missing replicas in a distributed system.

Under-replication usually happens when some DataNodes holding replicas are offline or unreachable, reducing the number of available copies.

Practice

1. What is the main purpose of splitting files into blocks in Hadoop's block storage?

easy

A. To delete duplicate files automatically

B. To reduce the file size permanently

C. To encrypt the data for security

D. To make large files easier to manage and process

Solution

1. Step 1: Understand file splitting in Hadoop

   Hadoop splits large files into smaller blocks to handle them efficiently across many machines.

2. Step 2: Identify the purpose of block storage

   This splitting helps in parallel processing and easier management of big data.

3. Final Answer:

   To make large files easier to manage and process -> Option D

4. Quick Check:

   Block storage = easier file management [OK]

Hint: Blocks split files for easier handling and processing [OK]

Common Mistakes:

• Thinking blocks reduce file size permanently
• Confusing block splitting with encryption
• Assuming blocks delete duplicates automatically

2. Which of the following is the correct way to set the replication factor to 3 in Hadoop configuration?

easy

A. replication=3

B. replication.factor=3

C. dfs.replication=3

D. dfs.replication_factor=3

Solution

1. Step 1: Recall Hadoop replication configuration property

   The correct property to set replication factor is dfs.replication.

2. Step 2: Match the exact syntax

   The value is assigned directly as dfs.replication=3 in the configuration file.

3. Final Answer:

   <code>dfs.replication=3</code> -> Option C

4. Quick Check:

   Replication factor property = dfs.replication [OK]

Hint: Use dfs.replication to set replication factor [OK]

Common Mistakes:

• Using incorrect property names like replication.factor
• Adding extra underscores in property name
• Omitting 'dfs.' prefix

3. Consider a Hadoop cluster with block size 128 MB and replication factor 3. If a file of 640 MB is stored, how many total blocks are stored across the cluster?

medium

A. 12 blocks

B. 15 blocks

C. 3 blocks

D. 5 blocks

Solution

1. Step 1: Calculate number of blocks for the file

   File size 640 MB divided by block size 128 MB gives 640 / 128 = 5 blocks.

2. Step 2: Multiply by replication factor

   Each block is replicated 3 times, so total blocks stored = 5 * 3 = 15 blocks.

3. Final Answer:

   15 blocks -> Option B

4. Quick Check:

   Total blocks = file blocks x replication factor = 15 [OK]

Hint: Multiply file blocks by replication factor [OK]

Common Mistakes:

• Counting only original blocks without replication
• Dividing file size by replication factor
• Using wrong block size in calculation

4. A Hadoop admin sets the replication factor to 2 but notices only one copy of each block is stored. What is the likely cause?

medium

A. The cluster has only one DataNode available

B. The block size is too large

C. The file is too small to replicate

D. Replication factor must be set to 1

Solution

1. Step 1: Understand replication behavior

   Replication factor sets desired copies, but actual copies depend on available DataNodes.

2. Step 2: Identify cluster limitation

   If only one DataNode exists, Hadoop cannot store multiple copies, so only one block copy is stored.

3. Final Answer:

   The cluster has only one DataNode available -> Option A

4. Quick Check:

   Replication copies depend on DataNodes count [OK]

Hint: Replication needs multiple DataNodes to store copies [OK]

Common Mistakes:

• Thinking block size affects replication copies
• Assuming small files don't replicate
• Believing replication factor must be 1

5. You want to optimize Hadoop storage for a critical dataset by increasing fault tolerance but have limited disk space. Which approach balances safety and storage best?

hard

A. Increase replication factor moderately and reduce block size slightly

B. Set replication factor to maximum and keep default block size

C. Keep replication factor low and increase block size greatly

D. Disable replication and rely on backups

Solution

1. Step 1: Understand trade-offs of replication and block size

   Higher replication increases fault tolerance but uses more disk space. Smaller block size can improve parallelism but may increase metadata overhead.

2. Step 2: Balance replication and block size

   Moderately increasing replication improves safety without excessive space use. Slightly reducing block size helps manage data efficiently.

3. Final Answer:

   Increase replication factor moderately and reduce block size slightly -> Option A

4. Quick Check:

   Moderate replication + smaller blocks = balanced safety and storage [OK]

Hint: Balance replication and block size for best storage use [OK]

Common Mistakes:

• Maxing replication wastes disk space
• Large blocks reduce fault tolerance
• Disabling replication risks data loss