Block storage and replication in Hadoop - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is block storage in Hadoop?

Block storage in Hadoop means breaking large files into smaller fixed-size pieces called blocks. These blocks are stored across different machines to manage big data efficiently.

beginner

Why does Hadoop replicate blocks?

Hadoop replicates blocks to make sure data is safe and available even if some machines fail. Replication copies blocks to multiple machines to avoid data loss.

intermediate

What is the default block size in Hadoop HDFS?

The default block size in Hadoop HDFS is usually 128 MB. This size helps balance between managing many blocks and efficient data processing.

intermediate

How does replication factor affect storage in Hadoop?

Replication factor is the number of copies of each block. A higher replication factor means more copies, which increases storage use but improves fault tolerance.

advanced

Explain how Hadoop ensures data reliability using block storage and replication.

Hadoop splits files into blocks and stores them on different machines. It makes multiple copies (replicas) of each block. If one machine fails, other copies keep data safe and accessible.

What is the main purpose of block replication in Hadoop?

A To speed up file deletion

B To improve data safety and availability

C To reduce the file size

D To compress data blocks

Replication creates copies of blocks to protect data if machines fail, ensuring safety and availability.

What happens to a large file in Hadoop before storage?

A It is split into smaller blocks

B It is stored as one big file

C It is converted to text

D It is deleted

Hadoop splits large files into fixed-size blocks for easier storage and processing.

What is the typical default block size in Hadoop HDFS?

A 10 MB

B 1 GB

C 64 KB

D 128 MB

The default block size in Hadoop HDFS is usually 128 MB.

If the replication factor is 3, how many copies of each block exist?

A 3

B 2

C 1

D 4

Replication factor 3 means three copies of each block are stored.

Which of these is NOT a benefit of block replication?

A Data fault tolerance

B Increased storage space

C Faster data deletion

D Data availability

Replication does not speed up deletion; it increases storage use but improves fault tolerance and availability.

Describe how Hadoop uses block storage and replication to protect data.

Think about how breaking files and copying blocks helps keep data safe.

Explain the role of the replication factor in Hadoop's data storage system.

Consider how more copies affect storage space and data safety.

Practice

1. What is the main purpose of splitting files into blocks in Hadoop's block storage?

easy

A. To delete duplicate files automatically

B. To reduce the file size permanently

C. To encrypt the data for security

D. To make large files easier to manage and process

Solution

1. Step 1: Understand file splitting in Hadoop

   Hadoop splits large files into smaller blocks to handle them efficiently across many machines.

2. Step 2: Identify the purpose of block storage

   This splitting helps in parallel processing and easier management of big data.

3. Final Answer:

   To make large files easier to manage and process -> Option D

4. Quick Check:

   Block storage = easier file management [OK]

Hint: Blocks split files for easier handling and processing [OK]

Common Mistakes:

• Thinking blocks reduce file size permanently
• Confusing block splitting with encryption
• Assuming blocks delete duplicates automatically

2. Which of the following is the correct way to set the replication factor to 3 in Hadoop configuration?

easy

A. replication=3

B. replication.factor=3

C. dfs.replication=3

D. dfs.replication_factor=3

Solution

1. Step 1: Recall Hadoop replication configuration property

   The correct property to set replication factor is dfs.replication.

2. Step 2: Match the exact syntax

   The value is assigned directly as dfs.replication=3 in the configuration file.

3. Final Answer:

   <code>dfs.replication=3</code> -> Option C

4. Quick Check:

   Replication factor property = dfs.replication [OK]

Hint: Use dfs.replication to set replication factor [OK]

Common Mistakes:

• Using incorrect property names like replication.factor
• Adding extra underscores in property name
• Omitting 'dfs.' prefix

3. Consider a Hadoop cluster with block size 128 MB and replication factor 3. If a file of 640 MB is stored, how many total blocks are stored across the cluster?

medium

A. 12 blocks

B. 15 blocks

C. 3 blocks

D. 5 blocks

Solution

1. Step 1: Calculate number of blocks for the file

   File size 640 MB divided by block size 128 MB gives 640 / 128 = 5 blocks.

2. Step 2: Multiply by replication factor

   Each block is replicated 3 times, so total blocks stored = 5 * 3 = 15 blocks.

3. Final Answer:

   15 blocks -> Option B

4. Quick Check:

   Total blocks = file blocks x replication factor = 15 [OK]

Hint: Multiply file blocks by replication factor [OK]

Common Mistakes:

• Counting only original blocks without replication
• Dividing file size by replication factor
• Using wrong block size in calculation

4. A Hadoop admin sets the replication factor to 2 but notices only one copy of each block is stored. What is the likely cause?

medium

A. The cluster has only one DataNode available

B. The block size is too large

C. The file is too small to replicate

D. Replication factor must be set to 1

Solution

1. Step 1: Understand replication behavior

   Replication factor sets desired copies, but actual copies depend on available DataNodes.

2. Step 2: Identify cluster limitation

   If only one DataNode exists, Hadoop cannot store multiple copies, so only one block copy is stored.

3. Final Answer:

   The cluster has only one DataNode available -> Option A

4. Quick Check:

   Replication copies depend on DataNodes count [OK]

Hint: Replication needs multiple DataNodes to store copies [OK]

Common Mistakes:

• Thinking block size affects replication copies
• Assuming small files don't replicate
• Believing replication factor must be 1

5. You want to optimize Hadoop storage for a critical dataset by increasing fault tolerance but have limited disk space. Which approach balances safety and storage best?

hard

A. Increase replication factor moderately and reduce block size slightly

B. Set replication factor to maximum and keep default block size

C. Keep replication factor low and increase block size greatly

D. Disable replication and rely on backups

Solution

1. Step 1: Understand trade-offs of replication and block size

   Higher replication increases fault tolerance but uses more disk space. Smaller block size can improve parallelism but may increase metadata overhead.

2. Step 2: Balance replication and block size

   Moderately increasing replication improves safety without excessive space use. Slightly reducing block size helps manage data efficiently.

3. Final Answer:

   Increase replication factor moderately and reduce block size slightly -> Option A

4. Quick Check:

   Moderate replication + smaller blocks = balanced safety and storage [OK]

Hint: Balance replication and block size for best storage use [OK]

Common Mistakes:

• Maxing replication wastes disk space
• Large blocks reduce fault tolerance
• Disabling replication risks data loss