Block storage and replication in Hadoop

Introduction

Block storage breaks big files into smaller pieces to store them easily. Replication makes copies of these pieces to keep data safe if something breaks.

When storing large files like videos or logs in a distributed system.

When you want to protect data from hardware failures by keeping copies.

When you need to read data quickly from multiple places at once.

When managing storage across many computers in a cluster.

When you want to balance storage load and improve fault tolerance.

Syntax

Hadoop

Blocks are fixed-size pieces of data (default 128MB in Hadoop).
Each block is stored on different nodes.
Replication factor defines how many copies of each block exist.
Example: replication factor = 3 means 3 copies of each block.

Block size can be changed in Hadoop configuration to optimize performance.

Replication ensures data is safe even if some nodes fail.

Examples

This example shows how a 300MB file is split into three blocks and each block is copied three times on different nodes.

Hadoop

Block size = 128MB
Replication factor = 3
File size = 300MB

Blocks:
Block 1: 128MB
Block 2: 128MB
Block 3: 44MB

Each block stored on 3 different nodes.

This command sets the replication factor of a file to 2, meaning two copies of each block will be stored.

Hadoop

Change replication factor:
hdfs dfs -setrep -w 2 /user/data/file.txt

Sample Program

This Python code uses Pydoop to connect to Hadoop HDFS, get file info including block size and replication factor, and list where each block is stored.

Hadoop

from pydoop import hdfs

# Connect to HDFS
fs = hdfs.hdfs()

# Check block size and replication for a file
file_path = '/user/hadoop/sample.txt'
info = fs.stat(file_path)

print(f"File: {file_path}")
print(f"Block size: {info.block_size} bytes")
print(f"Replication factor: {info.replication}")

# List block locations
blocks = fs.get_block_locations(file_path, 0, info.size)
for i, block in enumerate(blocks, 1):
    hosts = ', '.join(block.hosts)
    print(f"Block {i} stored on nodes: {hosts}")

fs.close()

Important Notes

Replication factor affects storage space: more copies use more space.

Block size affects performance: bigger blocks reduce overhead but may waste space.

Hadoop automatically manages block placement to balance load and reliability.

Summary

Block storage splits files into smaller pieces for easier management.

Replication keeps multiple copies of blocks to protect data.

Hadoop uses block size and replication factor settings to optimize storage and safety.