HDFS read and write operations in Hadoop - Deep Dive

Overview - HDFS read and write operations

What is it?

HDFS read and write operations are the ways data is stored and retrieved in the Hadoop Distributed File System. When writing, data is split into blocks and saved across many computers. When reading, these blocks are fetched and combined to give the original data. This system helps handle very large files efficiently by spreading the work.

Why it matters

Without efficient read and write operations, handling big data would be slow and unreliable. HDFS makes sure data is stored safely and can be accessed quickly even if some computers fail. This allows companies to analyze huge datasets and make decisions faster, powering many modern data applications.

Where it fits

Before learning HDFS operations, you should understand basic file systems and distributed computing concepts. After this, you can explore Hadoop MapReduce, YARN resource management, and advanced data processing frameworks like Apache Spark that use HDFS for storage.

Mental Model

Core Idea

HDFS breaks big data into blocks, stores them across many machines, and reads or writes these blocks in parallel to handle large-scale data efficiently and reliably.

Think of it like...

Imagine a big book that is too heavy to carry. You tear it into chapters and give each chapter to a different friend. When you want to read the book, you ask all your friends to send their chapters at the same time, then you put the chapters back together to read the whole story.

┌───────────────┐
│ Client Write  │
└──────┬────────┘
       │
       ▼
┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│ DataNode 1    │      │ DataNode 2    │      │ DataNode 3    │
│ Block 1       │      │ Block 2       │      │ Block 3       │
└───────────────┘      └───────────────┘      └───────────────┘
       ▲                    ▲                    ▲
       │                    │                    │
┌──────┴────────┐   ┌───────┴────────┐   ┌───────┴────────┐
│ Client Read   │   │ Client Read    │   │ Client Read    │
└───────────────┘   └────────────────┘   └────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding HDFS Block Storage

Concept: HDFS stores files by splitting them into fixed-size blocks and distributing these blocks across multiple machines.

When you save a file in HDFS, it is divided into blocks (usually 128 MB each). Each block is stored on different DataNodes (computers) in the cluster. This allows HDFS to handle very large files by spreading the data and workload.

Result

Files are stored as multiple blocks across different machines, enabling parallel storage and fault tolerance.

Understanding block storage is key because it explains how HDFS manages big data efficiently and recovers from failures.

2

Foundation

Role of NameNode and DataNodes

Concept: HDFS uses a NameNode to manage metadata and DataNodes to store actual data blocks.

The NameNode keeps track of where each block of a file is stored but does not store the data itself. DataNodes hold the blocks and handle read/write requests. The client communicates with the NameNode first to find block locations, then talks directly to DataNodes.

Result

The system separates metadata management and data storage, improving scalability and performance.

Knowing the roles of NameNode and DataNodes clarifies how HDFS coordinates data access and maintains system health.

3

Intermediate

How HDFS Write Operation Works

🤔Before reading on: do you think the client writes data to all DataNodes at once or one after another? Commit to your answer.

Concept: Writing data in HDFS involves the client sending data blocks to a pipeline of DataNodes for replication.

When writing, the client asks the NameNode for DataNodes to store replicas. The client sends the first block to the first DataNode, which forwards it to the next, and so on, forming a pipeline. This process repeats for all blocks until the file is fully written.

Result

Data is written in a chain to multiple DataNodes, ensuring copies exist for fault tolerance.

Understanding the pipeline write process reveals how HDFS balances speed and reliability during data storage.

4

Intermediate

How HDFS Read Operation Works

🤔Before reading on: do you think the client reads data from the NameNode or directly from DataNodes? Commit to your answer.

Concept: Reading data involves the client asking the NameNode for block locations and then fetching blocks directly from DataNodes.

To read a file, the client contacts the NameNode to get the list of DataNodes holding each block. Then, the client reads blocks in order directly from these DataNodes. This allows parallel and efficient data retrieval.

Result

The client reconstructs the file by reading blocks from multiple DataNodes in sequence.

Knowing that clients read directly from DataNodes explains how HDFS achieves high read throughput.

5

Intermediate

Data Replication and Fault Tolerance

Concept: HDFS replicates each block multiple times on different DataNodes to prevent data loss.

By default, each block is stored three times on separate machines. If one DataNode fails, the system still has copies elsewhere. The NameNode monitors DataNodes and triggers replication if copies are lost.

Result

Data remains safe and accessible even if some machines fail.

Replication is the backbone of HDFS reliability, ensuring continuous data availability.

6

Advanced

Pipeline Write and Acknowledgment Process

🤔Before reading on: do you think acknowledgments in the write pipeline flow from client to DataNodes or the other way? Commit to your answer.

Concept: During write, acknowledgments flow backward through the pipeline to confirm data receipt and durability.

When the client sends a block to the first DataNode, it forwards it down the pipeline. Each DataNode writes the block and sends an acknowledgment upstream. The client waits for all acknowledgments before sending the next block, ensuring data is safely stored.

Result

Write operations are confirmed step-by-step, preventing data loss during transfer.

Understanding acknowledgment flow explains how HDFS guarantees data integrity during writes.

7

Expert

Handling Data Consistency and Failures

🤔Before reading on: do you think HDFS allows clients to modify files after writing? Commit to your answer.

Concept: HDFS is designed for write-once, read-many files and handles failures by detecting and recovering missing blocks.

Once a file is written and closed, it cannot be modified. If a DataNode fails during write, the NameNode detects missing replicas and triggers re-replication. Clients retry writes if failures occur. This design simplifies consistency and fault handling.

Result

HDFS maintains strong consistency and recovers automatically from node failures.

Knowing HDFS's write-once model and failure recovery clarifies why it suits big data workloads but not frequent updates.

Under the Hood

HDFS uses a master-slave architecture where the NameNode manages metadata like file-to-block mapping and DataNode locations. DataNodes store blocks and send heartbeats to the NameNode to report health. During writes, data flows through a pipeline of DataNodes with acknowledgments confirming storage. Reads bypass the NameNode after block locations are known, going directly to DataNodes. Replication ensures fault tolerance by keeping multiple copies of each block on different nodes.

Why designed this way?

HDFS was designed to handle very large files on commodity hardware that can fail often. Separating metadata and data storage allows scaling to thousands of nodes. The pipeline write and replication model balances performance and reliability. The write-once-read-many model simplifies consistency and fits batch processing needs. Alternatives like traditional file systems could not scale or handle failures as efficiently.

┌───────────────┐          ┌───────────────┐
│    Client     │          │   NameNode    │
└──────┬────────┘          └──────┬────────┘
       │                          │
       │ Request block locations   │
       │─────────────────────────▶│
       │                          │
       │          Block locations │
       │◀─────────────────────────│
       │                          │
       │                          │
       ▼                          ▼
┌───────────────┐          ┌───────────────┐
│   DataNode 1  │◀────────▶│   DataNode 2  │
└───────────────┘          └───────────────┘
       ▲                          ▲
       │                          │
       └─────────────Pipeline─────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think HDFS allows files to be changed after writing? Commit to yes or no.

Common Belief: HDFS files can be modified anytime like regular files.

Reality: HDFS files are write-once and cannot be modified after closing.

Why it matters: Trying to update files causes errors and breaks assumptions about data consistency.

Quick: Do you think the NameNode stores the actual data blocks? Commit to yes or no.

Common Belief: The NameNode stores all the data blocks of files.

Reality: The NameNode only stores metadata; DataNodes store the actual data blocks.

Why it matters: Misunderstanding this can lead to wrong expectations about storage capacity and performance.

Quick: Do you think the client reads data through the NameNode? Commit to yes or no.

Common Belief: Clients read data by asking the NameNode for every block read.

Reality: Clients get block locations from the NameNode once, then read directly from DataNodes.

Why it matters: Believing otherwise can cause confusion about read performance and network traffic.

Quick: Do you think replication happens only after the entire file is written? Commit to yes or no.

Common Belief: HDFS replicates data blocks only after the whole file is saved.

Reality: Replication happens block-by-block during the write pipeline.

Why it matters: This misconception can lead to misunderstanding how HDFS ensures data durability during writes.

Expert Zone

1

The write pipeline uses a chain of DataNodes where each node forwards data and acknowledgments, optimizing network usage and latency.

2

HDFS's write-once model simplifies consistency but requires special handling for append operations introduced later.

3

The NameNode's metadata is a single point of failure, so high-availability setups use standby NameNodes with shared storage.

When NOT to use

HDFS is not suitable for workloads requiring frequent random writes or low-latency updates. For such cases, distributed databases or object stores like Apache HBase or Amazon S3 are better alternatives.

Production Patterns

In production, HDFS is used with YARN to run batch jobs and Spark for fast data processing. Data is ingested via tools like Apache Flume or Kafka, stored in HDFS, and processed in parallel. Monitoring tools track DataNode health and replication status to maintain cluster reliability.

Connections

Distributed Databases

Both use data partitioning and replication to handle large data across many machines.

Understanding HDFS replication helps grasp how distributed databases ensure data availability and fault tolerance.

Content Delivery Networks (CDNs)

CDNs replicate content across servers worldwide, similar to HDFS replicating data blocks across nodes.

Knowing HDFS replication clarifies how CDNs improve access speed and reliability by storing multiple copies.

Human Memory Systems

Like HDFS stores data in blocks across nodes, human memory stores information in chunks distributed across brain regions.

This connection shows how distributed storage principles appear in both technology and biology, highlighting efficiency and fault tolerance.

Common Pitfalls

#1 Trying to modify an existing HDFS file after writing.

Wrong approach:

hdfs dfs -appendToFile newdata.txt /user/hadoop/existingfile.txt

Correct approach:

Write new data to a new file and replace or merge files as needed.

Root cause:

Misunderstanding HDFS's write-once-read-many design that disallows in-place file modifications.

#2 Reading data by repeatedly asking the NameNode for each block.

Wrong approach:

Client requests block location from NameNode for every block read operation.

Correct approach:

Client requests block locations once, then reads blocks directly from DataNodes.

Root cause:

Confusing metadata management with data transfer responsibilities.

#3 Ignoring replication factor and storing only one copy of data.

Wrong approach:

Setting replication factor to 1 in production clusters.

Correct approach:

Use default replication factor (usually 3) to ensure fault tolerance.

Root cause:

Underestimating the importance of data redundancy for reliability.

Key Takeaways

HDFS splits large files into blocks and stores them across many machines to handle big data efficiently.

The NameNode manages metadata while DataNodes store actual data blocks, separating concerns for scalability.

Write operations use a pipeline of DataNodes with acknowledgments to ensure data is safely replicated.

Read operations fetch block locations from the NameNode once, then read data directly from DataNodes for speed.

HDFS is designed for write-once-read-many workloads with built-in replication for fault tolerance and reliability.

Practice

1. What does the write operation in HDFS primarily do?

easy

A. Creates or overwrites files in the distributed file system

B. Reads data from files line by line

C. Deletes files from the system

D. Lists all files in a directory

Solution

1. Step 1: Understand the purpose of write operation in HDFS

   The write operation is used to create new files or overwrite existing files in HDFS.

2. Step 2: Differentiate write from read and other operations

   Reading fetches data, deleting removes files, listing shows files. Writing specifically saves data.

3. Final Answer:

   Creates or overwrites files in the distributed file system -> Option A

4. Quick Check:

   Write = Create or overwrite files [OK]

Hint: Write means save or replace file content [OK]

Common Mistakes:

• Confusing write with read operation
• Thinking write deletes files
• Assuming write lists files

2. Which of the following is the correct way to open a file for reading in HDFS using Python's pyarrow library?

easy

A. fs.open('/path/to/file', 'w')

B. fs.open('/path/to/file', 'r')

C. fs.read('/path/to/file')

D. fs.write('/path/to/file')

Solution

1. Step 1: Identify the mode for reading files

   In file operations, 'r' mode means open for reading, 'w' means writing.

2. Step 2: Match the correct function call for reading

   Using fs.open with 'r' opens the file for reading in pyarrow.

3. Final Answer:

   fs.open('/path/to/file', 'r') -> Option B

4. Quick Check:

   Open file with 'r' to read [OK]

Hint: Use 'r' mode to open files for reading [OK]

Common Mistakes:

• Using 'w' mode when reading
• Confusing open with read or write methods
• Forgetting to specify mode

3. Consider this Python code snippet using pyarrow to read from HDFS:

with fs.open('/data/sample.txt', 'r') as f:
    lines = f.readlines()
print(len(lines))

If the file contains 5 lines, what will be the output?

medium

A. 5

B. 0

C. Error: file not closed

D. 1

Solution

1. Step 1: Understand what readlines() returns

   The readlines() method reads all lines and returns a list of lines.

2. Step 2: Count the number of lines in the list

   If the file has 5 lines, len(lines) will be 5.

3. Final Answer:

   5 -> Option A

4. Quick Check:

   readlines() returns list of lines, length = 5 [OK]

Hint: readlines() returns list; length equals number of lines [OK]

Common Mistakes:

• Assuming readlines() returns a string
• Forgetting to close the file (but 'with' handles it)
• Confusing number of lines with characters

4. What is wrong with this HDFS write code snippet using pyarrow?

f = fs.open('/data/output.txt', 'w')
f.write('Hello World')
# Missing f.close()

medium

A. File is not opened in read mode

B. Write method does not accept strings

C. File path is invalid

D. File is not closed, risking data loss

Solution

1. Step 1: Check if file is properly closed after writing

   The code opens file and writes but does not close it, which can cause data loss.

2. Step 2: Understand importance of closing streams in HDFS

   Closing the file ensures data is flushed and saved correctly in HDFS.

3. Final Answer:

   File is not closed, risking data loss -> Option D

4. Quick Check:

   Always close files after writing [OK]

Hint: Always close files after writing to save data [OK]

Common Mistakes:

• Ignoring file close after write
• Assuming write accepts only bytes (it accepts strings too)
• Confusing read and write modes

5. You want to write a large dataset to HDFS and then read it back line by line for processing. Which sequence of operations is best practice?

hard

A. Open file in read mode, read lines; then open in write mode, write data

B. Open file in write mode, write data; open same file in read mode without closing write stream, read lines

C. Open file in write mode, write data, close file; then open in read mode, read lines, close file

D. Write data without opening file; then read lines without opening file

Solution

1. Step 1: Write data properly by opening and closing the file

   Open the file in write mode, write the data, then close the file to save it safely.

2. Step 2: Read data by opening file in read mode and closing after reading

   After writing, open the file in read mode, read line by line, then close the file to release resources.

3. Final Answer:

   Open file in write mode, write data, close file; then open in read mode, read lines, close file -> Option C

4. Quick Check:

   Write then close; read then close [OK]

Hint: Always close write before reading same file [OK]

Common Mistakes:

• Reading before writing data
• Not closing write stream before reading
• Trying to read or write without opening file