HDFS read and write operations in Hadoop - Step-by-Step Execution

Concept Flow - HDFS read and write operations

Start Write Request

↓

Client contacts NameNode

↓

NameNode returns DataNode list

↓

Client writes data to DataNodes

↓

DataNodes replicate blocks

↓

Write Complete

↓

Start Read Request

↓

Client contacts NameNode

↓

NameNode returns DataNode locations

↓

Client reads data from DataNodes

↓

Read Complete

Shows the flow of how data is written to and read from HDFS, involving client, NameNode, and DataNodes.

Execution Sample

Hadoop

1. Client requests to write file
2. NameNode provides DataNode list
3. Client streams data to DataNodes
4. DataNodes replicate blocks
5. Client requests to read file
6. NameNode provides DataNode locations
7. Client reads data from DataNodes

This sequence shows the main steps for writing and reading files in HDFS.

Execution Table

Step	Operation	Actor	Action Detail	Result/Output
1	Write Request	Client	Client sends write request to NameNode	NameNode receives request
2	Block Allocation	NameNode	NameNode allocates blocks and returns DataNode list	Client gets DataNode list
3	Data Streaming	Client	Client streams data to first DataNode in pipeline	DataNode receives data block
4	Replication	DataNodes	DataNodes replicate data block to next DataNodes	Data replicated across DataNodes
5	Write Confirmation	DataNodes	DataNodes confirm write success to Client	Client confirms write complete
6	Read Request	Client	Client sends read request to NameNode	NameNode receives request
7	Block Location	NameNode	NameNode returns DataNode locations for blocks	Client gets DataNode locations
8	Data Reading	Client	Client reads data blocks from DataNodes	Data received by Client
9	Read Completion	Client	Client completes reading all blocks	Read operation complete

💡 All blocks written and replicated successfully; all blocks read completely.

Variable Tracker

Variable	Start	After Step 2	After Step 4	After Step 7	Final
Client Request	None	Write request sent	Streaming data ongoing	Read request sent	Read complete
NameNode Response	None	DataNode list sent	N/A	DataNode locations sent	N/A
DataNode State	Empty	Ready to receive	Data blocks replicated	Serving read requests	Idle

Key Moments - 3 Insights

Why does the client contact the NameNode before writing or reading data?

The client contacts the NameNode to get the list or locations of DataNodes that store or will store the data blocks, as shown in steps 2 and 7 in the execution_table.

How does data replication happen during write operations?

Data is streamed from the client to the first DataNode, which then replicates the data to other DataNodes in a pipeline, as detailed in steps 3 and 4.

Why does the client read data directly from DataNodes instead of the NameNode?

The NameNode only manages metadata and block locations; actual data is stored on DataNodes, so the client reads data directly from them as shown in step 8.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the client receive the list of DataNodes for writing?

A Step 3

B Step 6

C Step 2

D Step 7

💡 Hint

Check the 'NameNode' actions in the execution_table rows 2 and 7.

During which step do DataNodes replicate the data blocks?

A Step 3

B Step 4

C Step 5

D Step 8

💡 Hint

Look at the 'Replication' operation performed by DataNodes in the execution_table.

If the client reads data directly from the NameNode, what would change in the execution flow?

A NameNode would send data blocks in step 7

B DataNodes would not replicate data in step 4

C Client would skip contacting DataNodes in step 8

D Client would not send read request in step 6

💡 Hint

Consider the role of NameNode and DataNodes in the execution_table steps 7 and 8.

Concept Snapshot

HDFS Write:
- Client asks NameNode for DataNode list
- Client streams data to DataNodes
- DataNodes replicate blocks

HDFS Read:
- Client asks NameNode for DataNode locations
- Client reads data from DataNodes

NameNode manages metadata; DataNodes store actual data blocks.

Full Transcript

This visual execution trace shows how HDFS handles read and write operations. When writing, the client first contacts the NameNode to get the list of DataNodes to store data blocks. Then the client streams data to the first DataNode, which replicates the data to other DataNodes. After replication, the write is confirmed complete. For reading, the client asks the NameNode for the locations of the data blocks, then reads the data directly from the DataNodes. The NameNode only manages metadata and block locations, while DataNodes store and serve the actual data. This flow ensures efficient and reliable data storage and retrieval in HDFS.

Practice

1. What does the write operation in HDFS primarily do?

easy

A. Creates or overwrites files in the distributed file system

B. Reads data from files line by line

C. Deletes files from the system

D. Lists all files in a directory

Solution

1. Step 1: Understand the purpose of write operation in HDFS

   The write operation is used to create new files or overwrite existing files in HDFS.

2. Step 2: Differentiate write from read and other operations

   Reading fetches data, deleting removes files, listing shows files. Writing specifically saves data.

3. Final Answer:

   Creates or overwrites files in the distributed file system -> Option A

4. Quick Check:

   Write = Create or overwrite files [OK]

Hint: Write means save or replace file content [OK]

Common Mistakes:

• Confusing write with read operation
• Thinking write deletes files
• Assuming write lists files

2. Which of the following is the correct way to open a file for reading in HDFS using Python's pyarrow library?

easy

A. fs.open('/path/to/file', 'w')

B. fs.open('/path/to/file', 'r')

C. fs.read('/path/to/file')

D. fs.write('/path/to/file')

Solution

1. Step 1: Identify the mode for reading files

   In file operations, 'r' mode means open for reading, 'w' means writing.

2. Step 2: Match the correct function call for reading

   Using fs.open with 'r' opens the file for reading in pyarrow.

3. Final Answer:

   fs.open('/path/to/file', 'r') -> Option B

4. Quick Check:

   Open file with 'r' to read [OK]

Hint: Use 'r' mode to open files for reading [OK]

Common Mistakes:

• Using 'w' mode when reading
• Confusing open with read or write methods
• Forgetting to specify mode

3. Consider this Python code snippet using pyarrow to read from HDFS:

with fs.open('/data/sample.txt', 'r') as f:
    lines = f.readlines()
print(len(lines))

If the file contains 5 lines, what will be the output?

medium

A. 5

B. 0

C. Error: file not closed

D. 1

Solution

1. Step 1: Understand what readlines() returns

   The readlines() method reads all lines and returns a list of lines.

2. Step 2: Count the number of lines in the list

   If the file has 5 lines, len(lines) will be 5.

3. Final Answer:

   5 -> Option A

4. Quick Check:

   readlines() returns list of lines, length = 5 [OK]

Hint: readlines() returns list; length equals number of lines [OK]

Common Mistakes:

• Assuming readlines() returns a string
• Forgetting to close the file (but 'with' handles it)
• Confusing number of lines with characters

4. What is wrong with this HDFS write code snippet using pyarrow?

f = fs.open('/data/output.txt', 'w')
f.write('Hello World')
# Missing f.close()

medium

A. File is not opened in read mode

B. Write method does not accept strings

C. File path is invalid

D. File is not closed, risking data loss

Solution

1. Step 1: Check if file is properly closed after writing

   The code opens file and writes but does not close it, which can cause data loss.

2. Step 2: Understand importance of closing streams in HDFS

   Closing the file ensures data is flushed and saved correctly in HDFS.

3. Final Answer:

   File is not closed, risking data loss -> Option D

4. Quick Check:

   Always close files after writing [OK]

Hint: Always close files after writing to save data [OK]

Common Mistakes:

• Ignoring file close after write
• Assuming write accepts only bytes (it accepts strings too)
• Confusing read and write modes

5. You want to write a large dataset to HDFS and then read it back line by line for processing. Which sequence of operations is best practice?

hard

A. Open file in read mode, read lines; then open in write mode, write data

B. Open file in write mode, write data; open same file in read mode without closing write stream, read lines

C. Open file in write mode, write data, close file; then open in read mode, read lines, close file

D. Write data without opening file; then read lines without opening file

Solution

1. Step 1: Write data properly by opening and closing the file

   Open the file in write mode, write the data, then close the file to save it safely.

2. Step 2: Read data by opening file in read mode and closing after reading

   After writing, open the file in read mode, read line by line, then close the file to release resources.

3. Final Answer:

   Open file in write mode, write data, close file; then open in read mode, read lines, close file -> Option C

4. Quick Check:

   Write then close; read then close [OK]

Hint: Always close write before reading same file [OK]

Common Mistakes:

• Reading before writing data
• Not closing write stream before reading
• Trying to read or write without opening file