HDFS read and write operations in Hadoop - Step-by-Step Execution

Concept Flow - HDFS read and write operations

Start Write Request

↓

Client contacts NameNode

↓

NameNode returns DataNode list

↓

Client writes data to DataNodes

↓

DataNodes replicate blocks

↓

Write Complete

↓

Start Read Request

↓

Client contacts NameNode

↓

NameNode returns DataNode locations

↓

Client reads data from DataNodes

↓

Read Complete

Shows the flow of how data is written to and read from HDFS, involving client, NameNode, and DataNodes.

Execution Sample

Hadoop

1. Client requests to write file
2. NameNode provides DataNode list
3. Client streams data to DataNodes
4. DataNodes replicate blocks
5. Client requests to read file
6. NameNode provides DataNode locations
7. Client reads data from DataNodes

This sequence shows the main steps for writing and reading files in HDFS.

Execution Table

Step	Operation	Actor	Action Detail	Result/Output
1	Write Request	Client	Client sends write request to NameNode	NameNode receives request
2	Block Allocation	NameNode	NameNode allocates blocks and returns DataNode list	Client gets DataNode list
3	Data Streaming	Client	Client streams data to first DataNode in pipeline	DataNode receives data block
4	Replication	DataNodes	DataNodes replicate data block to next DataNodes	Data replicated across DataNodes
5	Write Confirmation	DataNodes	DataNodes confirm write success to Client	Client confirms write complete
6	Read Request	Client	Client sends read request to NameNode	NameNode receives request
7	Block Location	NameNode	NameNode returns DataNode locations for blocks	Client gets DataNode locations
8	Data Reading	Client	Client reads data blocks from DataNodes	Data received by Client
9	Read Completion	Client	Client completes reading all blocks	Read operation complete

💡 All blocks written and replicated successfully; all blocks read completely.

Variable Tracker

Variable	Start	After Step 2	After Step 4	After Step 7	Final
Client Request	None	Write request sent	Streaming data ongoing	Read request sent	Read complete
NameNode Response	None	DataNode list sent	N/A	DataNode locations sent	N/A
DataNode State	Empty	Ready to receive	Data blocks replicated	Serving read requests	Idle

Key Moments - 3 Insights

Why does the client contact the NameNode before writing or reading data?

The client contacts the NameNode to get the list or locations of DataNodes that store or will store the data blocks, as shown in steps 2 and 7 in the execution_table.

How does data replication happen during write operations?

Data is streamed from the client to the first DataNode, which then replicates the data to other DataNodes in a pipeline, as detailed in steps 3 and 4.

Why does the client read data directly from DataNodes instead of the NameNode?

The NameNode only manages metadata and block locations; actual data is stored on DataNodes, so the client reads data directly from them as shown in step 8.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the client receive the list of DataNodes for writing?

A Step 3

B Step 6

C Step 2

D Step 7

💡 Hint

Check the 'NameNode' actions in the execution_table rows 2 and 7.

During which step do DataNodes replicate the data blocks?

A Step 3

B Step 4

C Step 5

D Step 8

💡 Hint

Look at the 'Replication' operation performed by DataNodes in the execution_table.

If the client reads data directly from the NameNode, what would change in the execution flow?

A NameNode would send data blocks in step 7

B DataNodes would not replicate data in step 4

C Client would skip contacting DataNodes in step 8

D Client would not send read request in step 6

💡 Hint

Consider the role of NameNode and DataNodes in the execution_table steps 7 and 8.

Concept Snapshot

HDFS Write:
- Client asks NameNode for DataNode list
- Client streams data to DataNodes
- DataNodes replicate blocks

HDFS Read:
- Client asks NameNode for DataNode locations
- Client reads data from DataNodes

NameNode manages metadata; DataNodes store actual data blocks.

Full Transcript

This visual execution trace shows how HDFS handles read and write operations. When writing, the client first contacts the NameNode to get the list of DataNodes to store data blocks. Then the client streams data to the first DataNode, which replicates the data to other DataNodes. After replication, the write is confirmed complete. For reading, the client asks the NameNode for the locations of the data blocks, then reads the data directly from the DataNodes. The NameNode only manages metadata and block locations, while DataNodes store and serve the actual data. This flow ensures efficient and reliable data storage and retrieval in HDFS.