HDFS read and write operations in Hadoop - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is HDFS in Hadoop?

HDFS stands for Hadoop Distributed File System. It stores large data across many computers in a cluster, making data reliable and fast to access.

beginner

How does HDFS write operation work?

When writing data, HDFS splits the file into blocks and stores copies (replicas) of each block on different computers to keep data safe.

beginner

What happens during an HDFS read operation?

HDFS reads data by locating the blocks of the file and fetching them from the computers that store the replicas, usually choosing the closest one to speed up reading.

intermediate

Why does HDFS store multiple replicas of data blocks?

HDFS stores multiple copies of data blocks to protect against computer failures. If one copy is lost, others can be used to keep data safe and available.

intermediate

What role does the NameNode play in HDFS read and write operations?

The NameNode keeps track of where all data blocks are stored. It tells the system where to write new blocks and where to find blocks when reading files.

What does HDFS do when writing a large file?

A Stores the entire file on a single node

B Splits the file into blocks and stores replicas on different nodes

C Compresses the file and stores it on one node

D Sends the file to the client without storing

HDFS splits large files into blocks and stores multiple replicas on different nodes to ensure reliability and fault tolerance.

During a read operation, how does HDFS choose which replica to read?

A Chooses the closest replica to the client

B Randomly picks any replica

C Always reads from the first replica stored

D Reads from all replicas simultaneously

HDFS tries to read from the closest replica to reduce network delay and speed up data access.

What is the main function of the NameNode in HDFS?

A Stores actual data blocks

B Processes data computations

C Handles network communication

D Manages metadata and block locations

The NameNode manages metadata like file names and block locations but does not store the actual data blocks.

Why does HDFS replicate data blocks?

A To increase storage space

B To speed up writing only

C To improve data reliability and fault tolerance

D To reduce network traffic

Replication ensures data is safe even if some nodes fail, improving reliability and fault tolerance.

Which component in HDFS directs clients where to write or read data?

A NameNode

B ResourceManager

C JobTracker

D DataNode

The NameNode tells clients where data blocks are stored for reading or where to write new blocks.

Explain the process of writing a file to HDFS, including how data is split and stored.

Think about how HDFS breaks down big files and keeps copies safe.

Describe how HDFS reads a file and how it ensures fast and reliable access.

Consider how HDFS finds and retrieves data efficiently.

Practice

1. What does the write operation in HDFS primarily do?

easy

A. Creates or overwrites files in the distributed file system

B. Reads data from files line by line

C. Deletes files from the system

D. Lists all files in a directory

Solution

1. Step 1: Understand the purpose of write operation in HDFS

   The write operation is used to create new files or overwrite existing files in HDFS.

2. Step 2: Differentiate write from read and other operations

   Reading fetches data, deleting removes files, listing shows files. Writing specifically saves data.

3. Final Answer:

   Creates or overwrites files in the distributed file system -> Option A

4. Quick Check:

   Write = Create or overwrite files [OK]

Hint: Write means save or replace file content [OK]

Common Mistakes:

• Confusing write with read operation
• Thinking write deletes files
• Assuming write lists files

2. Which of the following is the correct way to open a file for reading in HDFS using Python's pyarrow library?

easy

A. fs.open('/path/to/file', 'w')

B. fs.open('/path/to/file', 'r')

C. fs.read('/path/to/file')

D. fs.write('/path/to/file')

Solution

1. Step 1: Identify the mode for reading files

   In file operations, 'r' mode means open for reading, 'w' means writing.

2. Step 2: Match the correct function call for reading

   Using fs.open with 'r' opens the file for reading in pyarrow.

3. Final Answer:

   fs.open('/path/to/file', 'r') -> Option B

4. Quick Check:

   Open file with 'r' to read [OK]

Hint: Use 'r' mode to open files for reading [OK]

Common Mistakes:

• Using 'w' mode when reading
• Confusing open with read or write methods
• Forgetting to specify mode

3. Consider this Python code snippet using pyarrow to read from HDFS:

with fs.open('/data/sample.txt', 'r') as f:
    lines = f.readlines()
print(len(lines))

If the file contains 5 lines, what will be the output?

medium

A. 5

B. 0

C. Error: file not closed

D. 1

Solution

1. Step 1: Understand what readlines() returns

   The readlines() method reads all lines and returns a list of lines.

2. Step 2: Count the number of lines in the list

   If the file has 5 lines, len(lines) will be 5.

3. Final Answer:

   5 -> Option A

4. Quick Check:

   readlines() returns list of lines, length = 5 [OK]

Hint: readlines() returns list; length equals number of lines [OK]

Common Mistakes:

• Assuming readlines() returns a string
• Forgetting to close the file (but 'with' handles it)
• Confusing number of lines with characters

4. What is wrong with this HDFS write code snippet using pyarrow?

f = fs.open('/data/output.txt', 'w')
f.write('Hello World')
# Missing f.close()

medium

A. File is not opened in read mode

B. Write method does not accept strings

C. File path is invalid

D. File is not closed, risking data loss

Solution

1. Step 1: Check if file is properly closed after writing

   The code opens file and writes but does not close it, which can cause data loss.

2. Step 2: Understand importance of closing streams in HDFS

   Closing the file ensures data is flushed and saved correctly in HDFS.

3. Final Answer:

   File is not closed, risking data loss -> Option D

4. Quick Check:

   Always close files after writing [OK]

Hint: Always close files after writing to save data [OK]

Common Mistakes:

• Ignoring file close after write
• Assuming write accepts only bytes (it accepts strings too)
• Confusing read and write modes

5. You want to write a large dataset to HDFS and then read it back line by line for processing. Which sequence of operations is best practice?

hard

A. Open file in read mode, read lines; then open in write mode, write data

B. Open file in write mode, write data; open same file in read mode without closing write stream, read lines

C. Open file in write mode, write data, close file; then open in read mode, read lines, close file

D. Write data without opening file; then read lines without opening file

Solution

1. Step 1: Write data properly by opening and closing the file

   Open the file in write mode, write the data, then close the file to save it safely.

2. Step 2: Read data by opening file in read mode and closing after reading

   After writing, open the file in read mode, read line by line, then close the file to release resources.

3. Final Answer:

   Open file in write mode, write data, close file; then open in read mode, read lines, close file -> Option C

4. Quick Check:

   Write then close; read then close [OK]

Hint: Always close write before reading same file [OK]

Common Mistakes:

• Reading before writing data
• Not closing write stream before reading
• Trying to read or write without opening file