HDFS read and write operations in Hadoop - Mini Project: Build & Apply

HDFS read and write operations

📖 Scenario: You work in a company that stores large files on Hadoop Distributed File System (HDFS). You need to practice reading from and writing to HDFS using simple commands and scripts.

🎯 Goal: Learn how to write a file to HDFS, read it back, and display its contents using Hadoop commands.

📋 What You'll Learn

Create a local text file with specific content

Write the local file to HDFS

Read the file from HDFS

Display the file contents on the console

💡 Why This Matters

🌍 Real World

HDFS is used to store large data files across many computers. Reading and writing files to HDFS is a daily task for data engineers and scientists working with big data.

💼 Career

Knowing how to handle files in HDFS is essential for roles like Hadoop administrator, data engineer, and big data analyst.

1

Create a local text file

Create a local text file called example.txt with the exact content: Hello HDFS!

Hadoop

# Create a local file named example.txt with the text 'Hello HDFS!'
# Your code here

Hint

Use the echo command to write text into a file.

2

Write the file to HDFS

Use the Hadoop command hdfs dfs -put example.txt /user/yourusername/ to write the local file example.txt to your HDFS home directory.

Hadoop

echo 'Hello HDFS!' > example.txt
# Write example.txt to HDFS under /user/yourusername/
# Your code here

Hint

Use hdfs dfs -put to copy a local file to HDFS.

3

Read the file from HDFS

Use the Hadoop command hdfs dfs -cat /user/yourusername/example.txt to read and display the contents of the file from HDFS.

Hadoop

echo 'Hello HDFS!' > example.txt
hdfs dfs -put example.txt /user/yourusername/
# Read and display the file contents from HDFS
# Your code here

Hint

Use hdfs dfs -cat to read a file from HDFS and print it to the console.

4

Display the file contents

Run the full script to display the output. The output should be exactly Hello HDFS!

Hadoop

echo 'Hello HDFS!' > example.txt
hdfs dfs -put example.txt /user/yourusername/
hdfs dfs -cat /user/yourusername/example.txt
# Run the commands and observe the output

Hint

The output should show the text you wrote to the file.

Practice

1. What does the write operation in HDFS primarily do?

easy

A. Creates or overwrites files in the distributed file system

B. Reads data from files line by line

C. Deletes files from the system

D. Lists all files in a directory

Solution

1. Step 1: Understand the purpose of write operation in HDFS

   The write operation is used to create new files or overwrite existing files in HDFS.

2. Step 2: Differentiate write from read and other operations

   Reading fetches data, deleting removes files, listing shows files. Writing specifically saves data.

3. Final Answer:

   Creates or overwrites files in the distributed file system -> Option A

4. Quick Check:

   Write = Create or overwrite files [OK]

Hint: Write means save or replace file content [OK]

Common Mistakes:

• Confusing write with read operation
• Thinking write deletes files
• Assuming write lists files

2. Which of the following is the correct way to open a file for reading in HDFS using Python's pyarrow library?

easy

A. fs.open('/path/to/file', 'w')

B. fs.open('/path/to/file', 'r')

C. fs.read('/path/to/file')

D. fs.write('/path/to/file')

Solution

1. Step 1: Identify the mode for reading files

   In file operations, 'r' mode means open for reading, 'w' means writing.

2. Step 2: Match the correct function call for reading

   Using fs.open with 'r' opens the file for reading in pyarrow.

3. Final Answer:

   fs.open('/path/to/file', 'r') -> Option B

4. Quick Check:

   Open file with 'r' to read [OK]

Hint: Use 'r' mode to open files for reading [OK]

Common Mistakes:

• Using 'w' mode when reading
• Confusing open with read or write methods
• Forgetting to specify mode

3. Consider this Python code snippet using pyarrow to read from HDFS:

with fs.open('/data/sample.txt', 'r') as f:
    lines = f.readlines()
print(len(lines))

If the file contains 5 lines, what will be the output?

medium

A. 5

B. 0

C. Error: file not closed

D. 1

Solution

1. Step 1: Understand what readlines() returns

   The readlines() method reads all lines and returns a list of lines.

2. Step 2: Count the number of lines in the list

   If the file has 5 lines, len(lines) will be 5.

3. Final Answer:

   5 -> Option A

4. Quick Check:

   readlines() returns list of lines, length = 5 [OK]

Hint: readlines() returns list; length equals number of lines [OK]

Common Mistakes:

• Assuming readlines() returns a string
• Forgetting to close the file (but 'with' handles it)
• Confusing number of lines with characters

4. What is wrong with this HDFS write code snippet using pyarrow?

f = fs.open('/data/output.txt', 'w')
f.write('Hello World')
# Missing f.close()

medium

A. File is not opened in read mode

B. Write method does not accept strings

C. File path is invalid

D. File is not closed, risking data loss

Solution

1. Step 1: Check if file is properly closed after writing

   The code opens file and writes but does not close it, which can cause data loss.

2. Step 2: Understand importance of closing streams in HDFS

   Closing the file ensures data is flushed and saved correctly in HDFS.

3. Final Answer:

   File is not closed, risking data loss -> Option D

4. Quick Check:

   Always close files after writing [OK]

Hint: Always close files after writing to save data [OK]

Common Mistakes:

• Ignoring file close after write
• Assuming write accepts only bytes (it accepts strings too)
• Confusing read and write modes

5. You want to write a large dataset to HDFS and then read it back line by line for processing. Which sequence of operations is best practice?

hard

A. Open file in read mode, read lines; then open in write mode, write data

B. Open file in write mode, write data; open same file in read mode without closing write stream, read lines

C. Open file in write mode, write data, close file; then open in read mode, read lines, close file

D. Write data without opening file; then read lines without opening file

Solution

1. Step 1: Write data properly by opening and closing the file

   Open the file in write mode, write the data, then close the file to save it safely.

2. Step 2: Read data by opening file in read mode and closing after reading

   After writing, open the file in read mode, read line by line, then close the file to release resources.

3. Final Answer:

   Open file in write mode, write data, close file; then open in read mode, read lines, close file -> Option C

4. Quick Check:

   Write then close; read then close [OK]

Hint: Always close write before reading same file [OK]

Common Mistakes:

• Reading before writing data
• Not closing write stream before reading
• Trying to read or write without opening file