HDFS read and write operations in Hadoop

Introduction

We use HDFS read and write operations to store and access big data files across many computers easily and safely.

When you want to save large data files from your program to a distributed storage system.

When you need to read big data files stored in HDFS for analysis or processing.

When you want to move data between your local computer and the Hadoop cluster.

When you want to check the contents of a file stored in HDFS.

When you want to append new data to an existing file in HDFS.

Syntax

Hadoop

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStream;

public class HDFSReadWrite {
    public static void main(String[] args) throws Exception {
        Configuration configuration = new Configuration();
        FileSystem hdfs = FileSystem.get(configuration);

        // Write operation
        Path writePath = new Path("/user/hadoop/example.txt");
        try (OutputStream outputStream = hdfs.create(writePath)) {
            String data = "Hello HDFS!";
            outputStream.write(data.getBytes());
        }

        // Read operation
        Path readPath = new Path("/user/hadoop/example.txt");
        try (BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(hdfs.open(readPath)))) {
            String line;
            while ((line = bufferedReader.readLine()) != null) {
                System.out.println(line);
            }
        }
    }
}

The FileSystem object connects to HDFS and lets you read or write files.

Use hdfs.create(path) to write a new file or overwrite an existing one.

Examples

This creates a new file and writes data to it.

Hadoop

// Writing to HDFS when file does not exist
Path newFilePath = new Path("/user/hadoop/newfile.txt");
try (OutputStream outputStream = hdfs.create(newFilePath)) {
    outputStream.write("Data for new file".getBytes());
}

This reads the first line from the existing file.

Hadoop

// Reading from HDFS when file exists
Path existingFilePath = new Path("/user/hadoop/newfile.txt");
try (BufferedReader reader = new BufferedReader(new InputStreamReader(hdfs.open(existingFilePath)))) {
    String line = reader.readLine();
    System.out.println(line);
}

If the file is empty, reading returns null immediately.

Hadoop

// Reading from HDFS when file is empty
Path emptyFilePath = new Path("/user/hadoop/emptyfile.txt");
try (BufferedReader reader = new BufferedReader(new InputStreamReader(hdfs.open(emptyFilePath)))) {
    String line = reader.readLine();
    System.out.println(line); // prints null
}

This overwrites the existing file with new data.

Hadoop

// Writing to HDFS overwrites existing file
Path overwritePath = new Path("/user/hadoop/newfile.txt");
try (OutputStream outputStream = hdfs.create(overwritePath)) {
    outputStream.write("Overwritten data".getBytes());
}

Sample Program

This program writes a text string to a file in HDFS and then reads it back to print on the screen.

Hadoop

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStream;

public class HDFSReadWriteExample {
    public static void main(String[] args) throws Exception {
        Configuration configuration = new Configuration();
        FileSystem hdfs = FileSystem.get(configuration);

        Path filePath = new Path("/user/hadoop/sample.txt");

        System.out.println("Writing data to HDFS file...");
        try (OutputStream outputStream = hdfs.create(filePath)) {
            String dataToWrite = "Welcome to HDFS read and write operations!";
            outputStream.write(dataToWrite.getBytes());
        }

        System.out.println("Reading data from HDFS file...");
        try (BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(hdfs.open(filePath)))) {
            String line;
            while ((line = bufferedReader.readLine()) != null) {
                System.out.println(line);
            }
        }
    }
}

Important Notes

Time complexity: Writing and reading depend on file size and network speed.

Space complexity: Uses memory for buffers during read/write.

Common mistake: Forgetting to close streams can cause data loss or errors.

Use write operation to save or overwrite files; use read operation to access stored data.

Summary

HDFS read and write let you store and access big data files across many machines.

Writing creates or overwrites files; reading fetches file contents line by line.

Always close streams to avoid errors and data loss.

Practice

1. What does the write operation in HDFS primarily do?

easy

A. Creates or overwrites files in the distributed file system

B. Reads data from files line by line

C. Deletes files from the system

D. Lists all files in a directory

Solution

1. Step 1: Understand the purpose of write operation in HDFS

   The write operation is used to create new files or overwrite existing files in HDFS.

2. Step 2: Differentiate write from read and other operations

   Reading fetches data, deleting removes files, listing shows files. Writing specifically saves data.

3. Final Answer:

   Creates or overwrites files in the distributed file system -> Option A

4. Quick Check:

   Write = Create or overwrite files [OK]

Hint: Write means save or replace file content [OK]

Common Mistakes:

• Confusing write with read operation
• Thinking write deletes files
• Assuming write lists files

2. Which of the following is the correct way to open a file for reading in HDFS using Python's pyarrow library?

easy

A. fs.open('/path/to/file', 'w')

B. fs.open('/path/to/file', 'r')

C. fs.read('/path/to/file')

D. fs.write('/path/to/file')

Solution

1. Step 1: Identify the mode for reading files

   In file operations, 'r' mode means open for reading, 'w' means writing.

2. Step 2: Match the correct function call for reading

   Using fs.open with 'r' opens the file for reading in pyarrow.

3. Final Answer:

   fs.open('/path/to/file', 'r') -> Option B

4. Quick Check:

   Open file with 'r' to read [OK]

Hint: Use 'r' mode to open files for reading [OK]

Common Mistakes:

• Using 'w' mode when reading
• Confusing open with read or write methods
• Forgetting to specify mode

3. Consider this Python code snippet using pyarrow to read from HDFS:

with fs.open('/data/sample.txt', 'r') as f:
    lines = f.readlines()
print(len(lines))

If the file contains 5 lines, what will be the output?

medium

A. 5

B. 0

C. Error: file not closed

D. 1

Solution

1. Step 1: Understand what readlines() returns

   The readlines() method reads all lines and returns a list of lines.

2. Step 2: Count the number of lines in the list

   If the file has 5 lines, len(lines) will be 5.

3. Final Answer:

   5 -> Option A

4. Quick Check:

   readlines() returns list of lines, length = 5 [OK]

Hint: readlines() returns list; length equals number of lines [OK]

Common Mistakes:

• Assuming readlines() returns a string
• Forgetting to close the file (but 'with' handles it)
• Confusing number of lines with characters

4. What is wrong with this HDFS write code snippet using pyarrow?

f = fs.open('/data/output.txt', 'w')
f.write('Hello World')
# Missing f.close()

medium

A. File is not opened in read mode

B. Write method does not accept strings

C. File path is invalid

D. File is not closed, risking data loss

Solution

1. Step 1: Check if file is properly closed after writing

   The code opens file and writes but does not close it, which can cause data loss.

2. Step 2: Understand importance of closing streams in HDFS

   Closing the file ensures data is flushed and saved correctly in HDFS.

3. Final Answer:

   File is not closed, risking data loss -> Option D

4. Quick Check:

   Always close files after writing [OK]

Hint: Always close files after writing to save data [OK]

Common Mistakes:

• Ignoring file close after write
• Assuming write accepts only bytes (it accepts strings too)
• Confusing read and write modes

5. You want to write a large dataset to HDFS and then read it back line by line for processing. Which sequence of operations is best practice?

hard

A. Open file in read mode, read lines; then open in write mode, write data

B. Open file in write mode, write data; open same file in read mode without closing write stream, read lines

C. Open file in write mode, write data, close file; then open in read mode, read lines, close file

D. Write data without opening file; then read lines without opening file

Solution

1. Step 1: Write data properly by opening and closing the file

   Open the file in write mode, write the data, then close the file to save it safely.

2. Step 2: Read data by opening file in read mode and closing after reading

   After writing, open the file in read mode, read line by line, then close the file to release resources.

3. Final Answer:

   Open file in write mode, write data, close file; then open in read mode, read lines, close file -> Option C

4. Quick Check:

   Write then close; read then close [OK]

Hint: Always close write before reading same file [OK]

Common Mistakes:

• Reading before writing data
• Not closing write stream before reading
• Trying to read or write without opening file