HDFS read and write operations in Hadoop - Mini Project: Build & Apply

HDFS read and write operations

📖 Scenario: You work in a company that stores large files on Hadoop Distributed File System (HDFS). You need to practice reading from and writing to HDFS using simple commands and scripts.

🎯 Goal: Learn how to write a file to HDFS, read it back, and display its contents using Hadoop commands.

📋 What You'll Learn

Create a local text file with specific content

Write the local file to HDFS

Read the file from HDFS

Display the file contents on the console

💡 Why This Matters

🌍 Real World

HDFS is used to store large data files across many computers. Reading and writing files to HDFS is a daily task for data engineers and scientists working with big data.

💼 Career

Knowing how to handle files in HDFS is essential for roles like Hadoop administrator, data engineer, and big data analyst.

1

Create a local text file

Create a local text file called example.txt with the exact content: Hello HDFS!

Hadoop

# Create a local file named example.txt with the text 'Hello HDFS!'
# Your code here

Need a hint?

Use the echo command to write text into a file.

2

Write the file to HDFS

Use the Hadoop command hdfs dfs -put example.txt /user/yourusername/ to write the local file example.txt to your HDFS home directory.

Hadoop

echo 'Hello HDFS!' > example.txt
# Write example.txt to HDFS under /user/yourusername/
# Your code here

Need a hint?

Use hdfs dfs -put to copy a local file to HDFS.

3

Read the file from HDFS

Use the Hadoop command hdfs dfs -cat /user/yourusername/example.txt to read and display the contents of the file from HDFS.

Hadoop

echo 'Hello HDFS!' > example.txt
hdfs dfs -put example.txt /user/yourusername/
# Read and display the file contents from HDFS
# Your code here

Need a hint?

Use hdfs dfs -cat to read a file from HDFS and print it to the console.

4

Display the file contents

Run the full script to display the output. The output should be exactly Hello HDFS!

Hadoop

echo 'Hello HDFS!' > example.txt
hdfs dfs -put example.txt /user/yourusername/
hdfs dfs -cat /user/yourusername/example.txt
# Run the commands and observe the output

Need a hint?

The output should show the text you wrote to the file.