HDFS command line interface in Hadoop

Introduction

HDFS command line interface lets you manage files in Hadoop's storage system easily using simple commands.

You want to upload files from your computer to Hadoop storage.

You need to check what files are stored in Hadoop.

You want to delete or move files inside Hadoop storage.

You want to see the content of a file stored in Hadoop.

You want to check the space used by files in Hadoop.

Syntax

Hadoop

hdfs dfs -command [options] [path]

# Example commands:
hdfs dfs -ls /path/to/directory
hdfs dfs -put localfile.txt /hdfs/path/
hdfs dfs -cat /hdfs/path/file.txt
hdfs dfs -rm /hdfs/path/file.txt

All commands start with hdfs dfs followed by the operation.

Paths starting with / refer to locations inside HDFS.

Examples

Lists all files and folders inside the /user/hadoop directory in HDFS.

Hadoop

hdfs dfs -ls /user/hadoop

Uploads a file named localfile.txt from your computer to the HDFS directory /user/hadoop/.

Hadoop

hdfs dfs -put localfile.txt /user/hadoop/

Shows the content of file.txt stored in HDFS on the screen.

Hadoop

hdfs dfs -cat /user/hadoop/file.txt

Deletes file.txt from the HDFS directory /user/hadoop/.

Hadoop

hdfs dfs -rm /user/hadoop/file.txt

Sample Program

This script shows how to list files, upload a file, check its content, and delete it using HDFS CLI.

Hadoop

# This is a shell script to demonstrate HDFS CLI commands

# Step 1: List files in root directory (likely empty or default files)
hdfs dfs -ls /

# Step 2: Upload a local file to HDFS
# Assume localfile.txt exists in current directory
hdfs dfs -put localfile.txt /user/hadoop/

# Step 3: List files in /user/hadoop to confirm upload
hdfs dfs -ls /user/hadoop

# Step 4: Show content of uploaded file
hdfs dfs -cat /user/hadoop/localfile.txt

# Step 5: Remove the file from HDFS
hdfs dfs -rm /user/hadoop/localfile.txt

# Step 6: List files again to confirm removal
hdfs dfs -ls /user/hadoop

Important Notes

Time complexity: Commands run quickly but depend on network and cluster size.

Space complexity: Commands themselves use little memory; storage depends on file size.

Common mistake: Forgetting to specify full HDFS path or mixing local and HDFS paths.

Use HDFS CLI for quick file management; use Hadoop APIs for complex data processing.

Summary

HDFS CLI helps manage files in Hadoop storage with simple commands.

Common commands include -ls, -put, -cat, and -rm.

Always use full HDFS paths and check file presence after operations.