HadoopHow-ToBeginner · 4 min read

How to Use Hadoop FS Commands: Syntax, Examples, and Tips

Use hadoop fs commands to interact with Hadoop Distributed File System (HDFS) for tasks like listing, copying, and deleting files. The basic syntax is hadoop fs -command [options] [path], where command specifies the operation such as -ls or -put.

📐

Syntax

The general syntax for Hadoop FS commands is:

hadoop fs: The base command to access HDFS.
-command: The operation you want to perform, like -ls to list files or -put to upload files.
[options]: Optional flags to modify the command behavior.
[path]: The HDFS path or local path involved in the operation.

bash

hadoop fs -command [options] [path]

💻

Example

This example shows how to list files in the root directory of HDFS and how to upload a local file to HDFS.

bash

hadoop fs -ls /
hadoop fs -put /local/path/file.txt /user/hadoop/

Output

Found 2 items -rw-r--r-- 3 hadoop supergroup 1234 2024-06-01 10:00 /file1.txt -rw-r--r-- 3 hadoop supergroup 5678 2024-06-01 10:05 /file2.txt

⚠️

Common Pitfalls

Common mistakes include:

Using local file paths instead of HDFS paths when required.
Forgetting to specify the full HDFS path, leading to errors or unexpected results.
Not having proper permissions to access or modify files in HDFS.
Confusing -put (upload) with -copyFromLocal which behaves similarly but has subtle differences.

Always check your paths and permissions before running commands.

bash

Wrong:
hadoop fs -ls file.txt

Right:
hadoop fs -ls /user/hadoop/file.txt

📊

Quick Reference

Command	Description
-ls [path]	List files and directories at the given HDFS path
-put	Upload a local file to HDFS
-get	Download a file from HDFS to local filesystem
-rm [path]	Remove a file or directory from HDFS
-mkdir [path]	Create a directory in HDFS
-cat [path]	Display contents of a file in HDFS

✅

Key Takeaways

Use hadoop fs -command [options] [path] to interact with HDFS files.

Always specify full HDFS paths to avoid confusion with local files.

Check permissions before running commands to prevent access errors.

Common commands include -ls, -put, -get, and -rm.

Refer to the quick reference table for common Hadoop FS commands.