HDFS command line interface in Hadoop - Step-by-Step Execution

Concept Flow - HDFS command line interface

Start HDFS CLI

↓

Enter Command

↓

Parse Command

↓

Execute Command on HDFS

↓

Return Output or Error

↓

Prompt for Next Command or Exit

The HDFS CLI starts, waits for user commands, parses and executes them on HDFS, then shows results or errors, and repeats.

Execution Sample

Hadoop

hdfs dfs -ls /user/hadoop
hdfs dfs -mkdir /user/hadoop/newdir
hdfs dfs -put localfile.txt /user/hadoop/newdir
hdfs dfs -cat /user/hadoop/newdir/localfile.txt

These commands list files, create a directory, upload a file, and display file contents in HDFS.

Execution Table

Step	Command	Action	Result	Output
1	hdfs dfs -ls /user/hadoop	List directory contents	Success	Found files: file1.txt, file2.txt
2	hdfs dfs -mkdir /user/hadoop/newdir	Create directory	Success	Directory /user/hadoop/newdir created
3	hdfs dfs -put localfile.txt /user/hadoop/newdir	Upload local file to HDFS	Success	File localfile.txt uploaded
4	hdfs dfs -cat /user/hadoop/newdir/localfile.txt	Display file contents	Success	Hello Hadoop!
5	hdfs dfs -rm /user/hadoop/newdir/localfile.txt	Remove file	Success	File deleted
6	hdfs dfs -ls /user/hadoop/newdir	List directory contents	Success	No files found
7	exit	Exit CLI	End	Session terminated

💡 User typed 'exit' to end the HDFS CLI session

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6	Final
/user/hadoop directory contents	file1.txt, file2.txt	file1.txt, file2.txt	file1.txt, file2.txt, newdir/	file1.txt, file2.txt, newdir/	file1.txt, file2.txt, newdir/	file1.txt, file2.txt, newdir/	file1.txt, file2.txt, newdir/	file1.txt, file2.txt, newdir/
/user/hadoop/newdir contents	N/A	N/A	empty	localfile.txt	localfile.txt	empty	empty	empty
localfile.txt in HDFS	N/A	N/A	N/A	exists	exists	deleted	deleted	deleted

Key Moments - 3 Insights

Why does 'hdfs dfs -ls' show 'newdir/' after creating the directory?

Because after step 2, the directory 'newdir' is created inside /user/hadoop, so listing shows it as a folder with a trailing slash (see execution_table step 2).

What happens if you try to 'cat' a file that does not exist?

The command will return an error saying file not found. In our trace, step 4 succeeds because the file exists, but if missing, it would show an error instead.

Why does the directory become empty after deleting the file?

Because after step 5, the only file in 'newdir' is removed, so listing the directory in step 6 shows no files (see execution_table steps 5 and 6).

Visual Quiz - 3 Questions

Look at the execution table, what files are listed in /user/hadoop after step 2?

A file1.txt, file2.txt

B newdir/ only

C file1.txt, file2.txt, newdir/

D No files

💡 Hint

Check the 'Output' column at step 2 in the execution_table.

At which step is the file 'localfile.txt' deleted from HDFS?

A Step 4

B Step 5

C Step 3

D Step 6

💡 Hint

Look for the 'Remove file' action in the execution_table.

If you run 'hdfs dfs -ls /user/hadoop/newdir' after step 5, what will you see?

A No files found

B Error: Directory not found

C localfile.txt

D newdir/

💡 Hint

Check the output of step 6 in the execution_table.

Concept Snapshot

HDFS CLI lets you interact with Hadoop files.
Use 'hdfs dfs -ls <path>' to list files.
Use 'hdfs dfs -mkdir <dir>' to create directories.
Use 'hdfs dfs -put <local> <hdfs>' to upload files.
Use 'hdfs dfs -cat <file>' to view contents.
Use 'hdfs dfs -rm <file>' to delete files.

Full Transcript

The HDFS command line interface (CLI) starts and waits for user commands. When you type a command like 'hdfs dfs -ls /user/hadoop', it lists files in that directory. Creating a directory with 'hdfs dfs -mkdir' adds a new folder visible in listings. Uploading files with 'hdfs dfs -put' copies local files to HDFS. Viewing file contents uses 'hdfs dfs -cat'. Deleting files removes them from HDFS. The CLI shows success or error messages after each command. You can exit the CLI by typing 'exit'. This trace showed commands listing files, creating directories, uploading, viewing, and deleting files step-by-step.

Practice

1. What does the HDFS command hdfs dfs -ls /user/hadoop do?

easy

A. Uploads local files to /user/hadoop in HDFS

B. Deletes the directory /user/hadoop from HDFS

C. Lists all files and directories in the HDFS path /user/hadoop

D. Displays the content of files in /user/hadoop

Solution

1. Step 1: Understand the command structure

   The command hdfs dfs -ls is used to list files and directories in HDFS.

2. Step 2: Analyze the given path

   The path /user/hadoop specifies the directory to list.

3. Final Answer:

   Lists all files and directories in the HDFS path /user/hadoop -> Option C

4. Quick Check:

   hdfs dfs -ls lists files [OK]

Hint: Remember: '-ls' always lists files in given HDFS path [OK]

Common Mistakes:

• Confusing '-ls' with '-rm' which deletes files
• Thinking '-ls' uploads files
• Assuming it shows file content

2. Which of the following is the correct syntax to upload a local file data.txt to HDFS directory /user/hadoop?

easy

A. hdfs dfs -put /user/hadoop data.txt

B. hdfs dfs -copyFromLocal /user/hadoop data.txt

C. hdfs dfs -get data.txt /user/hadoop

D. hdfs dfs -put data.txt /user/hadoop

Solution

1. Step 1: Identify the correct command for uploading

   -put uploads local files to HDFS, and the syntax is hdfs dfs -put <local_path> <hdfs_path>.

2. Step 2: Match the syntax with options

   hdfs dfs -put data.txt /user/hadoop correctly places data.txt as local file and /user/hadoop as HDFS destination.

3. Final Answer:

   hdfs dfs -put data.txt /user/hadoop -> Option D

4. Quick Check:

   -put local_file hdfs_path is correct [OK]

Hint: Remember: local file first, then HDFS path in '-put' [OK]

Common Mistakes:

• Swapping local and HDFS paths
• Using '-get' which downloads files
• Using '-copyFromLocal' with wrong argument order

3. What will be the output of the command hdfs dfs -cat /user/hadoop/sample.txt if the file contains:

Hello Hadoop
Welcome to HDFS

medium

A. Lists the file details of sample.txt

B. Prints the content of sample.txt to the console

C. Deletes sample.txt from HDFS

D. Uploads sample.txt to HDFS

Solution

1. Step 1: Understand the '-cat' command

   hdfs dfs -cat prints the content of a file in HDFS to the console.

2. Step 2: Analyze the file content

   The file sample.txt contains two lines: 'Hello Hadoop' and 'Welcome to HDFS'.

3. Final Answer:

   Prints the content of sample.txt to the console -> Option B

4. Quick Check:

   -cat shows file content [OK]

Hint: Use '-cat' to display file content in HDFS [OK]

Common Mistakes:

• Confusing '-cat' with '-ls' which lists files
• Thinking it deletes files
• Assuming it uploads files

4. You run the command hdfs dfs -rm /user/hadoop/data.csv but get an error saying file not found. What is the most likely cause?

medium

A. The file /user/hadoop/data.csv does not exist in HDFS

B. You used the wrong command; '-rm' cannot delete files

C. You need to use '-put' to delete files

D. The HDFS service is down

Solution

1. Step 1: Understand the error message

   The error 'file not found' means the specified file does not exist at the given path.

2. Step 2: Check command correctness

   -rm is the correct command to delete files in HDFS, so the error is not due to wrong command.

3. Final Answer:

   The file /user/hadoop/data.csv does not exist in HDFS -> Option A

4. Quick Check:

   File missing causes 'file not found' error [OK]

Hint: Check file existence before deleting with '-rm' [OK]

Common Mistakes:

• Assuming '-rm' is wrong command
• Trying to delete with '-put'
• Ignoring file path correctness

5. You want to copy all files from local directory /local/data/ to HDFS directory /user/hadoop/data/ and then verify the files are uploaded. Which sequence of commands is correct?

hard

A. hdfs dfs -put /local/data/* /user/hadoop/data/ then hdfs dfs -ls /user/hadoop/data/

B. hdfs dfs -get /local/data/* /user/hadoop/data/ then hdfs dfs -cat /user/hadoop/data/

C. hdfs dfs -rm /local/data/* then hdfs dfs -ls /user/hadoop/data/

D. hdfs dfs -copyToLocal /local/data/* /user/hadoop/data/ then hdfs dfs -rm /user/hadoop/data/

Solution

1. Step 1: Choose the correct upload command

   -put uploads local files to HDFS. Using /local/data/* uploads all files from local directory.

2. Step 2: Verify files in HDFS

   hdfs dfs -ls /user/hadoop/data/ lists files in the HDFS directory to confirm upload.

3. Final Answer:

   hdfs dfs -put /local/data/* /user/hadoop/data/ then hdfs dfs -ls /user/hadoop/data/ -> Option A

4. Quick Check:

   Upload with '-put' then list with '-ls' [OK]

Hint: Upload with '-put' then check with '-ls' [OK]

Common Mistakes:

• Using '-get' which downloads files
• Deleting local files before upload
• Using '-copyToLocal' which copies from HDFS to local