HDFS read and write operations in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this HDFS write command?

Consider the following Hadoop shell command to write a local file to HDFS:

hdfs dfs -put /local/path/file.txt /user/hadoop/

What will be the result if the file already exists in the destination path?

A The command fails with an error saying the file already exists.

B The command overwrites the existing file without warning.

C The command appends the new file content to the existing file.

D The command creates a duplicate file with a different name.

💡 Hint

Think about how HDFS handles file overwrites by default.

By default, the 'hdfs dfs -put' command does not overwrite existing files and will fail if the destination file exists.

❓ data_output

intermediate

What data is read by this HDFS command?

Given the command:

hdfs dfs -cat /user/hadoop/data/sample.txt

What will be the output?

A The entire content of the file sample.txt printed to the console.

B Only the first 10 lines of sample.txt.

C Metadata information about sample.txt but not the content.

D The command will copy sample.txt to the local file system.

💡 Hint

The '-cat' option is similar to the Unix 'cat' command.

The 'hdfs dfs -cat' command outputs the full content of the specified file to the console.

🔧 Debug

advanced

Why does this HDFS read command fail?

You run:

hdfs dfs -text /user/hadoop/data/sample.gz

But get an error:

gzip: stdin: not in gzip format

What is the most likely cause?

A The file path is incorrect and points to a directory.

B The HDFS cluster is down and cannot read files.

C The command requires root privileges to read compressed files.

D The file sample.gz is not actually compressed with gzip format.

💡 Hint

Check the file format and compression type.

The error indicates the file is not in gzip format despite the .gz extension, so the command fails to decompress.

🚀 Application

advanced

How to efficiently write large data to HDFS in a Spark job?

You have a Spark job that generates a large DataFrame. You want to save it to HDFS as a single compressed file. Which approach is best?

A Write the DataFrame as multiple files and then merge them manually in HDFS.

B Use DataFrame's write method with coalesce(1) and compression option.

C Save the DataFrame without compression to speed up the write.

D Write the DataFrame to local disk first, then upload to HDFS.

💡 Hint

Think about controlling the number of output files and compression in Spark.

Using coalesce(1) reduces output to one file, and compression reduces storage size efficiently.

🧠 Conceptual

expert

What happens internally during an HDFS read operation?

When a client reads a file from HDFS, which sequence best describes the internal process?

A Client requests data from DataNodes, which coordinate with NameNode to send blocks sequentially.

B Client reads the entire file from the NameNode, which streams data from DataNodes.

C Client contacts NameNode for block locations, then reads blocks directly from DataNodes in parallel.

D Client downloads the file from a single DataNode chosen by the NameNode.

💡 Hint

Recall the roles of NameNode and DataNodes in HDFS architecture.

The NameNode provides metadata and block locations; the client reads blocks directly from multiple DataNodes for efficiency.