HDFS read and write operations in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this HDFS write command?

Consider the following Hadoop shell command to write a local file to HDFS:

hdfs dfs -put /local/path/file.txt /user/hadoop/

What will be the result if the file already exists in the destination path?

A The command fails with an error saying the file already exists.

B The command overwrites the existing file without warning.

C The command appends the new file content to the existing file.

D The command creates a duplicate file with a different name.

💡 Hint

Think about how HDFS handles file overwrites by default.

By default, the 'hdfs dfs -put' command does not overwrite existing files and will fail if the destination file exists.

❓ data_output

intermediate

What data is read by this HDFS command?

Given the command:

hdfs dfs -cat /user/hadoop/data/sample.txt

What will be the output?

A The entire content of the file sample.txt printed to the console.

B Only the first 10 lines of sample.txt.

C Metadata information about sample.txt but not the content.

D The command will copy sample.txt to the local file system.

💡 Hint

The '-cat' option is similar to the Unix 'cat' command.

The 'hdfs dfs -cat' command outputs the full content of the specified file to the console.

🔧 Debug

advanced

Why does this HDFS read command fail?

You run:

hdfs dfs -text /user/hadoop/data/sample.gz

But get an error:

gzip: stdin: not in gzip format

What is the most likely cause?

A The file path is incorrect and points to a directory.

B The HDFS cluster is down and cannot read files.

C The command requires root privileges to read compressed files.

D The file sample.gz is not actually compressed with gzip format.

💡 Hint

Check the file format and compression type.

The error indicates the file is not in gzip format despite the .gz extension, so the command fails to decompress.

🚀 Application

advanced

How to efficiently write large data to HDFS in a Spark job?

You have a Spark job that generates a large DataFrame. You want to save it to HDFS as a single compressed file. Which approach is best?

A Write the DataFrame as multiple files and then merge them manually in HDFS.

B Use DataFrame's write method with coalesce(1) and compression option.

C Save the DataFrame without compression to speed up the write.

D Write the DataFrame to local disk first, then upload to HDFS.

💡 Hint

Think about controlling the number of output files and compression in Spark.

Using coalesce(1) reduces output to one file, and compression reduces storage size efficiently.

🧠 Conceptual

expert

What happens internally during an HDFS read operation?

When a client reads a file from HDFS, which sequence best describes the internal process?

A Client requests data from DataNodes, which coordinate with NameNode to send blocks sequentially.

B Client reads the entire file from the NameNode, which streams data from DataNodes.

C Client contacts NameNode for block locations, then reads blocks directly from DataNodes in parallel.

D Client downloads the file from a single DataNode chosen by the NameNode.

💡 Hint

Recall the roles of NameNode and DataNodes in HDFS architecture.

The NameNode provides metadata and block locations; the client reads blocks directly from multiple DataNodes for efficiency.

Practice

1. What does the write operation in HDFS primarily do?

easy

A. Creates or overwrites files in the distributed file system

B. Reads data from files line by line

C. Deletes files from the system

D. Lists all files in a directory

Solution

1. Step 1: Understand the purpose of write operation in HDFS

   The write operation is used to create new files or overwrite existing files in HDFS.

2. Step 2: Differentiate write from read and other operations

   Reading fetches data, deleting removes files, listing shows files. Writing specifically saves data.

3. Final Answer:

   Creates or overwrites files in the distributed file system -> Option A

4. Quick Check:

   Write = Create or overwrite files [OK]

Hint: Write means save or replace file content [OK]

Common Mistakes:

• Confusing write with read operation
• Thinking write deletes files
• Assuming write lists files

2. Which of the following is the correct way to open a file for reading in HDFS using Python's pyarrow library?

easy

A. fs.open('/path/to/file', 'w')

B. fs.open('/path/to/file', 'r')

C. fs.read('/path/to/file')

D. fs.write('/path/to/file')

Solution

1. Step 1: Identify the mode for reading files

   In file operations, 'r' mode means open for reading, 'w' means writing.

2. Step 2: Match the correct function call for reading

   Using fs.open with 'r' opens the file for reading in pyarrow.

3. Final Answer:

   fs.open('/path/to/file', 'r') -> Option B

4. Quick Check:

   Open file with 'r' to read [OK]

Hint: Use 'r' mode to open files for reading [OK]

Common Mistakes:

• Using 'w' mode when reading
• Confusing open with read or write methods
• Forgetting to specify mode

3. Consider this Python code snippet using pyarrow to read from HDFS:

with fs.open('/data/sample.txt', 'r') as f:
    lines = f.readlines()
print(len(lines))

If the file contains 5 lines, what will be the output?

medium

A. 5

B. 0

C. Error: file not closed

D. 1

Solution

1. Step 1: Understand what readlines() returns

   The readlines() method reads all lines and returns a list of lines.

2. Step 2: Count the number of lines in the list

   If the file has 5 lines, len(lines) will be 5.

3. Final Answer:

   5 -> Option A

4. Quick Check:

   readlines() returns list of lines, length = 5 [OK]

Hint: readlines() returns list; length equals number of lines [OK]

Common Mistakes:

• Assuming readlines() returns a string
• Forgetting to close the file (but 'with' handles it)
• Confusing number of lines with characters

4. What is wrong with this HDFS write code snippet using pyarrow?

f = fs.open('/data/output.txt', 'w')
f.write('Hello World')
# Missing f.close()

medium

A. File is not opened in read mode

B. Write method does not accept strings

C. File path is invalid

D. File is not closed, risking data loss

Solution

1. Step 1: Check if file is properly closed after writing

   The code opens file and writes but does not close it, which can cause data loss.

2. Step 2: Understand importance of closing streams in HDFS

   Closing the file ensures data is flushed and saved correctly in HDFS.

3. Final Answer:

   File is not closed, risking data loss -> Option D

4. Quick Check:

   Always close files after writing [OK]

Hint: Always close files after writing to save data [OK]

Common Mistakes:

• Ignoring file close after write
• Assuming write accepts only bytes (it accepts strings too)
• Confusing read and write modes

5. You want to write a large dataset to HDFS and then read it back line by line for processing. Which sequence of operations is best practice?

hard

A. Open file in read mode, read lines; then open in write mode, write data

B. Open file in write mode, write data; open same file in read mode without closing write stream, read lines

C. Open file in write mode, write data, close file; then open in read mode, read lines, close file

D. Write data without opening file; then read lines without opening file

Solution

1. Step 1: Write data properly by opening and closing the file

   Open the file in write mode, write the data, then close the file to save it safely.

2. Step 2: Read data by opening file in read mode and closing after reading

   After writing, open the file in read mode, read line by line, then close the file to release resources.

3. Final Answer:

   Open file in write mode, write data, close file; then open in read mode, read lines, close file -> Option C

4. Quick Check:

   Write then close; read then close [OK]

Hint: Always close write before reading same file [OK]

Common Mistakes:

• Reading before writing data
• Not closing write stream before reading
• Trying to read or write without opening file