HDFS read and write operations in Hadoop - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to write data to HDFS using the Hadoop FileSystem API.

Hadoop

fs = FileSystem.get(conf)
output = fs.create(Path("/user/hadoop/[1]"))
output.write(bytes("Hello HDFS", "utf-8"))
output.close()

Drag options to blanks, or click blank then click option'

Ainput.txt

Bdata.csv

Creadme.md

Doutput.txt

Attempts:

3 left

💡 Hint

Common Mistakes

Using an input file name when writing data.

Forgetting to close the output stream.

The file name 'output.txt' is commonly used for writing output data to HDFS.

2fill in blank

medium

Complete the code to read data from HDFS using the Hadoop FileSystem API.

Hadoop

fs = FileSystem.get(conf)
input = fs.open(Path("/user/hadoop/[1]"))
data = input.readLine()
input.close()

Drag options to blanks, or click blank then click option'

Aoutput.txt

Blog.txt

Cinput.txt

Dconfig.xml

Attempts:

3 left

💡 Hint

Common Mistakes

Using an output file name when reading data.

Not closing the input stream after reading.

The file 'input.txt' is typically used as the source file to read data from HDFS.

3fill in blank

hard

Fix the error in the code to properly write a string to HDFS.

Hadoop

fs = FileSystem.get(conf)
output = fs.create(Path("/user/hadoop/output.txt"))
output.write([1]("Hello HDFS", "utf-8"))
output.close()

Drag options to blanks, or click blank then click option'

Astr.encode

Bbytes

Cencode

DtoBytes

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'str.encode' which is not a standalone function.

Using 'encode' without calling it on the string.

Using a non-existent function 'toBytes'.

The 'bytes' function converts a string to bytes, which is required for writing to HDFS output stream.

4fill in blank

hard

Fill both blanks to create a dictionary comprehension that maps file names to their sizes in HDFS for files larger than 1000 bytes.

Hadoop

file_sizes = {file.getName(): file.getLen() for file in fs.listStatus(Path("/user/hadoop")) if file.getLen() [1] [2]

Drag options to blanks, or click blank then click option'

A>

B1000

C<

D500

Attempts:

3 left

💡 Hint

Common Mistakes

Using '<' instead of '>' for filtering larger files.

Using a smaller size threshold like 500 instead of 1000.

We want files larger than 1000 bytes, so the condition is file.getLen() > 1000.

5fill in blank

hard

Fill all three blanks to create a dictionary comprehension that maps file names to their modification times for files modified after 2023-01-01.

Hadoop

from datetime import datetime
threshold = datetime(2023, 1, 1)
file_mod_times = {file.getName(): datetime.fromtimestamp(file.getModificationTime() / 1000) for file in fs.listStatus(Path("/user/hadoop")) if datetime.fromtimestamp(file.getModificationTime() / 1000) [1] threshold and file.getName() [2] [3]

Drag options to blanks, or click blank then click option'

A>

B!=

C".tmp"

D==

Attempts:

3 left

💡 Hint

Common Mistakes

Using '<' instead of '>' for date comparison.

Using '==' instead of '!=' to exclude files.

Using wrong string for file exclusion.

We filter files modified after the threshold (>) and exclude temporary files by checking name not equal (!=) to '.tmp'.

Practice

1. What does the write operation in HDFS primarily do?

easy

A. Creates or overwrites files in the distributed file system

B. Reads data from files line by line

C. Deletes files from the system

D. Lists all files in a directory

Solution

1. Step 1: Understand the purpose of write operation in HDFS

   The write operation is used to create new files or overwrite existing files in HDFS.

2. Step 2: Differentiate write from read and other operations

   Reading fetches data, deleting removes files, listing shows files. Writing specifically saves data.

3. Final Answer:

   Creates or overwrites files in the distributed file system -> Option A

4. Quick Check:

   Write = Create or overwrite files [OK]

Hint: Write means save or replace file content [OK]

Common Mistakes:

• Confusing write with read operation
• Thinking write deletes files
• Assuming write lists files

2. Which of the following is the correct way to open a file for reading in HDFS using Python's pyarrow library?

easy

A. fs.open('/path/to/file', 'w')

B. fs.open('/path/to/file', 'r')

C. fs.read('/path/to/file')

D. fs.write('/path/to/file')

Solution

1. Step 1: Identify the mode for reading files

   In file operations, 'r' mode means open for reading, 'w' means writing.

2. Step 2: Match the correct function call for reading

   Using fs.open with 'r' opens the file for reading in pyarrow.

3. Final Answer:

   fs.open('/path/to/file', 'r') -> Option B

4. Quick Check:

   Open file with 'r' to read [OK]

Hint: Use 'r' mode to open files for reading [OK]

Common Mistakes:

• Using 'w' mode when reading
• Confusing open with read or write methods
• Forgetting to specify mode

3. Consider this Python code snippet using pyarrow to read from HDFS:

with fs.open('/data/sample.txt', 'r') as f:
    lines = f.readlines()
print(len(lines))

If the file contains 5 lines, what will be the output?

medium

A. 5

B. 0

C. Error: file not closed

D. 1

Solution

1. Step 1: Understand what readlines() returns

   The readlines() method reads all lines and returns a list of lines.

2. Step 2: Count the number of lines in the list

   If the file has 5 lines, len(lines) will be 5.

3. Final Answer:

   5 -> Option A

4. Quick Check:

   readlines() returns list of lines, length = 5 [OK]

Hint: readlines() returns list; length equals number of lines [OK]

Common Mistakes:

• Assuming readlines() returns a string
• Forgetting to close the file (but 'with' handles it)
• Confusing number of lines with characters

4. What is wrong with this HDFS write code snippet using pyarrow?

f = fs.open('/data/output.txt', 'w')
f.write('Hello World')
# Missing f.close()

medium

A. File is not opened in read mode

B. Write method does not accept strings

C. File path is invalid

D. File is not closed, risking data loss

Solution

1. Step 1: Check if file is properly closed after writing

   The code opens file and writes but does not close it, which can cause data loss.

2. Step 2: Understand importance of closing streams in HDFS

   Closing the file ensures data is flushed and saved correctly in HDFS.

3. Final Answer:

   File is not closed, risking data loss -> Option D

4. Quick Check:

   Always close files after writing [OK]

Hint: Always close files after writing to save data [OK]

Common Mistakes:

• Ignoring file close after write
• Assuming write accepts only bytes (it accepts strings too)
• Confusing read and write modes

5. You want to write a large dataset to HDFS and then read it back line by line for processing. Which sequence of operations is best practice?

hard

A. Open file in read mode, read lines; then open in write mode, write data

B. Open file in write mode, write data; open same file in read mode without closing write stream, read lines

C. Open file in write mode, write data, close file; then open in read mode, read lines, close file

D. Write data without opening file; then read lines without opening file

Solution

1. Step 1: Write data properly by opening and closing the file

   Open the file in write mode, write the data, then close the file to save it safely.

2. Step 2: Read data by opening file in read mode and closing after reading

   After writing, open the file in read mode, read line by line, then close the file to release resources.

3. Final Answer:

   Open file in write mode, write data, close file; then open in read mode, read lines, close file -> Option C

4. Quick Check:

   Write then close; read then close [OK]

Hint: Always close write before reading same file [OK]

Common Mistakes:

• Reading before writing data
• Not closing write stream before reading
• Trying to read or write without opening file