Handling large files efficiently in Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Reading a large file line by line

What is the output of this code snippet when reading a large file line by line?

Python

count = 0
with open('large_file.txt', 'r') as f:
    for line in f:
        count += 1
print(count)

A Prints the total number of characters in the file

B Prints the first line of the file

C Prints the total number of lines in the file

D Raises a FileNotFoundError if the file does not exist

💡 Hint

Think about what the loop is counting.

The code reads the file line by line and increments count for each line, so it prints the total number of lines.

❓ Predict Output

intermediate

Using read() vs readlines() for large files

What will happen if you use readlines() on a very large file compared to using a for loop over the file object?

Python

with open('large_file.txt', 'r') as f:
    lines = f.readlines()
print(len(lines))

A Reads the entire file into memory at once, which can cause high memory usage

B Reads the file line by line efficiently without high memory usage

C Raises a SyntaxError

D Prints the first line only

💡 Hint

Consider how readlines() works internally.

readlines() loads all lines into a list in memory, which can be inefficient for very large files.

🔧 Debug

advanced

Fixing memory error when processing large file

This code tries to read a large file and process each line, but it causes a MemoryError. Which option fixes the problem?

Python

with open('large_file.txt', 'r') as f:
    data = f.read()
    for line in data.split('\n'):
        process(line)

A Increase system memory

B Use f.readlines() instead of f.read()

C Use f.read(1024) to read 1024 bytes only

D Replace f.read() with a for loop: for line in f: process(line)

💡 Hint

Think about how to avoid loading the whole file at once.

Reading the file line by line with a for loop avoids loading the entire file into memory, preventing MemoryError.

🧠 Conceptual

advanced

Why use buffered reading for large files?

Why is buffered reading important when handling large files in Python?

A It converts the file to binary format

B It reduces the number of system calls and improves performance

C It encrypts the file contents automatically

D It loads the entire file into memory for faster access

💡 Hint

Think about how reading in chunks affects system resources.

Buffered reading reads data in chunks, reducing system calls and improving efficiency when processing large files.

❓ Predict Output

expert

Output of generator-based file processing

What is the output of this code snippet?

Python

def read_chunks(file_path, chunk_size=4):
    with open(file_path, 'r') as f:
        while chunk := f.read(chunk_size):
            yield chunk

result = []
for part in read_chunks('test.txt'):
    result.append(part)
print(result)

A A list of strings, each string is a chunk of 4 characters from the file

B A list of lines from the file

C A single string with the entire file content

D Raises a SyntaxError due to walrus operator

💡 Hint

Look at how the file is read in chunks and yielded.

The function reads the file in 4-character chunks and yields each chunk, so the result is a list of these chunks.

Practice

1.

Which method is best to read a very large text file without using too much memory?

with open('file.txt', 'r') as f:

easy

A. Convert the file to a list using list(f) immediately

B. Read the entire file at once using f.read()

C. Read the file line by line using a loop like for line in f:

D. Use f.readlines() to get all lines at once

Solution

1. Step 1: Understand memory usage when reading files

   Reading the entire file at once loads all content into memory, which is bad for large files.

2. Step 2: Use line-by-line reading to save memory

   Using for line in f: reads one line at a time, keeping memory low.

3. Final Answer:

   Read the file line by line using a loop like for line in f: -> Option C

4. Quick Check:

   Line-by-line reading = low memory use [OK]

Hint: Read files line-by-line to save memory with large files [OK]

Common Mistakes:

• Using f.read() loads whole file into memory
• Using f.readlines() loads all lines at once
• Converting file to list loads entire file

2.

Which of the following is the correct syntax to open a file for writing and ensure it closes automatically?

easy

A. f = open('file.txt', 'w')

B. with open('file.txt', 'w') as f:

C. open('file.txt', 'w')

D. file = open('file.txt', 'r')

Solution

1. Step 1: Identify syntax for safe file handling

   The with statement opens the file and ensures it closes automatically after the block.

2. Step 2: Check mode and variable assignment

   Using with open('file.txt', 'w') as f: opens for writing and assigns to f.

3. Final Answer:

   with open('file.txt', 'w') as f: -> Option B

4. Quick Check:

   Use with open() for safe file handling [OK]

Hint: Use with open() to auto-close files safely [OK]

Common Mistakes:

• Forgetting to close file after open()
• Using wrong mode like 'r' for writing
• Not assigning file object to a variable

3.

What will be the output of this code snippet when reading a large file in chunks?

with open('largefile.txt', 'r') as f:
    chunk = f.read(5)
    print(chunk)
    chunk = f.read(5)
    print(chunk)

medium

A. Prints first 5 characters, then next 5 characters of the file

B. Prints the entire file twice

C. Prints only the first 5 characters twice

D. Raises an error because read() needs no arguments

Solution

1. Step 1: Understand read(size) behavior

   Calling f.read(5) reads 5 characters from the current file position.

2. Step 2: Reading twice moves file pointer forward

   First read gets chars 1-5, second read gets chars 6-10.

3. Final Answer:

   Prints first 5 characters, then next 5 characters of the file -> Option A

4. Quick Check:

   read(5) reads 5 chars sequentially [OK]

Hint: read(n) reads next n characters sequentially [OK]

Common Mistakes:

• Thinking read() reads whole file always
• Assuming read(5) resets file pointer
• Believing read() without args is invalid

4.

Find the error in this code that tries to write lines to a file efficiently:

lines = ['line1\n', 'line2\n', 'line3\n']
file = open('output.txt', 'w')
for line in lines:
    file.write(line)
file.close()

medium

A. Using with open() is better to ensure file closes

B. The file should be opened in read mode 'r'

C. The loop should use readlines() instead of lines

D. The file is not closed properly

Solution

1. Step 1: Check file handling safety

   Opening file without with risks leaving it open if error occurs before close().

2. Step 2: Use with open() for automatic closing

   Replacing with with open('output.txt', 'w') as file: ensures file closes safely.

3. Final Answer:

   Using with open() is better to ensure file closes -> Option A

4. Quick Check:

   Use with open() to auto-close files [OK]

Hint: Always use with open() to avoid forgetting file.close() [OK]

Common Mistakes:

• Forgetting to close file on exceptions
• Opening file in wrong mode
• Misunderstanding readlines() vs list variable

5.

You need to process a huge log file and write only lines containing the word 'ERROR' to a new file. Which approach is best to handle this efficiently?

hard

A. Read entire file into memory, filter lines, then write all at once

B. Use readlines() to get all lines, then write filtered lines

C. Open output file in read mode and append lines

D. Read file line by line, write matching lines immediately to output file

Solution

1. Step 1: Avoid loading entire file into memory

   Reading whole file at once uses too much memory for huge files.

2. Step 2: Process line by line and write incrementally

   Reading each line and writing matching lines immediately saves memory and is efficient.

3. Final Answer:

   Read file line by line, write matching lines immediately to output file -> Option D

4. Quick Check:

   Line-by-line processing + incremental write = efficient [OK]

Hint: Filter and write lines one by one to save memory [OK]

Common Mistakes:

• Loading entire file into memory
• Using wrong file mode for output
• Appending to output file opened in read mode