Handling large files efficiently in Python

Introduction

Large files can be too big to load all at once in memory. Handling them efficiently means reading or writing them bit by bit to save memory and keep programs fast.

When reading a huge log file to find specific information without loading the whole file.

When processing large data files like CSVs or text files line by line.

When copying or moving big files without using too much memory.

When streaming data from a file to another place, like uploading or downloading.

When you want to avoid your program crashing because it runs out of memory.

Syntax

Python

with open('filename.txt', 'r') as file:
    for line in file:
        # process each line here

The with statement safely opens and closes the file.

Reading line by line uses little memory, good for big files.

Examples

This reads and prints each line one by one, removing extra spaces.

Python

with open('bigfile.txt', 'r') as file:
    for line in file:
        print(line.strip())

This reads the file in small chunks of 1024 bytes, useful for binary files.

Python

with open('bigfile.txt', 'rb') as file:
    chunk = file.read(1024)
    while chunk:
        # process chunk
        chunk = file.read(1024)

This writes a million lines to a file efficiently without loading all lines in memory.

Python

with open('output.txt', 'w') as file:
    for i in range(1000000):
        file.write(f'Line {i}\n')

Sample Program

This program creates a small file with 5 lines, then counts the lines by reading the file line by line without loading it all at once.

Python

def count_lines(filename):
    count = 0
    with open(filename, 'r') as file:
        for line in file:
            count += 1
    return count

filename = 'sample.txt'
with open(filename, 'w') as f:
    for i in range(5):
        f.write(f'Line number {i+1}\n')

lines = count_lines(filename)
print(f'The file {filename} has {lines} lines.')

Important Notes

Always use with to open files to avoid forgetting to close them.

Reading files line by line or in chunks helps keep memory use low.

For very large files, avoid reading the whole file into a list or string at once.

Summary

Use with open() to safely open and close files.

Read large files line by line or in small chunks to save memory.

Write to files incrementally instead of building big strings in memory.