Working with large files efficiently in NumPy - Step-by-Step Execution

Concept Flow - Working with large files efficiently

Open large file

↓

Read chunk of data

↓

Process chunk

↓

Store or aggregate results

↓

More data?

Yes→Read next chunk

No↓

Close file and output final result

This flow shows reading a large file in small parts, processing each part, and combining results to avoid memory overload.

Execution Sample

NumPy

import numpy as np
chunk_size = 100000
sums = 0
with open('large_file.txt') as f:
    while True:
        chunk = []
        for _ in range(chunk_size):
            line = f.readline()
            if not line:
                break
            chunk.append(line.rstrip())
        if not chunk:
            break
        data = np.array([float(x) for x in chunk])
        sums += data.sum()

This code reads a large text file in chunks, converts each chunk to numbers, sums them, and accumulates the total sum.

Execution Table

Step	Action	Chunk Read	Data Array	Chunk Sum	Total Sum
1	Open file and read first chunk	[100000 lines]	array of 100000 floats	sum1	sum1
2	Read second chunk	[100000 lines]	array of 100000 floats	sum2	sum1 + sum2
3	Read third chunk	[100000 lines]	array of 100000 floats	sum3	sum1 + sum2 + sum3
4	Read last chunk (less than chunk_size)	[remaining lines]	array of remaining floats	sum_last	sum1 + sum2 + sum3 + sum_last
5	No more data, close file	[]	[]	0	final sum

💡 File fully read; no more chunks to process.

Variable Tracker

Variable	Start	After 1	After 2	After 3	After 4	Final
chunk	None	[100000 lines]	[100000 lines]	[100000 lines]	[remaining lines]	[]
data	None	array(100000 floats)	array(100000 floats)	array(100000 floats)	array(remaining floats)	None
sums	0	sum1	sum1+sum2	sum1+sum2+sum3	sum1+sum2+sum3+sum_last	final sum

Key Moments - 3 Insights

Why do we read the file in chunks instead of all at once?

Reading the whole file at once can use too much memory and crash the program. The execution_table shows reading fixed-size chunks to keep memory use low.

What happens if the last chunk is smaller than the chunk size?

The last chunk reads only the remaining lines. The execution_table row 4 shows this smaller chunk is still processed correctly.

How is the total sum updated during the loop?

After processing each chunk, its sum is added to the total sums variable. The variable_tracker shows sums increasing step by step.

Visual Quiz - 3 Questions

Look at the execution_table, what is the value of 'chunk' at Step 3?

A [remaining lines]

B [100000 lines]

C []

D None

💡 Hint

Refer to the 'Chunk Read' column at Step 3 in execution_table.

At which step does the file reading stop according to the execution_table?

A Step 2

B Step 4

C Step 5

D Step 3

💡 Hint

Look at the exit_note and Step 5 row in execution_table.

If chunk_size was doubled, how would the 'sums' variable change in variable_tracker?

A It would update fewer times with larger increments

B It would not change at all

C It would update more times with smaller increments

D It would reset to zero each time

💡 Hint

Consider how chunk size affects number of chunks and sums updates in variable_tracker.

Concept Snapshot

Working with large files efficiently:
- Read file in small chunks to save memory
- Process each chunk separately
- Accumulate results step-by-step
- Avoid loading entire file at once
- Use loops and chunk size control

Full Transcript

This lesson shows how to handle large files by reading them in small parts called chunks. We open the file, read a chunk of lines, convert them to numbers using numpy, sum them, and add to a total sum. We repeat until no data remains. This method prevents memory overload by not loading the whole file at once. The execution table traces each step: reading chunks, processing data arrays, summing, and updating totals. The variable tracker shows how variables like chunk, data, and sums change after each iteration. Key moments clarify why chunking is needed, how the last chunk works, and how sums accumulate. The quiz tests understanding of chunk content, stopping step, and effect of changing chunk size. This approach is essential for efficient data science with large files.

Practice

1. What is the main advantage of using np.memmap when working with large binary files?

easy

A. It allows accessing data on disk without loading the entire file into memory.

B. It automatically compresses the file to save disk space.

C. It converts binary files into text files for easier reading.

D. It loads the entire file into memory for faster processing.

Solution

1. Step 1: Understand np.memmap functionality

   np.memmap creates a memory-map to an array stored in a binary file on disk, allowing access without loading all data into RAM.

2. Step 2: Compare options with this behavior

   Only It allows accessing data on disk without loading the entire file into memory. correctly describes this behavior. Options B, C, and D describe unrelated or incorrect features.

3. Final Answer:

   It allows accessing data on disk without loading the entire file into memory. -> Option A

4. Quick Check:

   np.memmap = Access data on disk [OK]

Hint: Remember: memmap reads from disk, not full memory load [OK]

Common Mistakes:

• Thinking memmap loads entire file into memory
• Confusing memmap with file compression
• Assuming memmap converts file formats

2. Which of the following is the correct syntax to create a memory-mapped array from a binary file named data.bin with dtype float32 and shape (1000, 1000)?

easy

A. np.memmap('data.bin', dtype='float64', mode='r', shape=(1000, 1000))

B. np.memmap('data.bin', dtype='int32', mode='w', shape=(1000, 1000))

C. np.memmap('data.bin', dtype='float32', mode='r+', shape=(1000, 1000))

D. np.memmap('data.bin', dtype='float32', mode='rw', shape=(1000, 1000))

Solution

1. Step 1: Identify correct dtype and mode

   The question asks for dtype 'float32' and a mode that allows reading and writing, which is 'r+'.

2. Step 2: Check each option

   np.memmap('data.bin', dtype='float32', mode='r+', shape=(1000, 1000)) matches dtype 'float32' and mode 'r+'. np.memmap('data.bin', dtype='int32', mode='w', shape=(1000, 1000)) has wrong dtype 'int32' and mode 'w' (write only). np.memmap('data.bin', dtype='float64', mode='r', shape=(1000, 1000)) has wrong dtype 'float64' and mode 'r' (read only). np.memmap('data.bin', dtype='float32', mode='rw', shape=(1000, 1000)) uses invalid mode 'rw'.

3. Final Answer:

   np.memmap('data.bin', dtype='float32', mode='r+', shape=(1000, 1000)) -> Option C

4. Quick Check:

   Correct dtype and mode = np.memmap('data.bin', dtype='float32', mode='r+', shape=(1000, 1000)) [OK]

Hint: Use mode 'r+' for read/write memmap [OK]

Common Mistakes:

• Using wrong dtype for the file data
• Using invalid mode like 'rw'
• Confusing read-only 'r' with read/write 'r+'

3. Consider the following code snippet:

import numpy as np
filename = 'largefile.dat'
# Create memmap
mmap = np.memmap(filename, dtype='int32', mode='r', shape=(4, 4))
print(mmap[2, 3])

If the file contains a 4x4 array with values from 0 to 15 in row-major order, what will be the output?

medium

A. 12

B. 14

C. 15

D. 11

Solution

1. Step 1: Understand data layout

   The file stores values 0 to 15 in a 4x4 array in row-major order: [[0,1,2,3],[4,5,6,7],[8,9,10,11],[12,13,14,15]]

2. Step 2: Find value at position (2, 3)

   Row 2 (0-based) is [8,9,10,11]. Index 3 in this row is 11.

3. Final Answer:

   11 -> Option D

4. Quick Check:

   Value at (2,3) = 11 [OK]

Hint: Remember zero-based indexing for arrays [OK]

Common Mistakes:

• Confusing row and column indices
• Using 1-based indexing instead of 0-based
• Mixing up row-major and column-major order

4. You try to create a memmap with this code:

mmap = np.memmap('data.bin', dtype='float32', mode='r+', shape=(1000, 1000))

but get an error: ValueError: cannot mmap an empty file. What is the likely cause and how to fix it?

medium

A. The dtype 'float32' is invalid; use 'float64' instead.

B. The file 'data.bin' is empty; initialize it with correct size before memmap.

C. The mode 'r+' is read-only; use 'w+' to write.

D. The shape (1000, 1000) is too large; reduce it to (100, 100).

Solution

1. Step 1: Understand error cause

   The error means the file exists but has zero bytes, so memmap cannot map it with the given shape and dtype.

2. Step 2: Fix by initializing file size

   To fix, create or resize the file to hold the required data (1000*1000*4 bytes for float32) before memmap.

3. Final Answer:

   The file 'data.bin' is empty; initialize it with correct size before memmap. -> Option B

4. Quick Check:

   Empty file causes mmap error [OK]

Hint: Ensure file size matches array size before memmap [OK]

Common Mistakes:

• Changing dtype without fixing file size
• Using wrong mode without file content
• Reducing shape without reason

5. You have a very large text file with 1 billion numbers separated by spaces. You want to analyze the data using numpy but cannot load all at once. Which approach is best to process this file efficiently?

hard

A. Read the file in chunks, convert each chunk to numpy arrays, and process incrementally.

B. Use np.memmap directly on the text file to access numbers.

C. Read the entire file into memory as a string, then convert to numpy array.

D. Convert the text file to CSV and load with pandas without chunking.

Solution

1. Step 1: Understand file type and memory limits

   The file is a large text file, not binary. np.memmap works only with binary files, so Use np.memmap directly on the text file to access numbers. is invalid.

2. Step 2: Choose efficient reading method

   Reading entire file at once (Read the entire file into memory as a string, then convert to numpy array.) is memory-heavy. Converting to CSV and loading without chunking (Convert the text file to CSV and load with pandas without chunking.) also risks memory overload. Reading in chunks and processing incrementally (Read the file in chunks, convert each chunk to numpy arrays, and process incrementally.) is memory efficient and practical.

3. Final Answer:

   Read the file in chunks, convert each chunk to numpy arrays, and process incrementally. -> Option A

4. Quick Check:

   Chunk reading for large text files = Read the file in chunks, convert each chunk to numpy arrays, and process incrementally. [OK]

Hint: Process large text files in chunks, not all at once [OK]

Common Mistakes:

• Trying to memmap text files
• Loading entire large file into memory
• Ignoring memory limits when converting formats