Memory-mapped arrays for large data in NumPy - Step-by-Step Execution

Concept Flow - Memory-mapped arrays for large data

Create or open large file

↓

Memory-map file as numpy array

↓

Access array data directly from disk

↓

Read or write parts of array

↓

Changes saved to disk automatically

↓

Close file

This flow shows how a large file is memory-mapped as a numpy array, allowing direct access to data on disk without loading all into memory.

Execution Sample

NumPy

import numpy as np
filename = 'large_data.dat'
# Create memmap array
arr = np.memmap(filename, dtype='float32', mode='w+', shape=(3,3))
arr[:] = np.arange(9).reshape(3,3)
arr.flush()

This code creates a 3x3 memory-mapped array, writes numbers 0 to 8, and saves changes to disk.

Execution Table

Step	Action	Array Content	Disk Write	Notes
1	Create memmap array with zeros	[[0. 0. 0.] [0. 0. 0.] [0. 0. 0.]]	No	Array backed by file, initially zeros
2	Assign values 0 to 8 reshaped	[[0. 1. 2.] [3. 4. 5.] [6. 7. 8.]]	No	Values set in memmap array in memory
3	Flush changes to disk	[[0. 1. 2.] [3. 4. 5.] [6. 7. 8.]]	Yes	Data written to file on disk
4	Close memmap object	[[0. 1. 2.] [3. 4. 5.] [6. 7. 8.]]	No	File closed, data saved

💡 All data written to disk and memmap closed

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4
arr	None	[[0. 0. 0.] [0. 0. 0.] [0. 0. 0.]]	[[0. 1. 2.] [3. 4. 5.] [6. 7. 8.]]	[[0. 1. 2.] [3. 4. 5.] [6. 7. 8.]]	Closed

Key Moments - 3 Insights

Why doesn't assigning values to the memmap array immediately write to disk?

Because changes are first made in memory for efficiency. The flush() method explicitly writes changes to disk, as shown in step 3 of the execution_table.

What happens if the file backing the memmap does not exist when mode='r+' is used?

An error occurs because mode='r+' requires the file to exist. You must create the file first or use mode='w+' to create a new file, as implied in step 1.

How does memmap help with large data compared to normal numpy arrays?

Memmap accesses data directly from disk without loading all into RAM, so it can handle data larger than memory. This is shown by the array content being backed by a file in step 1.

Visual Quiz - 3 Questions

Look at the execution_table at step 2, what is the value of arr[1,2]?

A 5.0

B 3.0

C 8.0

D 0.0

💡 Hint

Check the array content column at step 2 in execution_table

At which step are the changes saved to disk?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Look at the Disk Write column in execution_table

If we skip calling flush(), what would happen to the data on disk?

A Data on disk updates automatically

B Data on disk remains unchanged

C File is deleted

D Program crashes

💡 Hint

Refer to key_moments about when data is written to disk

Concept Snapshot

Memory-mapped arrays let numpy access large data files on disk as arrays.
Use np.memmap(filename, dtype, mode, shape) to create.
Changes are made in memory and saved with flush().
Good for data too big for RAM.
Access like normal numpy arrays.

Full Transcript

Memory-mapped arrays allow numpy to work with large data files by mapping them directly to arrays without loading all data into memory. You create a memmap array by specifying a filename, data type, mode, and shape. Initially, the array reflects the file content or zeros if new. You can assign values to the array in memory. These changes are not immediately saved to disk until you call flush(). Closing the memmap ensures data is saved and file is closed. This method is useful for very large datasets that do not fit in RAM, enabling efficient read and write access directly on disk.

Practice

1. What is the main advantage of using numpy.memmap for large data files?

easy

A. It allows working with data without loading the entire file into memory.

B. It compresses the data to save disk space.

C. It automatically cleans up unused files.

D. It speeds up CPU calculations by using GPU.

Solution

1. Step 1: Understand memory usage with large files

   Loading large files fully into RAM can cause memory errors or slowdowns.

2. Step 2: Role of numpy.memmap

   Memory-mapped arrays let you access parts of the file on disk as if they were in memory, avoiding full loading.

3. Final Answer:

   It allows working with data without loading the entire file into memory. -> Option A

4. Quick Check:

   Memory-mapping = Partial loading [OK]

Hint: Memory-mapping means partial file loading, not full load [OK]

Common Mistakes:

• Thinking memmap compresses data
• Assuming memmap speeds up CPU/GPU
• Believing memmap auto-deletes files

2. Which of the following is the correct way to create a memory-mapped array from a file named data.dat with shape (100, 100) and dtype float64?

easy

A. np.memmap('data.dat', shape=(100, 100), dtype='int32', mode='w+')

B. np.memmap('data.dat', dtype='float64', mode='r+', shape=(100, 100))

C. np.memmap('data.dat', dtype='float64', shape=(100, 100))

D. np.memmap('data.dat', mode='r', shape=(100, 100), dtype='float32')

Solution

1. Step 1: Check required parameters for memmap

   Must specify filename, dtype, mode, and shape for memmap creation.

2. Step 2: Validate each option

   np.memmap('data.dat', dtype='float64', mode='r+', shape=(100, 100)) correctly uses dtype='float64', mode='r+', and shape=(100, 100). Others have wrong dtype or missing mode.

3. Final Answer:

   np.memmap('data.dat', dtype='float64', mode='r+', shape=(100, 100)) -> Option B

4. Quick Check:

   Correct dtype, mode, shape = np.memmap('data.dat', dtype='float64', mode='r+', shape=(100, 100)) [OK]

Hint: Always specify dtype, mode, and shape explicitly [OK]

Common Mistakes:

• Omitting mode parameter
• Using wrong dtype
• Swapping dtype and shape order

3. Given the code below, what will be the output of print(mm[0, 0])?

import numpy as np
filename = 'test.dat'
mm = np.memmap(filename, dtype='int32', mode='w+', shape=(2, 2))
mm[0, 0] = 10
mm.flush()
mm2 = np.memmap(filename, dtype='int32', mode='r', shape=(2, 2))
print(mm2[0, 0])

medium

A. Raises an error

B. 0

C. 10

D. Uninitialized garbage value

Solution

1. Step 1: Write value and flush to disk

   Value 10 is assigned to mm[0,0] and flushed, saving to file.

2. Step 2: Read from file with new memmap

   mm2 reads the same file in read mode, so mm2[0,0] is 10.

3. Final Answer:

   10 -> Option C

4. Quick Check:

   Flush saves data, read memmap shows 10 [OK]

Hint: Flush saves changes; reading memmap shows updated value [OK]

Common Mistakes:

• Forgetting to flush before reading
• Expecting zero without initialization
• Confusing write and read modes

4. What is the error in the following code snippet?

import numpy as np
filename = 'data.dat'
mm = np.memmap(filename, dtype='float64', mode='r+', shape=(3, 3))
mm[0, 0] = 5.0
mm.flush()

Assume data.dat does not exist before running this code.

medium

A. No error; code runs fine.

B. TypeError due to wrong dtype.

C. ValueError because shape is incorrect.

D. FileNotFoundError because mode 'r+' requires existing file.

Solution

1. Step 1: Understand mode 'r+'

   Mode 'r+' opens file for reading and writing but requires the file to exist.

2. Step 2: Check file existence

   Since 'data.dat' does not exist, opening with 'r+' raises FileNotFoundError.

3. Final Answer:

   FileNotFoundError because mode 'r+' requires existing file. -> Option D

4. Quick Check:

   Mode 'r+' needs existing file [OK]

Hint: Mode 'r+' needs existing file; else FileNotFoundError [OK]

Common Mistakes:

• Assuming 'r+' creates file
• Confusing 'r+' with 'w+'
• Ignoring file existence requirement

5. You have a large dataset stored in a binary file large_data.dat with shape (10000, 10000) and dtype float32. You want to increment all values by 1 without loading the entire file into memory. Which approach using numpy.memmap is best?

hard

A. Create a memmap with mode 'r+', loop over rows, increment each row, flush after each.

B. Load entire file with np.fromfile, add 1, then save back.

C. Open memmap with mode 'w+', increment only first 100 elements, flush once.

D. Use memmap with mode 'r', add 1 to the array, then flush.

Solution

1. Step 1: Avoid loading entire file

   Loading full 10000x10000 array uses too much RAM, so avoid np.fromfile full load.

2. Step 2: Use memmap with mode 'r+' and process in chunks

   Memmap with 'r+' allows read/write access. Looping over rows increments data without full load. Flushing saves changes.

3. Step 3: Evaluate other options

   Load entire file with np.fromfile, add 1, then save back. loads full file (bad for memory). Open memmap with mode 'w+', increment only first 100 elements, flush once. only increments part of data. Use memmap with mode 'r', add 1 to the array, then flush. uses 'r' mode (read-only), so cannot modify.

4. Final Answer:

   Create a memmap with mode 'r+', loop over rows, increment each row, flush after each. -> Option A

5. Quick Check:

   Memmap + chunk update + flush = Create a memmap with mode 'r+', loop over rows, increment each row, flush after each. [OK]

Hint: Use memmap with 'r+' and update in chunks to save memory [OK]

Common Mistakes:

• Loading entire file into memory
• Using read-only mode for writing
• Modifying only part of data