Memory-mapped files with np.memmap in NumPy - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of np.memmap array modification

What is the output of the following code snippet?

NumPy

import numpy as np
filename = 'data.dat'
# Create a memmap file with 5 integers
arr = np.memmap(filename, dtype='int32', mode='w+', shape=(5,))
arr[:] = np.arange(5)
arr.flush()
# Reopen in read mode
arr2 = np.memmap(filename, dtype='int32', mode='r', shape=(5,))
print(arr2[2])

A 2

B 0

C 3

D Raises a ValueError

💡 Hint

Remember that np.memmap writes data to disk and reopening in read mode reads the saved data.

The memmap file is created and filled with values 0 to 4. Reopening in read mode reads the saved data, so arr2[2] is 2.

❓ data_output

intermediate

Shape of np.memmap array after reshaping

Given this code, what is the shape of the memmap array after reshaping?

NumPy

import numpy as np
filename = 'data2.dat'
arr = np.memmap(filename, dtype='float64', mode='w+', shape=(12,))
arr[:] = np.arange(12)
arr.flush()
arr2 = np.memmap(filename, dtype='float64', mode='r+', shape=(3,4))
print(arr2.shape)

A (4, 3)

B (12,)

C (3, 4)

D Raises a TypeError

💡 Hint

The shape parameter defines how the data is viewed, not the original file size.

The memmap is reopened with shape (3,4), so arr2.shape is (3,4).

🔧 Debug

advanced

Identify the error in np.memmap usage

What error does this code raise?

NumPy

import numpy as np
filename = 'data3.dat'
arr = np.memmap(filename, dtype='int16', mode='w+', shape=(10,))
arr[:] = np.arange(10)
arr.flush()
# Reopen with wrong dtype
arr2 = np.memmap(filename, dtype='int32', mode='r', shape=(10,))
print(arr2[0])

A ValueError: cannot reshape array of size 20 into shape (10,)

B TypeError: dtype mismatch

C No error, prints 0

D ValueError: buffer size mismatch

💡 Hint

The file was saved with int16 (2 bytes per element), but reopened as int32 (4 bytes per element).

The memmap file size does not match the expected size for int32 with shape (10,), causing a buffer size mismatch error.

🚀 Application

advanced

Efficiently modifying large data with np.memmap

You have a large binary file with 1 million float64 numbers. You want to add 10 to every element without loading the entire file into memory. Which code snippet correctly does this?

A

arr = np.memmap('large.dat', dtype='float64', mode='r')
arr += 10
arr.flush()

B

arr = np.memmap('large.dat', dtype='float64', mode='r+')
for i in range(len(arr)):
    arr[i] += 10
arr.flush()

C

arr = np.memmap('large.dat', dtype='float64', mode='r+')
arr += 10
arr.flush()

D

arr = np.memmap('large.dat', dtype='float64', mode='w+')
arr[:] = 10
arr.flush()

💡 Hint

Mode 'r+' allows read and write. Vectorized operations may load all data into memory.

Option B modifies elements one by one, avoiding loading entire file into memory. Option B uses vectorized addition which loads all data, Option B is read-only mode, Option B overwrites all data with 10.

🧠 Conceptual

expert

Understanding np.memmap file persistence

Which statement about np.memmap file persistence is TRUE?

A Data changes in a memmap array are saved to disk only after calling flush()

B Data changes in a memmap array are immediately saved to disk without flush()

C Memmap arrays do not save data to disk; they only cache in memory

D Calling flush() deletes the memmap file from disk

💡 Hint

Think about when data is guaranteed to be written to disk.

flush() forces changes to be written to disk. Without flush(), changes may remain in OS cache and not be saved immediately.

Practice

1. What is the main benefit of using np.memmap in data science?

easy

A. It allows working with large arrays stored on disk without loading all data into memory.

B. It automatically speeds up all calculations by using GPU acceleration.

C. It compresses data files to save disk space.

D. It converts arrays into Python lists for easier manipulation.

Solution

1. Step 1: Understand what np.memmap does

   np.memmap creates an array-like object that accesses data stored on disk instead of loading it fully into memory.

2. Step 2: Identify the main advantage

   This allows handling very large datasets without using large amounts of RAM, which is the main benefit.

3. Final Answer:

   It allows working with large arrays stored on disk without loading all data into memory. -> Option A

4. Quick Check:

   Memory-mapped files save RAM by accessing disk data [OK]

Hint: Remember: memmap works with disk data like memory arrays [OK]

Common Mistakes:

• Thinking memmap compresses data
• Assuming memmap loads all data into RAM
• Confusing memmap with GPU acceleration

2. Which of the following is the correct way to create a new memory-mapped file with np.memmap of shape (100, 100) and dtype float32?

easy

A. np.memmap('data.dat', dtype='float64', mode='w+', shape=(100, 100))

B. np.memmap('data.dat', dtype='float32', mode='r', shape=(100, 100))

C. np.memmap('data.dat', dtype='float32', mode='rw', shape=(100, 100))

D. np.memmap('data.dat', dtype='float32', mode='w+', shape=(100, 100))

Solution

1. Step 1: Check the mode for creating a new file

   Mode 'w+' creates a new file or overwrites existing one for reading and writing.

2. Step 2: Verify dtype and shape parameters

   The dtype should be 'float32' and shape (100, 100) as given.

3. Final Answer:

   np.memmap('data.dat', dtype='float32', mode='w+', shape=(100, 100)) -> Option D

4. Quick Check:

   Use mode='w+' to create new memmap files [OK]

Hint: Use mode='w+' to create or overwrite memmap files [OK]

Common Mistakes:

• Using mode='r' when creating a new file
• Using incorrect dtype like float64 instead of float32
• Using invalid mode 'rw' which does not exist

3. What will be the output of this code snippet?

import numpy as np
filename = 'test.dat'
# Create memmap file
fp = np.memmap(filename, dtype='int32', mode='w+', shape=(3,3))
fp[:] = np.arange(9).reshape(3,3)
fp.flush()
# Open memmap file in read mode
fp2 = np.memmap(filename, dtype='int32', mode='r', shape=(3,3))
print(fp2[1,2])

medium

A. 6

B. 5

C. 7

D. 8

Solution

1. Step 1: Understand the array content

   np.arange(9).reshape(3,3) creates a 3x3 array: [[0,1,2],[3,4,5],[6,7,8]]

2. Step 2: Identify the value at position [1,2]

   Row 1, column 2 is the third element in second row, which is 5.

3. Final Answer:

   5 -> Option B

4. Quick Check:

   Index [1,2] in arange(9).reshape(3,3) = 5 [OK]

Hint: Remember zero-based indexing for rows and columns [OK]

Common Mistakes:

• Confusing row and column indices
• Forgetting zero-based indexing
• Assuming flush() changes data values

4. Identify the error in this code snippet that tries to open a memmap file:

import numpy as np
filename = 'data.dat'
# Attempt to open memmap file
fp = np.memmap(filename, dtype='float64', mode='r+', shape=(10,10))
print(fp[0,0])

medium

A. File 'data.dat' does not exist, so mode 'r+' causes an error.

B. dtype 'float64' is not supported by np.memmap.

C. Shape parameter must be omitted when opening existing memmap files.

D. Mode 'r+' is read-only and cannot write to file.

Solution

1. Step 1: Understand mode 'r+'

   Mode 'r+' opens an existing file for reading and writing. If file does not exist, it raises an error.

2. Step 2: Check file existence

   If 'data.dat' does not exist, this code will raise a FileNotFoundError.

3. Final Answer:

   File 'data.dat' does not exist, so mode 'r+' causes an error. -> Option A

4. Quick Check:

   Mode 'r+' requires existing file [OK]

Hint: Use mode='w+' to create files, 'r+' needs existing file [OK]

Common Mistakes:

• Assuming 'r+' creates new files
• Thinking dtype 'float64' is invalid
• Believing shape must be omitted always

5. You have a very large dataset stored in a binary file 'large_data.dat' with shape (10000, 10000) and dtype float64. You want to compute the mean of the first column without loading the entire file into memory. Which approach using np.memmap is best?

hard

A. Open the file with mode='w+' and overwrite data before computing mean.

B. Load the entire file into a numpy array and then compute the mean of the first column.

C. Open the file with mode='r' and read only the first column slice to compute the mean.

D. Use np.memmap with mode='c' and compute mean on the whole array.

Solution

1. Step 1: Understand memory constraints

   The dataset is very large (10000x10000), so loading all data into memory is inefficient.

2. Step 2: Use memmap to read only needed data

   Opening with mode='r' allows read-only access. Slicing the first column reads only that part from disk, saving memory.

3. Step 3: Avoid unnecessary writes or full reads

   Mode 'w+' overwrites data, which is not desired. Mode 'c' is copy-on-write and still loads data. Loading full array wastes memory.

4. Final Answer:

   Open the file with mode='r' and read only the first column slice to compute the mean. -> Option C

5. Quick Check:

   Read-only memmap + slice = efficient mean calculation [OK]

Hint: Read only needed slices with mode='r' to save memory [OK]

Common Mistakes:

• Loading entire large file into memory
• Using mode='w+' which overwrites data
• Not slicing and reading whole array unnecessarily