Memory-mapped arrays for large data in NumPy - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of memory-mapped array modification

What is the output of the following code snippet that uses a memory-mapped array?

NumPy

import numpy as np
filename = 'test_mmap.dat'
# Create a memmap file with 5 integers
arr = np.memmap(filename, dtype='int32', mode='w+', shape=(5,))
arr[:] = np.arange(5)
arr.flush()
# Reopen memmap in read-write mode
arr2 = np.memmap(filename, dtype='int32', mode='r+', shape=(5,))
arr2[2] = 100
arr2.flush()
print(arr[:])

A [0 1 100 3 4]

B [0 1 2 3 4]

C [0 1 100 100 4]

D [0 1 2 100 4]

💡 Hint

Remember that memmap shares the same file on disk, so changes in one memmap reflect in others after flush.

The first memmap 'arr' writes values 0 to 4. The second memmap 'arr2' modifies index 2 to 100 and flushes. Since both memmaps point to the same file, reading from 'arr' after flush shows the updated value at index 2.

❓ data_output

intermediate

Number of elements in a memory-mapped array

Given a memory-mapped array created with shape (1000, 1000) and dtype float64, how many elements does it contain?

NumPy

import numpy as np
filename = 'large_mmap.dat'
arr = np.memmap(filename, dtype='float64', mode='w+', shape=(1000, 1000))
print(arr.size)

A 2000

B 1000

C 8000

D 1000000

💡 Hint

The size attribute counts total elements in all dimensions.

The array shape is (1000, 1000), so total elements = 1000 * 1000 = 1,000,000.

🔧 Debug

advanced

Identify the error when opening a memmap file

What error will this code raise when trying to open a memmap file that does not exist in read-only mode?

NumPy

import numpy as np
filename = 'nonexistent.dat'
arr = np.memmap(filename, dtype='float32', mode='r', shape=(10,))

A ValueError

B FileNotFoundError

C TypeError

D No error, creates a new file

💡 Hint

Read-only mode requires the file to exist.

Opening a memmap in mode 'r' requires the file to exist. If it does not, Python raises FileNotFoundError.

🧠 Conceptual

advanced

Why use memory-mapped arrays for large data?

Which of the following is the main advantage of using memory-mapped arrays for large datasets?

A They allow processing data larger than RAM by loading only needed parts from disk.

B They automatically compress data to save disk space.

C They speed up CPU computations by using GPU acceleration.

D They convert data into text format for easier reading.

💡 Hint

Think about how memory-mapping helps with limited memory.

Memory-mapped arrays let you work with data larger than your computer's RAM by loading only the parts you access into memory, avoiding full data loading.

🚀 Application

expert

Predict the output after multiple memmap modifications

Consider this code that creates and modifies a memory-mapped array multiple times. What is the final printed output?

NumPy

import numpy as np
filename = 'multi_mod.dat'
# Create memmap and initialize
arr = np.memmap(filename, dtype='int16', mode='w+', shape=(4,))
arr[:] = [10, 20, 30, 40]
arr.flush()
# Open second memmap and modify
arr2 = np.memmap(filename, dtype='int16', mode='r+', shape=(4,))
arr2[1] = 200
arr2.flush()
# Modify original memmap without flush
arr[2] = 300
print(arr2[:])

A [10 20 30 40]

B [10 200 300 40]

C [10 200 30 40]

D [10 20 300 40]

💡 Hint

Changes in one memmap are visible to others only after flush.

arr2 modifies index 1 to 200 and flushes, so arr2 sees that change. arr modifies index 2 to 300 but does not flush, so arr2 still sees old value 30 at index 2.

Practice

1. What is the main advantage of using numpy.memmap for large data files?

easy

A. It allows working with data without loading the entire file into memory.

B. It compresses the data to save disk space.

C. It automatically cleans up unused files.

D. It speeds up CPU calculations by using GPU.

Solution

1. Step 1: Understand memory usage with large files

   Loading large files fully into RAM can cause memory errors or slowdowns.

2. Step 2: Role of numpy.memmap

   Memory-mapped arrays let you access parts of the file on disk as if they were in memory, avoiding full loading.

3. Final Answer:

   It allows working with data without loading the entire file into memory. -> Option A

4. Quick Check:

   Memory-mapping = Partial loading [OK]

Hint: Memory-mapping means partial file loading, not full load [OK]

Common Mistakes:

• Thinking memmap compresses data
• Assuming memmap speeds up CPU/GPU
• Believing memmap auto-deletes files

2. Which of the following is the correct way to create a memory-mapped array from a file named data.dat with shape (100, 100) and dtype float64?

easy

A. np.memmap('data.dat', shape=(100, 100), dtype='int32', mode='w+')

B. np.memmap('data.dat', dtype='float64', mode='r+', shape=(100, 100))

C. np.memmap('data.dat', dtype='float64', shape=(100, 100))

D. np.memmap('data.dat', mode='r', shape=(100, 100), dtype='float32')

Solution

1. Step 1: Check required parameters for memmap

   Must specify filename, dtype, mode, and shape for memmap creation.

2. Step 2: Validate each option

   np.memmap('data.dat', dtype='float64', mode='r+', shape=(100, 100)) correctly uses dtype='float64', mode='r+', and shape=(100, 100). Others have wrong dtype or missing mode.

3. Final Answer:

   np.memmap('data.dat', dtype='float64', mode='r+', shape=(100, 100)) -> Option B

4. Quick Check:

   Correct dtype, mode, shape = np.memmap('data.dat', dtype='float64', mode='r+', shape=(100, 100)) [OK]

Hint: Always specify dtype, mode, and shape explicitly [OK]

Common Mistakes:

• Omitting mode parameter
• Using wrong dtype
• Swapping dtype and shape order

3. Given the code below, what will be the output of print(mm[0, 0])?

import numpy as np
filename = 'test.dat'
mm = np.memmap(filename, dtype='int32', mode='w+', shape=(2, 2))
mm[0, 0] = 10
mm.flush()
mm2 = np.memmap(filename, dtype='int32', mode='r', shape=(2, 2))
print(mm2[0, 0])

medium

A. Raises an error

B. 0

C. 10

D. Uninitialized garbage value

Solution

1. Step 1: Write value and flush to disk

   Value 10 is assigned to mm[0,0] and flushed, saving to file.

2. Step 2: Read from file with new memmap

   mm2 reads the same file in read mode, so mm2[0,0] is 10.

3. Final Answer:

   10 -> Option C

4. Quick Check:

   Flush saves data, read memmap shows 10 [OK]

Hint: Flush saves changes; reading memmap shows updated value [OK]

Common Mistakes:

• Forgetting to flush before reading
• Expecting zero without initialization
• Confusing write and read modes

4. What is the error in the following code snippet?

import numpy as np
filename = 'data.dat'
mm = np.memmap(filename, dtype='float64', mode='r+', shape=(3, 3))
mm[0, 0] = 5.0
mm.flush()

Assume data.dat does not exist before running this code.

medium

A. No error; code runs fine.

B. TypeError due to wrong dtype.

C. ValueError because shape is incorrect.

D. FileNotFoundError because mode 'r+' requires existing file.

Solution

1. Step 1: Understand mode 'r+'

   Mode 'r+' opens file for reading and writing but requires the file to exist.

2. Step 2: Check file existence

   Since 'data.dat' does not exist, opening with 'r+' raises FileNotFoundError.

3. Final Answer:

   FileNotFoundError because mode 'r+' requires existing file. -> Option D

4. Quick Check:

   Mode 'r+' needs existing file [OK]

Hint: Mode 'r+' needs existing file; else FileNotFoundError [OK]

Common Mistakes:

• Assuming 'r+' creates file
• Confusing 'r+' with 'w+'
• Ignoring file existence requirement

5. You have a large dataset stored in a binary file large_data.dat with shape (10000, 10000) and dtype float32. You want to increment all values by 1 without loading the entire file into memory. Which approach using numpy.memmap is best?

hard

A. Create a memmap with mode 'r+', loop over rows, increment each row, flush after each.

B. Load entire file with np.fromfile, add 1, then save back.

C. Open memmap with mode 'w+', increment only first 100 elements, flush once.

D. Use memmap with mode 'r', add 1 to the array, then flush.

Solution

1. Step 1: Avoid loading entire file

   Loading full 10000x10000 array uses too much RAM, so avoid np.fromfile full load.

2. Step 2: Use memmap with mode 'r+' and process in chunks

   Memmap with 'r+' allows read/write access. Looping over rows increments data without full load. Flushing saves changes.

3. Step 3: Evaluate other options

   Load entire file with np.fromfile, add 1, then save back. loads full file (bad for memory). Open memmap with mode 'w+', increment only first 100 elements, flush once. only increments part of data. Use memmap with mode 'r', add 1 to the array, then flush. uses 'r' mode (read-only), so cannot modify.

4. Final Answer:

   Create a memmap with mode 'r+', loop over rows, increment each row, flush after each. -> Option A

5. Quick Check:

   Memmap + chunk update + flush = Create a memmap with mode 'r+', loop over rows, increment each row, flush after each. [OK]

Hint: Use memmap with 'r+' and update in chunks to save memory [OK]

Common Mistakes:

• Loading entire file into memory
• Using read-only mode for writing
• Modifying only part of data