Memory-mapped files with np.memmap in NumPy - Mini Project: Build & Apply

Memory-mapped files with np.memmap

📖 Scenario: You are working with a large dataset of numbers that cannot fit entirely into your computer's memory. To handle this, you decide to use memory-mapped files, which allow you to work with data stored on disk as if it were in memory.

🎯 Goal: Learn how to create a memory-mapped file using np.memmap, write data to it, and read data back efficiently without loading the entire file into memory.

📋 What You'll Learn

Create a numpy memmap file with specific shape and data type

Write data to the memmap file

Read data from the memmap file using slicing

Print the read data to verify correctness

💡 Why This Matters

🌍 Real World

Memory-mapped files are used when working with very large datasets that do not fit into memory, such as large images, scientific data, or logs.

💼 Career

Data scientists and engineers use memory-mapped files to efficiently process big data without running out of memory, improving performance and scalability.

1

Create a memory-mapped file

Create a memory-mapped file called data_memmap using np.memmap with filename 'data.dat', data type float32, mode 'w+' (write plus read), and shape (5, 5).

NumPy

# Create a memory-mapped file called data_memmap
# Your code here

Need a hint?

Use np.memmap with the correct filename, dtype, mode, and shape parameters.

2

Write data to the memory-mapped file

Assign the numpy array np.arange(25, dtype='float32').reshape(5, 5) to the memory-mapped file data_memmap to fill it with values from 0 to 24.

NumPy

import numpy as np

data_memmap = np.memmap('data.dat', dtype='float32', mode='w+', shape=(5, 5))
# Write values 0 to 24 reshaped to (5,5) into data_memmap
# Your code here

Need a hint?

Use slicing data_memmap[:] to assign the new array.

3

Read a slice from the memory-mapped file

Create a variable called slice_data that reads the first 3 rows and 2 columns from data_memmap using slicing.

NumPy

import numpy as np

data_memmap = np.memmap('data.dat', dtype='float32', mode='w+', shape=(5, 5))
data_memmap[:] = np.arange(25, dtype='float32').reshape(5, 5)
# Read the first 3 rows and 2 columns into slice_data
# Your code here

Need a hint?

Use slicing with [:3, :2] to get the first 3 rows and 2 columns.

4

Print the sliced data

Print the variable slice_data to display the selected part of the memory-mapped file.

NumPy

import numpy as np

data_memmap = np.memmap('data.dat', dtype='float32', mode='w+', shape=(5, 5))
data_memmap[:] = np.arange(25, dtype='float32').reshape(5, 5)
slice_data = data_memmap[:3, :2]
# Print slice_data
# Your code here

Need a hint?

Use print(slice_data) to show the data.