Memory-mapped arrays for large data in NumPy - Mini Project: Build & Apply

Memory-mapped arrays for large data

📋 What You'll Learn

💡 Why This Matters

🌍 Real World

Memory-mapped arrays are used when working with very large datasets that do not fit into memory, such as big images, scientific data, or logs. They allow efficient access to parts of the data without loading everything at once.

💼 Career

Data scientists and engineers often handle large datasets. Knowing how to use memory-mapped arrays helps in optimizing memory usage and speeding up data processing tasks.

1

Create a large numpy array and save it to a file

Create a numpy array called large_array with values from 0 to 9999 using np.arange(10000). Then save this array to a file named 'large_data.dat' using large_array.tofile('large_data.dat').

NumPy

import numpy as np
# Create a numpy array called large_array with values 0 to 9999
# Save large_array to 'large_data.dat' file
# Your code here

Need a hint?

Use np.arange(10000) to create numbers from 0 to 9999. Use tofile method to save the array to a file.

2

Set the slice size for reading data

Create a variable called slice_size and set it to 100. This variable will control how many elements we read from the memory-mapped array.

NumPy

import numpy as np
large_array = np.arange(10000, dtype=np.int64)
large_array.tofile('large_data.dat')
# Set slice_size to 100
# Your code here

Need a hint?

Just create a variable named slice_size and assign it the value 100.

3

Create a memory-mapped array and read a slice

Create a memory-mapped array called mmap_array from the file 'large_data.dat' using np.memmap with dtype np.int64 and mode 'r'. Then create a variable called data_slice that contains the first slice_size elements of mmap_array using slicing.

NumPy

import numpy as np
large_array = np.arange(10000, dtype=np.int64)
large_array.tofile('large_data.dat')
slice_size = 100
# Create a memory-mapped array mmap_array from 'large_data.dat'
# Read the first slice_size elements into data_slice
# Your code here

Need a hint?

Use np.memmap with the file name, dtype np.int64, and mode 'r' to create the memory-mapped array. Use slicing [:slice_size] to get the first part.

4

Print the sliced data

Print the variable data_slice to display the first 100 elements read from the memory-mapped array.

NumPy

import numpy as np
large_array = np.arange(10000, dtype=np.int64)
large_array.tofile('large_data.dat')
slice_size = 100
mmap_array = np.memmap('large_data.dat', dtype=np.int64, mode='r')
data_slice = mmap_array[:slice_size]
# Print data_slice
# Your code here

Need a hint?

Use print(data_slice) to show the first 100 elements.