Working with large files efficiently in NumPy - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Reading a large binary file with memory mapping

What is the shape of the numpy array data after running this code snippet?

NumPy

import numpy as np
filename = 'large_file.dat'
data = np.memmap(filename, dtype='float32', mode='r', shape=(1000, 1000))
print(data.shape)

A Raises ValueError

B (1000000,)

C (1000,)

D (1000, 1000)

💡 Hint

The shape parameter defines the array dimensions when using np.memmap.

The np.memmap creates an array view of the file with the given shape. Here, shape is (1000, 1000), so the output shape is (1000, 1000).

❓ data_output

intermediate

Effect of chunk reading on memory usage

Given a large CSV file too big to fit in memory, which code snippet correctly reads it in chunks and prints the total number of rows?

NumPy

import pandas as pd
filename = 'large_data.csv'
chunk_size = 10000
row_count = 0
for chunk in pd.read_csv(filename, chunksize=chunk_size):
    row_count += len(chunk)
print(row_count)

A Raises a TypeError

B Prints the total number of rows in the file

C Prints the number of columns in the file

D Prints the number of chunks read

💡 Hint

Reading in chunks allows processing parts of the file without loading all data at once.

The code sums the length of each chunk, resulting in the total number of rows in the file.

🔧 Debug

advanced

Fixing memory error when loading large numpy array

This code tries to load a large numpy array from a file but causes a MemoryError. Which option fixes the issue by using memory mapping?

NumPy

import numpy as np
data = np.load('large_array.npy')
print(data.sum())

A data = np.load('large_array.npy', mmap_mode='r')

B data = np.loadtxt('large_array.npy')

C data = np.memmap('large_array.npy', dtype='float64', mode='r')

D data = np.load('large_array.npy', allow_pickle=True)

💡 Hint

Use the mmap_mode parameter in np.load to avoid loading all data into memory.

Option A uses mmap_mode='r' which memory maps the file, preventing MemoryError by loading data on demand.

❓ visualization

advanced

Visualizing data read in chunks

You want to plot the sum of values in each chunk of a large CSV file. Which code snippet produces a line plot of chunk sums?

NumPy

import pandas as pd
import matplotlib.pyplot as plt
filename = 'large_data.csv'
chunk_size = 5000
chunk_sums = []
for chunk in pd.read_csv(filename, chunksize=chunk_size):
    chunk_sums.append(chunk['value'].sum())
plt.plot(chunk_sums)
plt.xlabel('Chunk number')
plt.ylabel('Sum of values')
plt.title('Sum per chunk')
plt.show()

A Plots a line graph showing sum of 'value' column for each chunk

B Plots a bar chart of the number of rows per chunk

C Plots a scatter plot of 'value' vs index for the entire file

D Raises KeyError because 'value' column does not exist

💡 Hint

Summing values per chunk and plotting them shows trends without loading full data.

The code sums the 'value' column in each chunk and plots these sums as a line graph.

🧠 Conceptual

expert

Choosing the best method for large file processing

You have a 100GB CSV file and limited RAM (8GB). You want to compute the average of a numeric column efficiently. Which approach is best?

A Convert CSV to numpy memmap and compute mean on memmap array

B Load entire file with pandas.read_csv() and compute mean directly

C Use pandas.read_csv() with chunksize to process file in parts and compute running average

D Use Python's built-in open() and read line by line, converting to float and summing

💡 Hint

Think about memory limits and efficient partial processing.

Reading in chunks avoids memory overload and allows incremental computation of the average.