Working with large files efficiently in NumPy - Mini Project: Build & Apply

Working with large files efficiently

📖 Scenario: You work as a data analyst for a weather station. You receive daily temperature data files that are very large. Loading the entire file at once can be slow and use too much memory. You want to learn how to read and process these large files efficiently using numpy.

🎯 Goal: Learn how to load a large file in smaller parts (chunks) using numpy and calculate the average temperature from the entire file without loading it all at once.

📋 What You'll Learn

Use numpy to load data

Read the file in chunks to save memory

Calculate the average temperature from all chunks

💡 Why This Matters

🌍 Real World

Reading large data files in chunks helps avoid memory overload and speeds up processing in real-world data analysis tasks.

💼 Career

Data scientists and analysts often work with large datasets that cannot fit into memory. Knowing how to process data in chunks is a valuable skill.

1

Create a large temperature data file

Create a numpy array called temperatures with 100000 values starting from 10.0 to 40.0 (inclusive) using np.linspace. Then save this array to a file named temp_data.txt using np.savetxt.

NumPy

import numpy as np
# Create the temperatures array and save it to 'temp_data.txt'
# Your code here

Hint

Use np.linspace(10.0, 40.0, 100000) to create the array and np.savetxt to save it.

2

Set chunk size for reading the file

Create a variable called chunk_size and set it to 10000. This will be the number of temperature values you read at a time from the file.

NumPy

import numpy as np

# Create the temperatures array with 100000 values from 10.0 to 40.0
temperatures = np.linspace(10.0, 40.0, 100000)

# Save the array to 'temp_data.txt'
np.savetxt('temp_data.txt', temperatures)

# Set the chunk size for reading
# Your code here

Hint

Set chunk_size to 10000 to read 10000 values at a time.

3

Read the file in chunks and calculate total sum and count

Open the file 'temp_data.txt' for reading. Use a while loop to read chunk_size lines at a time. Convert each chunk to a numpy array of floats. Keep track of the total sum of all temperatures in a variable called total_sum and the total count of values in a variable called total_count. Stop reading when no more data is left.

NumPy

import numpy as np

# Create the temperatures array with 100000 values from 10.0 to 40.0
temperatures = np.linspace(10.0, 40.0, 100000)

# Save the array to 'temp_data.txt'
np.savetxt('temp_data.txt', temperatures)

# Set the chunk size for reading
chunk_size = 10000

# Initialize total_sum and total_count
# Open the file and read in chunks
# Your code here

Hint

Use a while True loop and read chunk_size lines each time. Convert lines to floats with np.array(..., dtype=float). Add sums and counts to total_sum and total_count.

4

Calculate and print the average temperature

Calculate the average temperature by dividing total_sum by total_count. Print the average temperature rounded to 2 decimal places using print(f"Average temperature: {average:.2f}").

NumPy

import numpy as np

# Create the temperatures array with 100000 values from 10.0 to 40.0
temperatures = np.linspace(10.0, 40.0, 100000)

# Save the array to 'temp_data.txt'
np.savetxt('temp_data.txt', temperatures)

# Set the chunk size for reading
chunk_size = 10000

# Initialize total_sum and total_count
total_sum = 0.0
total_count = 0

# Open the file and read in chunks
with open('temp_data.txt', 'r') as file:
    while True:
        lines = [file.readline() for _ in range(chunk_size)]
        lines = [line.strip() for line in lines if line.strip() != '']
        if not lines:
            break
        chunk = np.array(lines, dtype=float)
        total_sum += chunk.sum()
        total_count += chunk.size

# Calculate and print the average temperature
# Your code here

Hint

Divide total_sum by total_count and print with 2 decimals using an f-string.

Practice

1. What is the main advantage of using np.memmap when working with large binary files?

easy

A. It allows accessing data on disk without loading the entire file into memory.

B. It automatically compresses the file to save disk space.

C. It converts binary files into text files for easier reading.

D. It loads the entire file into memory for faster processing.

Solution

1. Step 1: Understand np.memmap functionality

   np.memmap creates a memory-map to an array stored in a binary file on disk, allowing access without loading all data into RAM.

2. Step 2: Compare options with this behavior

   Only It allows accessing data on disk without loading the entire file into memory. correctly describes this behavior. Options B, C, and D describe unrelated or incorrect features.

3. Final Answer:

   It allows accessing data on disk without loading the entire file into memory. -> Option A

4. Quick Check:

   np.memmap = Access data on disk [OK]

Hint: Remember: memmap reads from disk, not full memory load [OK]

Common Mistakes:

• Thinking memmap loads entire file into memory
• Confusing memmap with file compression
• Assuming memmap converts file formats

2. Which of the following is the correct syntax to create a memory-mapped array from a binary file named data.bin with dtype float32 and shape (1000, 1000)?

easy

A. np.memmap('data.bin', dtype='float64', mode='r', shape=(1000, 1000))

B. np.memmap('data.bin', dtype='int32', mode='w', shape=(1000, 1000))

C. np.memmap('data.bin', dtype='float32', mode='r+', shape=(1000, 1000))

D. np.memmap('data.bin', dtype='float32', mode='rw', shape=(1000, 1000))

Solution

1. Step 1: Identify correct dtype and mode

   The question asks for dtype 'float32' and a mode that allows reading and writing, which is 'r+'.

2. Step 2: Check each option

   np.memmap('data.bin', dtype='float32', mode='r+', shape=(1000, 1000)) matches dtype 'float32' and mode 'r+'. np.memmap('data.bin', dtype='int32', mode='w', shape=(1000, 1000)) has wrong dtype 'int32' and mode 'w' (write only). np.memmap('data.bin', dtype='float64', mode='r', shape=(1000, 1000)) has wrong dtype 'float64' and mode 'r' (read only). np.memmap('data.bin', dtype='float32', mode='rw', shape=(1000, 1000)) uses invalid mode 'rw'.

3. Final Answer:

   np.memmap('data.bin', dtype='float32', mode='r+', shape=(1000, 1000)) -> Option C

4. Quick Check:

   Correct dtype and mode = np.memmap('data.bin', dtype='float32', mode='r+', shape=(1000, 1000)) [OK]

Hint: Use mode 'r+' for read/write memmap [OK]

Common Mistakes:

• Using wrong dtype for the file data
• Using invalid mode like 'rw'
• Confusing read-only 'r' with read/write 'r+'

3. Consider the following code snippet:

import numpy as np
filename = 'largefile.dat'
# Create memmap
mmap = np.memmap(filename, dtype='int32', mode='r', shape=(4, 4))
print(mmap[2, 3])

If the file contains a 4x4 array with values from 0 to 15 in row-major order, what will be the output?

medium

A. 12

B. 14

C. 15

D. 11

Solution

1. Step 1: Understand data layout

   The file stores values 0 to 15 in a 4x4 array in row-major order: [[0,1,2,3],[4,5,6,7],[8,9,10,11],[12,13,14,15]]

2. Step 2: Find value at position (2, 3)

   Row 2 (0-based) is [8,9,10,11]. Index 3 in this row is 11.

3. Final Answer:

   11 -> Option D

4. Quick Check:

   Value at (2,3) = 11 [OK]

Hint: Remember zero-based indexing for arrays [OK]

Common Mistakes:

• Confusing row and column indices
• Using 1-based indexing instead of 0-based
• Mixing up row-major and column-major order

4. You try to create a memmap with this code:

mmap = np.memmap('data.bin', dtype='float32', mode='r+', shape=(1000, 1000))

but get an error: ValueError: cannot mmap an empty file. What is the likely cause and how to fix it?

medium

A. The dtype 'float32' is invalid; use 'float64' instead.

B. The file 'data.bin' is empty; initialize it with correct size before memmap.

C. The mode 'r+' is read-only; use 'w+' to write.

D. The shape (1000, 1000) is too large; reduce it to (100, 100).

Solution

1. Step 1: Understand error cause

   The error means the file exists but has zero bytes, so memmap cannot map it with the given shape and dtype.

2. Step 2: Fix by initializing file size

   To fix, create or resize the file to hold the required data (1000*1000*4 bytes for float32) before memmap.

3. Final Answer:

   The file 'data.bin' is empty; initialize it with correct size before memmap. -> Option B

4. Quick Check:

   Empty file causes mmap error [OK]

Hint: Ensure file size matches array size before memmap [OK]

Common Mistakes:

• Changing dtype without fixing file size
• Using wrong mode without file content
• Reducing shape without reason

5. You have a very large text file with 1 billion numbers separated by spaces. You want to analyze the data using numpy but cannot load all at once. Which approach is best to process this file efficiently?

hard

A. Read the file in chunks, convert each chunk to numpy arrays, and process incrementally.

B. Use np.memmap directly on the text file to access numbers.

C. Read the entire file into memory as a string, then convert to numpy array.

D. Convert the text file to CSV and load with pandas without chunking.

Solution

1. Step 1: Understand file type and memory limits

   The file is a large text file, not binary. np.memmap works only with binary files, so Use np.memmap directly on the text file to access numbers. is invalid.

2. Step 2: Choose efficient reading method

   Reading entire file at once (Read the entire file into memory as a string, then convert to numpy array.) is memory-heavy. Converting to CSV and loading without chunking (Convert the text file to CSV and load with pandas without chunking.) also risks memory overload. Reading in chunks and processing incrementally (Read the file in chunks, convert each chunk to numpy arrays, and process incrementally.) is memory efficient and practical.

3. Final Answer:

   Read the file in chunks, convert each chunk to numpy arrays, and process incrementally. -> Option A

4. Quick Check:

   Chunk reading for large text files = Read the file in chunks, convert each chunk to numpy arrays, and process incrementally. [OK]

Hint: Process large text files in chunks, not all at once [OK]

Common Mistakes:

• Trying to memmap text files
• Loading entire large file into memory
• Ignoring memory limits when converting formats