Monitoring memory usage in NumPy - Mini Project: Build & Apply

Monitoring memory usage

📖 Scenario: You are working on a data science project that uses numpy arrays. You want to monitor how much memory your arrays use to optimize your program's performance.

🎯 Goal: Build a simple Python script that creates a numpy array, sets a threshold for memory usage, checks the array's memory usage, and prints a message if the usage exceeds the threshold.

📋 What You'll Learn

Create a numpy array with specific values

Define a memory usage threshold variable

Calculate the memory usage of the array in bytes

Compare the memory usage with the threshold

Print a message showing the memory usage and if it exceeds the threshold

💡 Why This Matters

🌍 Real World

Monitoring memory usage helps keep programs efficient and prevents crashes due to running out of memory.

💼 Career

DevOps engineers and data scientists often monitor memory to optimize applications and infrastructure.

1

Create a numpy array

Import numpy as np and create a numpy array called data with these exact values: [10, 20, 30, 40, 50].

NumPy

# Your code here

Hint

Use np.array() to create the array with the given list of numbers.

2

Set memory usage threshold

Create a variable called memory_threshold and set it to 100 to represent 100 bytes.

NumPy

import numpy as np

data = np.array([10, 20, 30, 40, 50])
# Your code here

Hint

Just assign the number 100 to the variable memory_threshold.

3

Calculate memory usage of the array

Create a variable called memory_used and set it to the number of bytes used by the data array using the nbytes attribute.

NumPy

import numpy as np

data = np.array([10, 20, 30, 40, 50])
memory_threshold = 100
# Your code here

Hint

Use data.nbytes to get the memory size in bytes.

4

Print memory usage and check threshold

Write a print statement that shows the text "Memory used:" followed by the value of memory_used. Then write an if statement that prints "Memory usage exceeds threshold!" if memory_used is greater than memory_threshold.

NumPy

import numpy as np

data = np.array([10, 20, 30, 40, 50])
memory_threshold = 100
memory_used = data.nbytes
# Your code here

Hint

Use print("Memory used:", memory_used) and an if statement to compare memory_used and memory_threshold.

Practice

1. What does the nbytes attribute of a NumPy array represent?

easy

A. The size of each element in the array

B. The total memory in bytes used by the array data

C. The number of elements in the array

D. The number of dimensions of the array

Solution

1. Step 1: Understand what nbytes means

   The nbytes attribute gives the total number of bytes consumed by the elements of the array.

2. Step 2: Differentiate from other attributes

   Other attributes like size give element count, itemsize gives size per element, and ndim gives dimensions.

3. Final Answer:

   The total memory in bytes used by the array data -> Option B

4. Quick Check:

   nbytes = total bytes used [OK]

Hint: Remember: nbytes = total bytes used by array data [OK]

Common Mistakes:

• Confusing nbytes with size (number of elements)
• Thinking nbytes is size of one element
• Mixing nbytes with number of dimensions

2. Which of the following is the correct way to get the memory usage of a NumPy array arr?

easy

A. arr.memsize

B. arr.memory()

C. arr.size()

D. arr.nbytes

Solution

1. Step 1: Recall the attribute for memory usage

   The correct attribute to get total memory used by array data is nbytes, accessed without parentheses.

2. Step 2: Check other options for correctness

   memory() and memsize do not exist; size() is a method but returns element count, not memory.

3. Final Answer:

   arr.nbytes -> Option D

4. Quick Check:

   Use nbytes attribute, no parentheses [OK]

Hint: Use arr.nbytes without parentheses to get memory [OK]

Common Mistakes:

• Using parentheses with nbytes (it's an attribute, not a method)
• Confusing size() method with memory usage
• Trying non-existent methods like memory()

3. What is the output of the following code?

import numpy as np
arr = np.array([1, 2, 3, 4], dtype=np.int32)
print(arr.nbytes)

medium

A. 8

B. 32

C. 16

D. 4

Solution

1. Step 1: Calculate number of elements and size per element

   The array has 4 elements, each of type int32 which uses 4 bytes.

2. Step 2: Multiply elements by bytes per element

   Total bytes = 4 elements * 4 bytes = 16 bytes.

3. Final Answer:

   16 -> Option C

4. Quick Check:

   4 elements x 4 bytes = 16 bytes [OK]

Hint: Multiply element count by itemsize for nbytes [OK]

Common Mistakes:

• Assuming int32 uses 8 bytes
• Confusing nbytes with number of elements
• Forgetting to multiply by element size

4. You run this code but get an error:

import numpy as np
arr = np.array([1, 2, 3])
print(arr.nbytes())

What is the cause of the error?

medium

A. nbytes is an attribute, not a method; remove parentheses

B. Array must be created with dtype specified

C. nbytes only works on 2D arrays

D. You must import nbytes separately

Solution

1. Step 1: Identify the error cause

   The code calls arr.nbytes() with parentheses, but nbytes is an attribute, not a function.

2. Step 2: Correct usage

   Access nbytes without parentheses: arr.nbytes.

3. Final Answer:

   nbytes is an attribute, not a method; remove parentheses -> Option A

4. Quick Check:

   Attributes have no parentheses [OK]

Hint: Remember: attributes like nbytes have no parentheses [OK]

Common Mistakes:

• Adding parentheses to attributes
• Thinking dtype is required for nbytes
• Believing nbytes only works on certain array shapes

5. You have a large NumPy array big_arr with 1 million elements of type float64. You want to reduce memory usage without losing much precision. Which approach helps monitor and reduce memory usage effectively?

hard

A. Check big_arr.nbytes and convert to float32 if memory is high

B. Use big_arr.size to reduce elements by half

C. Delete big_arr and recreate with zeros

D. Convert big_arr to Python list to save memory

Solution

1. Step 1: Monitor memory usage with nbytes

   Use big_arr.nbytes to see current memory consumption in bytes.

2. Step 2: Reduce memory by changing data type

   Converting from float64 (8 bytes) to float32 (4 bytes) halves memory usage with minimal precision loss.

3. Final Answer:

   Check big_arr.nbytes and convert to float32 if memory is high -> Option A

4. Quick Check:

   Use nbytes + dtype change to reduce memory [OK]

Hint: Check nbytes then downcast dtype to save memory [OK]

Common Mistakes:

• Using size instead of nbytes to measure memory
• Converting to Python list increases memory
• Deleting array without monitoring memory