Memory usage analysis in Data Analysis Python - Step-by-Step Execution

Concept Flow - Memory usage analysis

Start: Import memory_profiler

↓

Define function to analyze memory

↓

Run function with @profile decorator

↓

Track memory before and after key steps

↓

Print or plot memory usage

↓

Interpret memory peaks and usage

↓

Optimize code if needed

↓

End

This flow shows how to use memory profiling tools to measure memory before and after code steps, helping find memory peaks and optimize usage.

Execution Sample

Data Analysis Python

from memory_profiler import profile

@profile
def analyze_memory():
    data = [x * 2 for x in range(1000000)]
    return sum(data)

analyze_memory()

This code measures memory used by creating a large list and summing it, showing memory before and after the list creation.

Execution Table

Step	Action	Memory Before (MB)	Memory After (MB)	Memory Change (MB)
1	Start function analyze_memory	50.0	50.0	0.0
2	Create list data with 1,000,000 elements	50.0	150.0	100.0
3	Calculate sum of data	150.0	150.5	0.5
4	Return sum and end function	150.5	50.0	-100.5

💡 Function ends, memory used by list is released, returning to initial memory level

Variable Tracker

Variable	Start	After Step 2	After Step 3	Final
data	None	[list of 1,000,000 ints]	[list of 1,000,000 ints]	None (released)
sum(data)	None	None	999999000000	999999000000

Key Moments - 3 Insights

Why does memory usage jump so much at step 2?

At step 2, the list 'data' with 1,000,000 elements is created, which uses a lot of memory. This is shown in the execution_table where memory increases by 100 MB.

Why does memory drop back to start after the function ends?

After the function returns, the large list 'data' is no longer needed and is released, so memory usage drops back to the initial level, as seen in step 4.

Why is the memory change small when calculating the sum?

Calculating the sum uses little extra memory because it processes the list without creating a large new object, so memory only increases slightly at step 3.

Visual Quiz - 3 Questions

Look at the execution table, what is the memory usage after creating the list 'data'?

A 50.0 MB

B 100.0 MB

C 150.0 MB

D 150.5 MB

💡 Hint

Check the 'Memory After (MB)' column at step 2 in the execution_table.

At which step does the memory usage return to the starting level?

A Step 4

B Step 3

C Step 2

D Never

💡 Hint

Look at the 'Memory After (MB)' column and find when it goes back to 50.0 MB.

If the list size doubled, how would the memory change at step 2 likely change?

A Memory change would stay the same

B Memory change would roughly double

C Memory change would halve

D Memory change would be zero

💡 Hint

Consider that memory usage depends on the size of the list created at step 2.

Concept Snapshot

Memory usage analysis tracks how much memory code uses at each step.
Use tools like memory_profiler in Python with @profile decorator.
Measure memory before and after key actions.
Look for big jumps to find heavy memory use.
Memory drops when objects are released.
Helps optimize code to use less memory.

Full Transcript

Memory usage analysis helps us see how much memory our code uses step-by-step. We import a tool called memory_profiler and decorate our function with @profile. When we run the function, it shows memory before and after important actions like creating a big list or calculating a sum. We see memory jump when the list is made and drop back when the function ends and the list is released. This helps us find where memory is used most and optimize our code to be more efficient.

Practice

1. What does the memory_usage(deep=True) method in pandas do?

easy

A. It counts the number of rows in the DataFrame.

B. It deletes unused memory from the DataFrame.

C. It compresses the DataFrame to use less memory.

D. It shows detailed memory usage of each column including object types.

Solution

1. Step 1: Understand memory_usage method

   The memory_usage method reports memory used by each column in a DataFrame.

2. Step 2: Effect of deep=True parameter

   Using deep=True makes it include memory used by object types like strings, giving detailed info.

3. Final Answer:

   It shows detailed memory usage of each column including object types. -> Option D

4. Quick Check:

   memory_usage(deep=True) = detailed memory info [OK]

Hint: Remember deep=True reveals full memory use including strings [OK]

Common Mistakes:

• Thinking it deletes or compresses data
• Confusing it with counting rows
• Ignoring deep=True effect

2. Which of the following is the correct syntax to get memory usage of a pandas DataFrame df including deep analysis?

easy

A. memory_usage(df, deep=True)

B. df.memoryusage(deep=True)

C. df.memory_usage(deep=True)

D. df.get_memory_usage(deep=True)

Solution

1. Step 1: Recall pandas method name

   The correct method is memory_usage with underscore, called on DataFrame.

2. Step 2: Check method call syntax

   It is called as df.memory_usage(deep=True). Other options have wrong method names or syntax.

3. Final Answer:

   df.memory_usage(deep=True) -> Option C

4. Quick Check:

   Correct method name and call = df.memory_usage(deep=True) [OK]

Hint: Use exact method name: memory_usage with underscore [OK]

Common Mistakes:

• Misspelling method as memoryusage
• Using function style instead of method
• Adding get_ prefix incorrectly

3. Given the following code, what is the output?

import pandas as pd
df = pd.DataFrame({
  'A': [1, 2, 3],
  'B': ['x', 'y', 'z']
})
print(df.memory_usage(deep=True))

medium

A. Error because deep=True is invalid

B. Index memory + int64 column + object column memory sizes

C. Total memory size as a single number

D. Only memory size of numeric columns

Solution

1. Step 1: Understand DataFrame columns

   Column 'A' is integers, 'B' is strings (object type), and index is default integer index.

2. Step 2: memory_usage(deep=True) output

   This method returns a Series showing memory used by index and each column separately, including deep memory for object column 'B'.

3. Final Answer:

   Index memory + int64 column + object column memory sizes -> Option B

4. Quick Check:

   memory_usage(deep=True) lists memory per column [OK]

Hint: memory_usage(deep=True) shows memory per column including index [OK]

Common Mistakes:

• Expecting a single total number output
• Ignoring index memory
• Thinking deep=True causes error

4. What is wrong with this code snippet?

import pandas as pd
df = pd.DataFrame({'col': ['a', 'b', 'c']})
print(df.memory_usage(deep=False))

medium

A. It does not show full memory usage of string data.

B. The method memory_usage does not exist.

C. deep=False causes a syntax error.

D. DataFrame must have numeric columns to use memory_usage.

Solution

1. Step 1: Check memory_usage with deep=False

   By default or with deep=False, memory_usage does not count full memory used by object columns like strings.

2. Step 2: Effect on output

   This means the reported memory for 'col' will be underestimated, missing actual string memory.

3. Final Answer:

   It does not show full memory usage of string data. -> Option A

4. Quick Check:

   deep=False skips detailed object memory [OK]

Hint: Use deep=True to get full string memory usage [OK]

Common Mistakes:

• Thinking memory_usage method is missing
• Assuming deep=False causes error
• Believing numeric columns are required

5. You have a large DataFrame with many string columns. You want to reduce memory usage without losing data. Which approach helps most based on memory usage analysis?

hard

A. Convert string columns to categorical type after checking memory usage.

B. Drop all string columns to save memory.

C. Use memory_usage(deep=False) to estimate memory and ignore strings.

D. Convert all columns to float type regardless of original data.

Solution

1. Step 1: Analyze memory usage of string columns

   Using memory_usage(deep=True) shows string columns use a lot of memory.

2. Step 2: Convert strings to categorical

   Changing string columns to categorical reduces memory by storing unique values once.

3. Step 3: Avoid data loss

   Converting to categorical keeps data but uses less memory, unlike dropping or wrong type conversion.

4. Final Answer:

   Convert string columns to categorical type after checking memory usage. -> Option A

5. Quick Check:

   Convert strings to categorical for memory saving [OK]

Hint: Convert strings to categorical to save memory without losing data [OK]

Common Mistakes:

• Dropping important columns blindly
• Ignoring deep=True memory insights
• Converting all columns to float causing errors