Memory usage analysis in Pandas - Step-by-Step Execution

Concept Flow - Memory usage analysis

Load DataFrame

↓

Call memory_usage()

↓

Calculate memory per column

↓

Sum memory if deep=True

↓

Return memory usage info

↓

Use info for optimization

This flow shows how pandas calculates memory usage of each DataFrame column and optionally sums it for total usage.

Execution Sample

Pandas

import pandas as pd

df = pd.DataFrame({
    'A': range(3),
    'B': ['x', 'y', 'z']
})

mem = df.memory_usage(deep=True)

This code creates a DataFrame and calculates memory usage of each column including object data.

Execution Table

Step	Action	Column	Memory Usage (bytes)	Total Memory (bytes)
1	Calculate memory for column	Index	128	128
2	Calculate memory for column	A	24	152
3	Calculate memory for column	B	177	329
4	Sum all columns memory	All	329	329
5	Return memory usage Series and total	-	-	-

💡 All columns processed and total memory usage calculated

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
memory_per_column	{}	{'Index':128}	{'Index':128, 'A':24}	{'Index':128, 'A':24, 'B':177}	{'Index':128, 'A':24, 'B':177}
total_memory	0	0	0	0	329

Key Moments - 2 Insights

Why does the 'Index' column appear in memory usage?

The index is part of the DataFrame structure and uses memory, so pandas includes it in the memory_usage output as shown in steps 1 and 4.

What does the 'deep=True' parameter do?

It makes pandas calculate the actual memory used by object types like strings, not just pointers, which is why column 'B' has 177 bytes in step 3.

Visual Quiz - 3 Questions

Look at the execution table, what is the memory usage of column 'A' at step 2?

A 128 bytes

B 24 bytes

C 99 bytes

D 251 bytes

💡 Hint

Check the 'Memory Usage (bytes)' column for step 2 in the execution_table.

At which step is the total memory usage calculated?

A Step 4

B Step 3

C Step 2

D Step 5

💡 Hint

Look for the row where 'Sum all columns memory' is the action in the execution_table.

If 'deep=False' was used, how would the memory for column 'B' change?

A It would be larger

B It would be the same

C It would be smaller

D Column 'B' would be excluded

💡 Hint

Recall that 'deep=True' counts actual string memory; without it, only pointers are counted, so memory is less.

Concept Snapshot

pandas.DataFrame.memory_usage(deep=False)
- Returns memory usage per column including index
- deep=True counts actual object memory (e.g., strings)
- Useful to find heavy columns
- Helps optimize DataFrame memory
- Returns Series with bytes per column plus index

Full Transcript

This visual trace shows how pandas calculates memory usage of a DataFrame. First, it loads the DataFrame with columns 'A' and 'B'. Then, calling memory_usage(deep=True) calculates memory for each column including the index. The index uses 128 bytes, column 'A' uses 24 bytes, and column 'B' uses 177 bytes because it contains strings. These values are summed to get total memory usage of 329 bytes. The deep=True parameter is important to count actual string memory, not just pointers. This helps understand which parts of the DataFrame use the most memory and can guide optimization.

Practice

1. What does the memory_usage() function in pandas primarily show?

easy

A. The amount of memory used by each column and the index in a DataFrame

B. The total number of rows in a DataFrame

C. The data types of each column in a DataFrame

D. The number of missing values in each column

Solution

1. Step 1: Understand the purpose of memory_usage()

   This function reports memory consumption of each column and the index in a pandas DataFrame.

2. Step 2: Compare with other options

   Options A, B, and D describe different DataFrame properties unrelated to memory usage.

3. Final Answer:

   The amount of memory used by each column and the index in a DataFrame -> Option A

4. Quick Check:

   memory_usage() = memory per column and index [OK]

Hint: memory_usage() shows memory per column and index [OK]

Common Mistakes:

• Confusing memory_usage() with shape or info()
• Thinking it shows data types or missing values
• Ignoring that it includes index memory

2. Which of the following is the correct syntax to get detailed memory usage of a DataFrame df including object types?

easy

A. df.memory_usage()

B. df.memory_usage(True)

C. df.memory_usage(deep=True)

D. df.memory_usage(deep=False)

Solution

1. Step 1: Recall the parameter for detailed memory info

   The deep=True argument tells pandas to include full memory usage for object columns.

2. Step 2: Check syntax correctness

   df.memory_usage(deep=True) uses the correct keyword argument deep=True. df.memory_usage(True) uses positional argument incorrectly. Options A, B, and C do not provide detailed memory info.

3. Final Answer:

   df.memory_usage(deep=True) -> Option C

4. Quick Check:

   Use deep=True for detailed memory [OK]

Hint: Use deep=True to get full memory info on objects [OK]

Common Mistakes:

• Passing True without keyword (positional argument error)
• Not using deep=True and missing object memory
• Using deep=False which is default and less detailed

3. Given the following code:

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, 3],
    'B': ['x', 'y', 'z']
})
print(df.memory_usage(deep=True))

What will be the output type of print(df.memory_usage(deep=True))?

medium

A. A pandas Series showing memory usage per column and index

B. A single integer showing total memory usage

C. A dictionary with column names as keys and memory usage as values

D. A DataFrame with memory usage details

Solution

1. Step 1: Understand the return type of memory_usage()

   The memory_usage() method returns a pandas Series with index as column names plus the index, and values as memory usage in bytes.

2. Step 2: Check the code output type

   Since deep=True is used, it includes detailed memory for object columns but still returns a Series, not a dict or DataFrame.

3. Final Answer:

   A pandas Series showing memory usage per column and index -> Option A

4. Quick Check:

   memory_usage() returns Series [OK]

Hint: memory_usage() returns Series with memory per column [OK]

Common Mistakes:

• Expecting total memory as single integer
• Thinking it returns a dictionary or DataFrame
• Confusing output type with other pandas methods

4. You run df.memory_usage(deep=True) but get an error: TypeError: memory_usage() got an unexpected keyword argument 'deep'. What is the likely cause?

medium

A. You forgot to import pandas as pd

B. You are using an older pandas version that does not support the 'deep' argument

C. You passed 'deep' as a positional argument instead of keyword

D. The DataFrame is empty

Solution

1. Step 1: Analyze the error message

   The error says 'unexpected keyword argument deep', meaning the function does not recognize 'deep'.

2. Step 2: Identify cause based on pandas version

   Older pandas versions did not support the 'deep' argument in memory_usage(). This is the most common cause.

3. Final Answer:

   You are using an older pandas version that does not support the 'deep' argument -> Option B

4. Quick Check:

   Old pandas versions lack deep argument [OK]

Hint: Check pandas version if 'deep' argument causes error [OK]

Common Mistakes:

• Assuming import error causes this
• Thinking positional argument fixes it
• Blaming empty DataFrame for this error

5. You have a large DataFrame with many object columns. You want to reduce memory usage by converting some columns to categorical type. Which approach helps you analyze memory before and after conversion?

hard

A. Use df.memory_usage() without deep argument before conversion

B. Use df.info() only after conversion

C. Use df.describe() before conversion

D. Use df.memory_usage(deep=True) before and after converting columns to 'category' dtype

Solution

1. Step 1: Understand memory usage measurement

   Using memory_usage(deep=True) gives detailed memory info including object columns, which is important before conversion.

2. Step 2: Compare memory before and after conversion

   By running this method before and after converting columns to 'category', you can see the memory savings clearly.

3. Final Answer:

   Use df.memory_usage(deep=True) before and after converting columns to 'category' dtype -> Option D

4. Quick Check:

   memory_usage(deep=True) shows detailed memory changes [OK]

Hint: Compare memory_usage(deep=True) before and after changes [OK]

Common Mistakes:

• Using info() which shows less detailed memory
• Using describe() which summarizes stats, not memory
• Using memory_usage() without deep misses object memory