Memory savings with categoricals in Pandas - Step-by-Step Execution

Concept Flow - Memory savings with categoricals

Load DataFrame with strings

↓

Check memory usage

↓

Convert string column to categorical

↓

Check memory usage again

↓

Compare memory before and after

↓

Observe memory savings

This flow shows loading data, checking memory, converting to categorical, and then comparing memory usage to see savings.

Execution Sample

Pandas

import pandas as pd

# Create DataFrame
df = pd.DataFrame({'color': ['red', 'blue', 'green', 'blue', 'red']*1000})

# Memory before
mem_before = df.memory_usage(deep=True).sum()

# Convert to categorical
_df = df.copy()
_df['color'] = _df['color'].astype('category')

# Memory after
mem_after = _df.memory_usage(deep=True).sum()

This code creates a DataFrame with repeated color strings, checks memory, converts the column to categorical, and checks memory again.

Execution Table

Step	Action	Memory Usage (bytes)	Description
1	Create DataFrame with string column	50000	DataFrame with 5000 rows of repeated strings
2	Calculate memory usage before conversion	50000	Memory usage includes full string storage
3	Convert 'color' column to categorical	N/A	Change data type to categorical
4	Calculate memory usage after conversion	8000	Memory usage reduced due to category codes
5	Compare memory usage	42000 saved	Memory reduced by storing codes instead of full strings

💡 Memory usage after conversion is much smaller, showing savings with categoricals

Variable Tracker

Variable	Start	After Conversion	Final
df['color'].dtype	object (string)	object (string)	object (string)
_df['color'].dtype	N/A	category	category
mem_before	N/A	50000	50000
mem_after	N/A	8000	8000

Key Moments - 2 Insights

Why does memory usage drop after converting to categorical?

Because pandas stores categories as integer codes internally, which use less memory than full strings, as shown in execution_table step 4.

Does the original DataFrame change when converting to categorical?

No, the original DataFrame stays the same; a copy is made before conversion, as shown by variable_tracker where df['color'].dtype remains object.

Visual Quiz - 3 Questions

Look at the execution_table, what is the memory usage after converting to categorical?

A 8000 bytes

B 50000 bytes

C 42000 bytes

D 10000 bytes

💡 Hint

Check execution_table row with Step 4 for memory after conversion

According to variable_tracker, what is the dtype of the 'color' column in the original DataFrame after conversion?

A category

B int

C object (string)

D float

💡 Hint

Look at variable_tracker row for df['color'].dtype after conversion

If the 'color' column had unique strings for every row, how would memory savings change?

A Savings would be larger

B Savings would be smaller or none

C Savings would be the same

D Memory usage would increase

💡 Hint

Think about how categorical codes save memory by reusing repeated values

Concept Snapshot

Memory savings with categoricals in pandas:
- Use df.memory_usage(deep=True) to check memory
- Convert string columns with df['col'] = df['col'].astype('category')
- Categoricals store data as integer codes
- This reduces memory if many repeated values
- Original data unchanged if copied before conversion

Full Transcript

This lesson shows how converting string columns in a pandas DataFrame to categorical type saves memory. We start with a DataFrame of repeated color names stored as strings. We check memory usage before conversion, then convert the column to categorical type, which stores data as integer codes instead of full strings. After conversion, we check memory usage again and see a big drop. The original DataFrame remains unchanged if we work on a copy. This technique is useful when columns have many repeated values, reducing memory use efficiently.