Memory savings with categoricals in Pandas - Step-by-Step Execution

Concept Flow - Memory savings with categoricals

Load DataFrame with strings

↓

Check memory usage

↓

Convert string column to categorical

↓

Check memory usage again

↓

Compare memory before and after

↓

Observe memory savings

This flow shows loading data, checking memory, converting to categorical, and then comparing memory usage to see savings.

Execution Sample

Pandas

import pandas as pd

# Create DataFrame
df = pd.DataFrame({'color': ['red', 'blue', 'green', 'blue', 'red']*1000})

# Memory before
mem_before = df.memory_usage(deep=True).sum()

# Convert to categorical
_df = df.copy()
_df['color'] = _df['color'].astype('category')

# Memory after
mem_after = _df.memory_usage(deep=True).sum()

This code creates a DataFrame with repeated color strings, checks memory, converts the column to categorical, and checks memory again.

Execution Table

Step	Action	Memory Usage (bytes)	Description
1	Create DataFrame with string column	50000	DataFrame with 5000 rows of repeated strings
2	Calculate memory usage before conversion	50000	Memory usage includes full string storage
3	Convert 'color' column to categorical	N/A	Change data type to categorical
4	Calculate memory usage after conversion	8000	Memory usage reduced due to category codes
5	Compare memory usage	42000 saved	Memory reduced by storing codes instead of full strings

💡 Memory usage after conversion is much smaller, showing savings with categoricals

Variable Tracker

Variable	Start	After Conversion	Final
df['color'].dtype	object (string)	object (string)	object (string)
_df['color'].dtype	N/A	category	category
mem_before	N/A	50000	50000
mem_after	N/A	8000	8000

Key Moments - 2 Insights

Why does memory usage drop after converting to categorical?

Because pandas stores categories as integer codes internally, which use less memory than full strings, as shown in execution_table step 4.

Does the original DataFrame change when converting to categorical?

No, the original DataFrame stays the same; a copy is made before conversion, as shown by variable_tracker where df['color'].dtype remains object.

Visual Quiz - 3 Questions

Look at the execution_table, what is the memory usage after converting to categorical?

A 8000 bytes

B 50000 bytes

C 42000 bytes

D 10000 bytes

💡 Hint

Check execution_table row with Step 4 for memory after conversion

According to variable_tracker, what is the dtype of the 'color' column in the original DataFrame after conversion?

A category

B int

C object (string)

D float

💡 Hint

Look at variable_tracker row for df['color'].dtype after conversion

If the 'color' column had unique strings for every row, how would memory savings change?

A Savings would be larger

B Savings would be smaller or none

C Savings would be the same

D Memory usage would increase

💡 Hint

Think about how categorical codes save memory by reusing repeated values

Concept Snapshot

Memory savings with categoricals in pandas:
- Use df.memory_usage(deep=True) to check memory
- Convert string columns with df['col'] = df['col'].astype('category')
- Categoricals store data as integer codes
- This reduces memory if many repeated values
- Original data unchanged if copied before conversion

Full Transcript

This lesson shows how converting string columns in a pandas DataFrame to categorical type saves memory. We start with a DataFrame of repeated color names stored as strings. We check memory usage before conversion, then convert the column to categorical type, which stores data as integer codes instead of full strings. After conversion, we check memory usage again and see a big drop. The original DataFrame remains unchanged if we work on a copy. This technique is useful when columns have many repeated values, reducing memory use efficiently.

Practice

1.

What is the main benefit of converting a pandas column to category type?

easy

A. It sorts the data automatically

B. It saves memory by storing repeated values efficiently

C. It converts numbers to strings

D. It removes missing values

Solution

1. Step 1: Understand what category type does

   The category type stores repeated text values once and uses codes internally.

2. Step 2: Identify the main benefit

   This reduces memory usage because repeated strings are not stored multiple times.

3. Final Answer:

   It saves memory by storing repeated values efficiently -> Option B

4. Quick Check:

   Memory savings = A [OK]

Hint: Remember: categories store repeated text once [OK]

Common Mistakes:

• Thinking it sorts data automatically
• Confusing category with string conversion
• Assuming it removes missing values

2.

Which of the following is the correct way to convert a pandas DataFrame column df['color'] to a categorical type?

import pandas as pd
df = pd.DataFrame({'color': ['red', 'blue', 'red', 'green']})

easy

A. df['color'] = df['color'].astype('category')

B. df['color'] = df['color'].to_category()

C. df['color'] = pd.category(df['color'])

D. df['color'] = df['color'].convert('category')

Solution

1. Step 1: Recall pandas syntax for type conversion

   Use astype('category') to convert a column to categorical type.

2. Step 2: Check each option

   Only df['color'] = df['color'].astype('category') uses the correct astype method with 'category' string.

3. Final Answer:

   df['color'] = df['color'].astype('category') -> Option A

4. Quick Check:

   astype('category') = A [OK]

Hint: Use astype('category') to convert columns [OK]

Common Mistakes:

• Using non-existent methods like to_category()
• Trying to call pd.category() which does not exist
• Using convert() instead of astype()

3.

What will be the output of the following code?

import pandas as pd
import sys

df = pd.DataFrame({'fruit': ['apple', 'banana', 'apple', 'banana', 'cherry']})
mem_before = df['fruit'].memory_usage()
df['fruit'] = df['fruit'].astype('category')
mem_after = df['fruit'].memory_usage()
print(mem_before, mem_after)

medium

A. 112 80

B. 80 80

C. 80 112

D. 112 112

Solution

1. Step 1: Calculate memory before conversion

   Strings take more memory; here memory_usage() returns 112 bytes before conversion.

2. Step 2: Calculate memory after conversion to category

   Category stores codes and unique values separately, reducing memory to 80 bytes.

3. Final Answer:

   112 80 -> Option A

4. Quick Check:

   Memory before > after = C [OK]

Hint: Memory decreases after astype('category') [OK]

Common Mistakes:

• Assuming memory stays the same
• Thinking memory increases after conversion
• Confusing order of printed values

4.

Identify the error in the following code that tries to convert a column to categorical and save memory:

import pandas as pd
df = pd.DataFrame({'city': ['NY', 'LA', 'NY', 'SF']})
df['city'] = df['city'].astype(category)

medium

A. DataFrame creation syntax is wrong

B. astype cannot convert to category type

C. Missing quotes around 'category' in astype

D. Column 'city' does not exist

Solution

1. Step 1: Check astype argument

   The argument to astype must be a string 'category', but here category is unquoted, causing a NameError.

2. Step 2: Verify other parts

   DataFrame creation and column name are correct; astype supports 'category'.

3. Final Answer:

   Missing quotes around 'category' in astype -> Option C

4. Quick Check:

   Quotes needed for 'category' string [OK]

Hint: Always quote 'category' in astype('category') [OK]

Common Mistakes:

• Forgetting quotes around 'category'
• Thinking astype can't convert to category
• Assuming column name is wrong

5.

You have a DataFrame with a column df['status'] containing 1 million rows with only 3 unique values: 'new', 'in_progress', 'done'. How can you reduce memory usage effectively?

hard

A. Replace values with integers manually without category

B. Convert df['status'] to string type

C. Drop the df['status'] column

D. Convert df['status'] to category type

Solution

1. Step 1: Understand the data characteristics

   With only 3 unique values repeated 1 million times, storing as category saves memory by storing unique values once.

2. Step 2: Compare options

   Converting to string keeps large memory; dropping column loses data; manual integer replacement is error-prone and less efficient.

3. Final Answer:

   Convert df['status'] to category type -> Option D

4. Quick Check:

   Few unique values + large rows = category [OK]

Hint: Use category for few unique values in large data [OK]

Common Mistakes:

• Using string type which uses more memory
• Dropping column losing important data
• Manual integer replacement without category benefits