Why categorical type matters in Pandas - Visual Breakdown

Concept Flow - Why categorical type matters

Load data with text categories

↓

Convert column to categorical type

↓

Check memory usage

↓

Perform operations (filter, group)

↓

Observe speed and memory benefits

This flow shows how converting text columns to categorical type saves memory and speeds up operations.

Execution Sample

Pandas

import pandas as pd

# Create DataFrame with text categories
df = pd.DataFrame({'color': ['red', 'blue', 'red', 'green', 'blue'] * 1000})

# Convert to categorical type
df['color_cat'] = df['color'].astype('category')

# Check memory usage
print(df.memory_usage(deep=True))

This code creates a DataFrame with repeated text categories, converts the column to categorical type, and checks memory usage.

Execution Table

Step	Action	Memory Usage (bytes)	Explanation
1	Create DataFrame with text column 'color'	80000	Text column stores repeated strings, uses more memory
2	Convert 'color' to categorical type as 'color_cat'	32000	Categorical stores codes internally, less memory
3	Filter rows where color_cat == 'red'	N/A	Filtering uses codes, faster than string comparison
4	Group by 'color_cat' and count	N/A	Grouping uses categorical codes, efficient operation
5	End	N/A	Demonstrates memory and speed benefits of categorical type

💡 Execution stops after showing memory usage and example operations with categorical type

Variable Tracker

Variable	Start	After Conversion	After Filtering	After Grouping
df['color']	object dtype, text strings	object dtype, unchanged	object dtype, filtered subset	object dtype, filtered subset
df['color_cat']	Does not exist	category dtype with codes	category dtype, filtered subset	category dtype, grouped summary

Key Moments - 3 Insights

Why does converting to categorical reduce memory usage?

Because categorical stores each unique value once and uses integer codes internally, as shown in execution_table step 2 where memory drops from 80000 to 32000 bytes.

Does the original text column 'color' change after conversion?

No, the original 'color' column remains unchanged as object dtype; the new 'color_cat' column holds the categorical data, as seen in variable_tracker.

Why are operations like filtering and grouping faster with categorical?

Because operations use integer codes instead of string comparisons, making them more efficient, as shown in execution_table steps 3 and 4.

Visual Quiz - 3 Questions

Look at the execution_table at step 2, what happens to memory usage after converting to categorical?

A Memory usage increases

B Memory usage stays the same

C Memory usage decreases

D Memory usage is not measured

💡 Hint

Check the 'Memory Usage (bytes)' column at step 1 and 2 in execution_table

According to variable_tracker, what is the dtype of 'df["color"]' after conversion?

A object dtype

B category dtype

C int dtype

D float dtype

💡 Hint

Look at the 'After Conversion' column for 'df["color"]' in variable_tracker

If we did not convert to categorical, how would filtering by 'red' compare in speed?

A It would be faster

B It would be slower

C Speed would be the same

D Filtering would not work

💡 Hint

Refer to execution_table steps 3 and 4 explanation about filtering and grouping speed

Concept Snapshot

Why categorical type matters:
- Converts repeated text to integer codes
- Saves memory by storing unique values once
- Speeds up filtering and grouping operations
- Original text column remains unchanged
- Use .astype('category') to convert columns

Full Transcript

This lesson shows why using the categorical data type in pandas is important. When you have columns with repeated text values, like colors, storing them as text uses more memory. By converting these columns to categorical type, pandas stores each unique value once and uses small integer codes internally. This reduces memory usage significantly. Also, operations like filtering and grouping become faster because they work with integers instead of strings. The original text column stays the same, and the new categorical column holds the optimized data. This is shown step-by-step with memory usage checks and example operations.