Why categorical type matters in Pandas - Visual Breakdown

Concept Flow - Why categorical type matters

Load data with text categories

↓

Convert column to categorical type

↓

Check memory usage

↓

Perform operations (filter, group)

↓

Observe speed and memory benefits

This flow shows how converting text columns to categorical type saves memory and speeds up operations.

Execution Sample

Pandas

import pandas as pd

# Create DataFrame with text categories
df = pd.DataFrame({'color': ['red', 'blue', 'red', 'green', 'blue'] * 1000})

# Convert to categorical type
df['color_cat'] = df['color'].astype('category')

# Check memory usage
print(df.memory_usage(deep=True))

This code creates a DataFrame with repeated text categories, converts the column to categorical type, and checks memory usage.

Execution Table

Step	Action	Memory Usage (bytes)	Explanation
1	Create DataFrame with text column 'color'	80000	Text column stores repeated strings, uses more memory
2	Convert 'color' to categorical type as 'color_cat'	32000	Categorical stores codes internally, less memory
3	Filter rows where color_cat == 'red'	N/A	Filtering uses codes, faster than string comparison
4	Group by 'color_cat' and count	N/A	Grouping uses categorical codes, efficient operation
5	End	N/A	Demonstrates memory and speed benefits of categorical type

💡 Execution stops after showing memory usage and example operations with categorical type

Variable Tracker

Variable	Start	After Conversion	After Filtering	After Grouping
df['color']	object dtype, text strings	object dtype, unchanged	object dtype, filtered subset	object dtype, filtered subset
df['color_cat']	Does not exist	category dtype with codes	category dtype, filtered subset	category dtype, grouped summary

Key Moments - 3 Insights

Why does converting to categorical reduce memory usage?

Because categorical stores each unique value once and uses integer codes internally, as shown in execution_table step 2 where memory drops from 80000 to 32000 bytes.

Does the original text column 'color' change after conversion?

No, the original 'color' column remains unchanged as object dtype; the new 'color_cat' column holds the categorical data, as seen in variable_tracker.

Why are operations like filtering and grouping faster with categorical?

Because operations use integer codes instead of string comparisons, making them more efficient, as shown in execution_table steps 3 and 4.

Visual Quiz - 3 Questions

Look at the execution_table at step 2, what happens to memory usage after converting to categorical?

A Memory usage increases

B Memory usage stays the same

C Memory usage decreases

D Memory usage is not measured

💡 Hint

Check the 'Memory Usage (bytes)' column at step 1 and 2 in execution_table

According to variable_tracker, what is the dtype of 'df["color"]' after conversion?

A object dtype

B category dtype

C int dtype

D float dtype

💡 Hint

Look at the 'After Conversion' column for 'df["color"]' in variable_tracker

If we did not convert to categorical, how would filtering by 'red' compare in speed?

A It would be faster

B It would be slower

C Speed would be the same

D Filtering would not work

💡 Hint

Refer to execution_table steps 3 and 4 explanation about filtering and grouping speed

Concept Snapshot

Why categorical type matters:
- Converts repeated text to integer codes
- Saves memory by storing unique values once
- Speeds up filtering and grouping operations
- Original text column remains unchanged
- Use .astype('category') to convert columns

Full Transcript

This lesson shows why using the categorical data type in pandas is important. When you have columns with repeated text values, like colors, storing them as text uses more memory. By converting these columns to categorical type, pandas stores each unique value once and uses small integer codes internally. This reduces memory usage significantly. Also, operations like filtering and grouping become faster because they work with integers instead of strings. The original text column stays the same, and the new categorical column holds the optimized data. This is shown step-by-step with memory usage checks and example operations.

Practice

1. Why is using the categorical data type in pandas helpful when you have a column with many repeated text values?

easy

A. It converts text to numbers for calculations.

B. It automatically sorts the data alphabetically.

C. It saves memory by storing repeated values efficiently.

D. It removes duplicate rows from the DataFrame.

Solution

1. Step 1: Understand memory usage of text data

   Repeated text values take more memory because each string is stored separately.

2. Step 2: How categorical type optimizes memory

   Categorical type stores unique values once and uses codes to represent them, saving memory.

3. Final Answer:

   It saves memory by storing repeated values efficiently. -> Option C

4. Quick Check:

   Memory saving = A [OK]

Hint: Categorical stores unique values once, saving memory [OK]

Common Mistakes:

• Thinking categorical sorts data automatically
• Believing categorical removes duplicates
• Assuming categorical converts text to numbers for math

2. Which of the following is the correct way to convert a pandas column df['color'] to categorical type?

easy

A. df['color'] = df['color'].convert('category')

B. df['color'] = pd.to_numeric(df['color'])

C. df['color'] = df['color'].astype('string')

D. df['color'] = df['color'].astype('category')

Solution

1. Step 1: Identify correct pandas method for type conversion

   The astype() method is used to change data types in pandas.

2. Step 2: Check correct argument for categorical type

   Passing the string 'category' to astype() converts the column to categorical type.

3. Final Answer:

   df['color'] = df['color'].astype('category') -> Option D

4. Quick Check:

   astype('category') = D [OK]

Hint: Use astype('category') to convert column type [OK]

Common Mistakes:

• Using pd.to_numeric for text columns
• Trying astype('string') instead of 'category'
• Using a non-existent convert() method

3. Given this code:

import pandas as pd
colors = ['red', 'blue', 'red', 'green', 'blue', 'blue']
df = pd.DataFrame({'color': colors})
df['color'] = df['color'].astype('category')
print(df['color'].cat.codes.tolist())

What is the output?

medium

A. [2, 0, 2, 1, 0, 0]

B. [1, 2, 1, 3, 2, 2]

C. [0, 1, 0, 2, 1, 1]

D. [0, 0, 0, 0, 0, 0]

Solution

1. Step 1: Understand category codes assignment

   Categories are assigned codes in alphabetical order: 'blue' = 0, 'green' = 1, 'red' = 2.

2. Step 2: Check actual categories order

   By default, pandas assigns categories in alphabetical order, not order of appearance.

3. Step 3: Map colors to codes

   Mapping list: 'red'->2, 'blue'->0, 'red'->2, 'green'->1, 'blue'->0, 'blue'->0.

4. Final Answer:

   [2, 0, 2, 1, 0, 0] -> Option A

5. Quick Check:

   Category codes = A [OK]

Hint: Category codes start at 0 in alphabetical order [OK]

Common Mistakes:

• Assuming order of appearance for codes
• Thinking codes start at 1
• Confusing category codes with original values

4. You run this code:

df['size'] = ['small', 'medium', 'large', 'medium']
df['size'] = df['size'].astype('category')
df['size'] = df['size'].cat.set_categories(['small', 'medium', 'large'], ordered=True)
print(df['size'].sort_values())

But the output is not sorted as expected. What is the likely problem?

medium

A. The set_categories method does not sort the data.

B. The categories were not set as ordered before conversion.

C. You must convert to string type before sorting.

D. The sort_values() method does not work on categorical data.

Solution

1. Step 1: Understand what set_categories does

   It sets the categories and order but does not reorder the actual data.

2. Step 2: How sorting works on ordered categorical

   Sorting uses the category order, but sort_values() sorts rows, not the category levels themselves.

3. Final Answer:

   The set_categories method does not sort the data. -> Option A

4. Quick Check:

   set_categories() sets order, doesn't sort data [OK]

Hint: set_categories sets order, sort_values sorts rows [OK]

Common Mistakes:

• Expecting set_categories to reorder data automatically
• Thinking sort_values fails on categorical columns
• Trying to convert to string before sorting

5. You have a large DataFrame with a column 'city' containing many repeated city names. You want to reduce memory usage and speed up filtering for cities in a specific order: 'New York', 'Los Angeles', 'Chicago'. Which approach is best?

hard

A. Keep 'city' as string type and sort the DataFrame by 'city' alphabetically

B. Convert 'city' to categorical with ordered categories: ['New York', 'Los Angeles', 'Chicago']

C. Convert 'city' to numeric codes without setting categories

D. Remove duplicate city names from the DataFrame

Solution

1. Step 1: Why use categorical with order

   Ordered categorical saves memory and allows meaningful sorting/filtering by specified city order.

2. Step 2: Why other options are less suitable

   Keeping strings uses more memory; numeric codes without categories lose order meaning; removing duplicates loses data.

3. Final Answer:

   Convert 'city' to categorical with ordered categories: ['New York', 'Los Angeles', 'Chicago'] -> Option B

4. Quick Check:

   Ordered categorical = A [OK]

Hint: Use ordered categorical for memory and custom order [OK]

Common Mistakes:

• Not setting categories order
• Using numeric codes without categories
• Removing duplicates instead of optimizing type