Categorical data type optimization in Data Analysis Python - Step-by-Step Execution

Concept Flow - Categorical data type optimization

Load DataFrame with object columns

↓

Check unique values count

↓

Convert suitable columns to 'category'

↓

Compare memory usage before and after

↓

Use optimized DataFrame for analysis

This flow shows how to convert object columns with repeated values into categorical type to save memory and speed up analysis.

Execution Sample

Data Analysis Python

import pandas as pd

# Create sample data
s = pd.Series(['apple', 'banana', 'apple', 'orange', 'banana', 'banana'])

# Convert to category
d = s.astype('category')

# Show memory usage
print(s.memory_usage(), d.memory_usage())

This code creates a series of fruit names, converts it to a categorical type, and compares memory usage.

Execution Table

Step	Action	Input/Variable	Result/Output
1	Create Series s	['apple', 'banana', 'apple', 'orange', 'banana', 'banana']	s: object dtype, 6 elements
2	Check s.memory_usage()	s	Memory usage: 176 bytes
3	Convert s to category	s.astype('category')	d: categorical dtype, 6 elements
4	Check d.memory_usage()	d	Memory usage: 136 bytes
5	Compare memory usage	s.memory_usage() vs d.memory_usage()	Categorical uses less memory
6	Use d for analysis	d	Optimized data for faster operations

💡 Conversion done, memory usage reduced, ready for optimized analysis

Variable Tracker

Variable	Start	After Conversion	Final
s	Series of strings (object)	Same (unchanged)	Series of strings (object)
d	Not defined	Series of categorical type	Series of categorical type

Key Moments - 3 Insights

Why does converting to 'category' reduce memory usage?

Because categorical stores unique values once and uses integer codes for data, as shown in execution_table step 4 where memory usage drops.

Can all object columns be converted to categorical?

No, only columns with repeated values and limited unique entries benefit; otherwise, it may not save memory or speed.

Does converting to categorical change the data values?

No, the visible data stays the same; only internal storage changes, as seen in variable_tracker where values remain consistent.

Visual Quiz - 3 Questions

Look at the execution_table, what is the memory usage of the original Series s at step 2?

A 136 bytes

B 176 bytes

C 6 bytes

D 200 bytes

💡 Hint

Check the 'Result/Output' column at step 2 in the execution_table.

At which step does the Series get converted to categorical type?

A Step 1

B Step 5

C Step 3

D Step 6

💡 Hint

Look for the action mentioning 'Convert s to category' in the execution_table.

If the Series had all unique values, what would likely happen to memory usage after conversion?

A Memory usage would stay about the same or increase

B Memory usage would become zero

C Memory usage would decrease significantly

D Conversion would fail

💡 Hint

Recall key_moments about when categorical conversion is beneficial.

Concept Snapshot

Categorical data type optimization:
- Use pd.Series.astype('category') to convert object columns
- Saves memory by storing unique values once
- Speeds up operations on repeated values
- Best for columns with few unique entries
- Check memory usage before and after conversion

Full Transcript

We start with a Series of strings representing fruits. Initially, it uses more memory because each string is stored fully. By converting this Series to a categorical type, pandas stores each unique fruit once and uses integer codes for the data. This reduces memory usage, as shown by comparing memory usage before and after conversion. Not all columns benefit; only those with repeated values and limited unique entries. The data values remain the same visually, only the internal storage changes. This optimization helps speed up data analysis and reduces memory load.

Practice

1.

What is the main benefit of converting a DataFrame column to category type in pandas?

easy

A. It reduces memory usage by storing unique values once

B. It automatically sorts the column values alphabetically

C. It converts numeric values to strings

D. It removes missing values from the column

Solution

1. Step 1: Understand what categorical type does

   The categorical type stores unique values only once and uses codes internally, saving memory.

2. Step 2: Compare with other options

   Sorting, converting to strings, or removing missing values are not automatic effects of categorical conversion.

3. Final Answer:

   It reduces memory usage by storing unique values once -> Option A

4. Quick Check:

   Categorical type = Memory saving [OK]

Hint: Categorical stores unique values once to save memory [OK]

Common Mistakes:

• Thinking it sorts values automatically
• Confusing with string conversion
• Assuming it removes missing data

2.

Which of the following is the correct syntax to convert a pandas DataFrame column df['color'] to categorical type?

import pandas as pd
df = pd.DataFrame({'color': ['red', 'blue', 'red', 'green']})

easy

A. df['color'] = df.astype('category')

B. df['color'] = df['color'].astype('category')

C. df['color'] = pd.category(df['color'])

D. df['color'] = df['color'].to_category()

Solution

1. Step 1: Identify correct method to convert column type

   The astype('category') method converts a Series to categorical type.

2. Step 2: Check other options for errors

   df['color'] = df.astype('category') tries to convert whole DataFrame, not column. df['color'] = pd.category(df['color']) uses non-existent pd.category(). df['color'] = df['color'].to_category() uses wrong method name.

3. Final Answer:

   df['color'] = df['color'].astype('category') -> Option B

4. Quick Check:

   astype('category') = Correct syntax [OK]

Hint: Use astype('category') on the specific column [OK]

Common Mistakes:

• Applying astype to whole DataFrame instead of column
• Using non-existent pd.category() function
• Using wrong method like to_category()

3.

What will be the output of the following code?

import pandas as pd

df = pd.DataFrame({'fruit': ['apple', 'banana', 'apple', 'orange', 'banana']})
df['fruit'] = df['fruit'].astype('category')
print(df['fruit'].cat.categories)

medium

A. Index(['apple', 'banana', 'orange'], dtype='object')

B. ['apple', 'banana', 'apple', 'orange', 'banana']

C. ['apple', 'banana', 'orange', 'apple', 'banana']

D. Error: 'cat' attribute not found

Solution

1. Step 1: Convert column to categorical and check categories

   After conversion, cat.categories shows unique sorted categories.

2. Step 2: Identify unique values in sorted order

   Unique fruits are 'apple', 'banana', 'orange' in alphabetical order.

3. Final Answer:

   Index(['apple', 'banana', 'orange'], dtype='object') -> Option A

4. Quick Check:

   cat.categories = unique sorted values [OK]

Hint: cat.categories shows unique sorted categories [OK]

Common Mistakes:

• Expecting original list instead of unique categories
• Confusing categories with values
• Thinking cat attribute is missing

4.

Identify the error in this code snippet that tries to convert a column to categorical:

import pandas as pd
df = pd.DataFrame({'size': ['S', 'M', 'L', 'M']})
df['size'] = df['size'].astype(category)

medium

A. astype() cannot convert to categorical

B. DataFrame creation syntax is wrong

C. Missing quotes around 'category' in astype()

D. Column 'size' does not exist

Solution

1. Step 1: Check astype() argument

   The argument to astype() must be a string 'category', not a bare word.

2. Step 2: Verify other parts of code

   DataFrame creation and column name are correct; astype() supports 'category'.

3. Final Answer:

   Missing quotes around 'category' in astype() -> Option C

4. Quick Check:

   Quotes needed for dtype string [OK]

Hint: Always put quotes around dtype names in astype() [OK]

Common Mistakes:

• Forgetting quotes around 'category'
• Assuming astype() can't convert to category
• Mistaking column name errors

5.

You have a DataFrame with a column 'city' containing 1 million rows but only 5 unique city names. Which approach optimizes memory and speeds up grouping by city?

hard

A. Use df.groupby('city') without any conversion

B. Convert 'city' column to string type before grouping

C. Drop duplicates from 'city' column before grouping

D. Convert 'city' column to categorical type before grouping

Solution

1. Step 1: Understand memory and speed benefits of categorical

   Converting to categorical stores unique values once, reducing memory and speeding grouping.

2. Step 2: Evaluate other options

   String type uses more memory; dropping duplicates loses data; grouping without conversion is slower.

3. Final Answer:

   Convert 'city' column to categorical type before grouping -> Option D

4. Quick Check:

   Categorical = Memory + speed optimization [OK]

Hint: Convert to categorical before grouping large repeated data [OK]

Common Mistakes:

• Using string type wastes memory
• Dropping duplicates loses data
• Grouping without conversion is slower