Why categorical type matters in Pandas

Introduction

Using categorical type helps pandas store and work with repeated text data more efficiently. It saves memory and speeds up some operations.

You have a column with a few repeated text values like colors or categories.

You want to reduce memory use when working with large datasets.

You want faster sorting or grouping by text columns.

You want to control the order of categories for analysis or plotting.

Syntax

Pandas

df['column'] = df['column'].astype('category')

This changes the column to categorical type in pandas.

You can also specify the order of categories if needed.

Examples

Convert the 'color' column to categorical type.

Pandas

df['color'] = df['color'].astype('category')

Create an ordered categorical column for sizes.

Pandas

df['size'] = pd.Categorical(df['size'], categories=['small', 'medium', 'large'], ordered=True)

Sample Program

This code shows how converting a text column to categorical reduces memory use and lists the categories.

Pandas

import pandas as pd

# Create a sample DataFrame
data = {'color': ['red', 'blue', 'red', 'green', 'blue', 'blue']}
df = pd.DataFrame(data)

# Show memory usage before
mem_before = df.memory_usage(deep=True).sum()

# Convert 'color' to categorical
df['color'] = df['color'].astype('category')

# Show memory usage after
mem_after = df.memory_usage(deep=True).sum()

print(f"Memory before: {mem_before} bytes")
print(f"Memory after: {mem_after} bytes")

# Show categories
print(f"Categories: {df['color'].cat.categories.tolist()}")

Important Notes

Categorical columns use less memory because pandas stores the text once and uses codes internally.

Operations like grouping or sorting can be faster with categorical data.

Be careful when adding new categories; you may need to update the category list.

Summary

Categorical type saves memory for repeated text data.

It can speed up some data operations.

You can set an order for categories if needed.

Practice

1. Why is using the categorical data type in pandas helpful when you have a column with many repeated text values?

easy

A. It converts text to numbers for calculations.

B. It automatically sorts the data alphabetically.

C. It saves memory by storing repeated values efficiently.

D. It removes duplicate rows from the DataFrame.

Solution

1. Step 1: Understand memory usage of text data

   Repeated text values take more memory because each string is stored separately.

2. Step 2: How categorical type optimizes memory

   Categorical type stores unique values once and uses codes to represent them, saving memory.

3. Final Answer:

   It saves memory by storing repeated values efficiently. -> Option C

4. Quick Check:

   Memory saving = A [OK]

Hint: Categorical stores unique values once, saving memory [OK]

Common Mistakes:

• Thinking categorical sorts data automatically
• Believing categorical removes duplicates
• Assuming categorical converts text to numbers for math

2. Which of the following is the correct way to convert a pandas column df['color'] to categorical type?

easy

A. df['color'] = df['color'].convert('category')

B. df['color'] = pd.to_numeric(df['color'])

C. df['color'] = df['color'].astype('string')

D. df['color'] = df['color'].astype('category')

Solution

1. Step 1: Identify correct pandas method for type conversion

   The astype() method is used to change data types in pandas.

2. Step 2: Check correct argument for categorical type

   Passing the string 'category' to astype() converts the column to categorical type.

3. Final Answer:

   df['color'] = df['color'].astype('category') -> Option D

4. Quick Check:

   astype('category') = D [OK]

Hint: Use astype('category') to convert column type [OK]

Common Mistakes:

• Using pd.to_numeric for text columns
• Trying astype('string') instead of 'category'
• Using a non-existent convert() method

3. Given this code:

import pandas as pd
colors = ['red', 'blue', 'red', 'green', 'blue', 'blue']
df = pd.DataFrame({'color': colors})
df['color'] = df['color'].astype('category')
print(df['color'].cat.codes.tolist())

What is the output?

medium

A. [2, 0, 2, 1, 0, 0]

B. [1, 2, 1, 3, 2, 2]

C. [0, 1, 0, 2, 1, 1]

D. [0, 0, 0, 0, 0, 0]

Solution

1. Step 1: Understand category codes assignment

   Categories are assigned codes in alphabetical order: 'blue' = 0, 'green' = 1, 'red' = 2.

2. Step 2: Check actual categories order

   By default, pandas assigns categories in alphabetical order, not order of appearance.

3. Step 3: Map colors to codes

   Mapping list: 'red'->2, 'blue'->0, 'red'->2, 'green'->1, 'blue'->0, 'blue'->0.

4. Final Answer:

   [2, 0, 2, 1, 0, 0] -> Option A

5. Quick Check:

   Category codes = A [OK]

Hint: Category codes start at 0 in alphabetical order [OK]

Common Mistakes:

• Assuming order of appearance for codes
• Thinking codes start at 1
• Confusing category codes with original values

4. You run this code:

df['size'] = ['small', 'medium', 'large', 'medium']
df['size'] = df['size'].astype('category')
df['size'] = df['size'].cat.set_categories(['small', 'medium', 'large'], ordered=True)
print(df['size'].sort_values())

But the output is not sorted as expected. What is the likely problem?

medium

A. The set_categories method does not sort the data.

B. The categories were not set as ordered before conversion.

C. You must convert to string type before sorting.

D. The sort_values() method does not work on categorical data.

Solution

1. Step 1: Understand what set_categories does

   It sets the categories and order but does not reorder the actual data.

2. Step 2: How sorting works on ordered categorical

   Sorting uses the category order, but sort_values() sorts rows, not the category levels themselves.

3. Final Answer:

   The set_categories method does not sort the data. -> Option A

4. Quick Check:

   set_categories() sets order, doesn't sort data [OK]

Hint: set_categories sets order, sort_values sorts rows [OK]

Common Mistakes:

• Expecting set_categories to reorder data automatically
• Thinking sort_values fails on categorical columns
• Trying to convert to string before sorting

5. You have a large DataFrame with a column 'city' containing many repeated city names. You want to reduce memory usage and speed up filtering for cities in a specific order: 'New York', 'Los Angeles', 'Chicago'. Which approach is best?

hard

A. Keep 'city' as string type and sort the DataFrame by 'city' alphabetically

B. Convert 'city' to categorical with ordered categories: ['New York', 'Los Angeles', 'Chicago']

C. Convert 'city' to numeric codes without setting categories

D. Remove duplicate city names from the DataFrame

Solution

1. Step 1: Why use categorical with order

   Ordered categorical saves memory and allows meaningful sorting/filtering by specified city order.

2. Step 2: Why other options are less suitable

   Keeping strings uses more memory; numeric codes without categories lose order meaning; removing duplicates loses data.

3. Final Answer:

   Convert 'city' to categorical with ordered categories: ['New York', 'Los Angeles', 'Chicago'] -> Option B

4. Quick Check:

   Ordered categorical = A [OK]

Hint: Use ordered categorical for memory and custom order [OK]

Common Mistakes:

• Not setting categories order
• Using numeric codes without categories
• Removing duplicates instead of optimizing type