Why categorical type matters in Pandas - See It in Action

Why categorical type matters

📖 Scenario: Imagine you work in a store that sells different types of fruits. You have a list of fruits sold each day, but some fruits appear many times. You want to organize this data efficiently and understand how many times each fruit was sold.

🎯 Goal: You will create a list of fruits sold, convert it to a categorical type in pandas, and then compare the memory usage and counts of each fruit. This will show why using categorical data is helpful.

📋 What You'll Learn

Create a pandas Series with fruit names

Create a categorical version of the Series

Compare memory usage of both Series

Count how many times each fruit appears

💡 Why This Matters

🌍 Real World

Stores, surveys, and many datasets have repeated categories like product types or answers. Using categorical types helps save memory and speeds up analysis.

💼 Career

Data analysts and scientists often work with large datasets. Knowing how to use categorical data types helps optimize performance and resource use.

1

Create a pandas Series with fruit names

Import pandas as pd and create a pandas Series called fruits with these exact values: ["apple", "banana", "apple", "orange", "banana", "apple"].

Pandas

# Import pandas and create the fruits Series
# Your code here

Hint

Use pd.Series() to create the Series from the list of fruits.

2

Create a categorical version of the Series

Create a new variable called fruits_cat by converting the fruits Series to categorical type using pd.Series.astype('category').

Pandas

import pandas as pd
fruits = pd.Series(["apple", "banana", "apple", "orange", "banana", "apple"])
# Convert fruits to categorical type and store in fruits_cat
# Your code here

Hint

Use astype('category') on the fruits Series.

3

Compare memory usage of both Series

Create two variables: mem_fruits and mem_fruits_cat. Use the memory_usage(deep=True) method on fruits and fruits_cat respectively to get their memory usage.

Pandas

import pandas as pd
fruits = pd.Series(["apple", "banana", "apple", "orange", "banana", "apple"])
fruits_cat = fruits.astype('category')
# Get memory usage of fruits and fruits_cat
# Your code here

Hint

Use memory_usage(deep=True) on both Series and save the results.

4

Print memory usage and counts of each fruit

Print the memory usage variables mem_fruits and mem_fruits_cat on separate lines. Then print the counts of each fruit in fruits_cat using the value_counts() method.

Pandas

import pandas as pd
fruits = pd.Series(["apple", "banana", "apple", "orange", "banana", "apple"])
fruits_cat = fruits.astype('category')
mem_fruits = fruits.memory_usage(deep=True)
mem_fruits_cat = fruits_cat.memory_usage(deep=True)
# Print memory usage and counts of fruits_cat
# Your code here

Hint

Use print() to show memory usage and value_counts() to count fruits.

Practice

1. Why is using the categorical data type in pandas helpful when you have a column with many repeated text values?

easy

A. It converts text to numbers for calculations.

B. It automatically sorts the data alphabetically.

C. It saves memory by storing repeated values efficiently.

D. It removes duplicate rows from the DataFrame.

Solution

1. Step 1: Understand memory usage of text data

   Repeated text values take more memory because each string is stored separately.

2. Step 2: How categorical type optimizes memory

   Categorical type stores unique values once and uses codes to represent them, saving memory.

3. Final Answer:

   It saves memory by storing repeated values efficiently. -> Option C

4. Quick Check:

   Memory saving = A [OK]

Hint: Categorical stores unique values once, saving memory [OK]

Common Mistakes:

• Thinking categorical sorts data automatically
• Believing categorical removes duplicates
• Assuming categorical converts text to numbers for math

2. Which of the following is the correct way to convert a pandas column df['color'] to categorical type?

easy

A. df['color'] = df['color'].convert('category')

B. df['color'] = pd.to_numeric(df['color'])

C. df['color'] = df['color'].astype('string')

D. df['color'] = df['color'].astype('category')

Solution

1. Step 1: Identify correct pandas method for type conversion

   The astype() method is used to change data types in pandas.

2. Step 2: Check correct argument for categorical type

   Passing the string 'category' to astype() converts the column to categorical type.

3. Final Answer:

   df['color'] = df['color'].astype('category') -> Option D

4. Quick Check:

   astype('category') = D [OK]

Hint: Use astype('category') to convert column type [OK]

Common Mistakes:

• Using pd.to_numeric for text columns
• Trying astype('string') instead of 'category'
• Using a non-existent convert() method

3. Given this code:

import pandas as pd
colors = ['red', 'blue', 'red', 'green', 'blue', 'blue']
df = pd.DataFrame({'color': colors})
df['color'] = df['color'].astype('category')
print(df['color'].cat.codes.tolist())

What is the output?

medium

A. [2, 0, 2, 1, 0, 0]

B. [1, 2, 1, 3, 2, 2]

C. [0, 1, 0, 2, 1, 1]

D. [0, 0, 0, 0, 0, 0]

Solution

1. Step 1: Understand category codes assignment

   Categories are assigned codes in alphabetical order: 'blue' = 0, 'green' = 1, 'red' = 2.

2. Step 2: Check actual categories order

   By default, pandas assigns categories in alphabetical order, not order of appearance.

3. Step 3: Map colors to codes

   Mapping list: 'red'->2, 'blue'->0, 'red'->2, 'green'->1, 'blue'->0, 'blue'->0.

4. Final Answer:

   [2, 0, 2, 1, 0, 0] -> Option A

5. Quick Check:

   Category codes = A [OK]

Hint: Category codes start at 0 in alphabetical order [OK]

Common Mistakes:

• Assuming order of appearance for codes
• Thinking codes start at 1
• Confusing category codes with original values

4. You run this code:

df['size'] = ['small', 'medium', 'large', 'medium']
df['size'] = df['size'].astype('category')
df['size'] = df['size'].cat.set_categories(['small', 'medium', 'large'], ordered=True)
print(df['size'].sort_values())

But the output is not sorted as expected. What is the likely problem?

medium

A. The set_categories method does not sort the data.

B. The categories were not set as ordered before conversion.

C. You must convert to string type before sorting.

D. The sort_values() method does not work on categorical data.

Solution

1. Step 1: Understand what set_categories does

   It sets the categories and order but does not reorder the actual data.

2. Step 2: How sorting works on ordered categorical

   Sorting uses the category order, but sort_values() sorts rows, not the category levels themselves.

3. Final Answer:

   The set_categories method does not sort the data. -> Option A

4. Quick Check:

   set_categories() sets order, doesn't sort data [OK]

Hint: set_categories sets order, sort_values sorts rows [OK]

Common Mistakes:

• Expecting set_categories to reorder data automatically
• Thinking sort_values fails on categorical columns
• Trying to convert to string before sorting

5. You have a large DataFrame with a column 'city' containing many repeated city names. You want to reduce memory usage and speed up filtering for cities in a specific order: 'New York', 'Los Angeles', 'Chicago'. Which approach is best?

hard

A. Keep 'city' as string type and sort the DataFrame by 'city' alphabetically

B. Convert 'city' to categorical with ordered categories: ['New York', 'Los Angeles', 'Chicago']

C. Convert 'city' to numeric codes without setting categories

D. Remove duplicate city names from the DataFrame

Solution

1. Step 1: Why use categorical with order

   Ordered categorical saves memory and allows meaningful sorting/filtering by specified city order.

2. Step 2: Why other options are less suitable

   Keeping strings uses more memory; numeric codes without categories lose order meaning; removing duplicates loses data.

3. Final Answer:

   Convert 'city' to categorical with ordered categories: ['New York', 'Los Angeles', 'Chicago'] -> Option B

4. Quick Check:

   Ordered categorical = A [OK]

Hint: Use ordered categorical for memory and custom order [OK]

Common Mistakes:

• Not setting categories order
• Using numeric codes without categories
• Removing duplicates instead of optimizing type