Why categorical type matters in Pandas - Challenge Your Understanding

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of memory usage with categorical vs object type

Consider a pandas DataFrame with a column of repeated string values. What will be the approximate memory usage difference when the column is converted to categorical type compared to keeping it as object type?

Pandas

import pandas as pd
import numpy as np

size = 100000
values = ['apple', 'banana', 'cherry', 'date']
data = pd.DataFrame({'fruit': np.random.choice(values, size)})

mem_before = data.memory_usage(deep=True).sum()
data['fruit'] = data['fruit'].astype('category')
mem_after = data.memory_usage(deep=True).sum()

print(mem_before, mem_after)

A Code raises a TypeError because categorical conversion is invalid

B mem_after is about the same as mem_before, no significant difference

C mem_after is significantly smaller than mem_before, roughly 5-10 times less memory used

D mem_after is larger than mem_before because categorical adds overhead

💡 Hint

Think about how categorical stores repeated values as codes instead of full strings.

Categorical type stores repeated values as integer codes and a separate categories list, which reduces memory usage significantly compared to storing full strings repeatedly.

❓ data_output

intermediate

Number of unique categories after conversion

Given a pandas Series with repeated string values, what will be the number of categories after converting it to categorical type?

Pandas

import pandas as pd
s = pd.Series(['red', 'blue', 'red', 'green', 'blue', 'blue'])
cat = s.astype('category')
print(len(cat.cat.categories))

A 6

B Raises AttributeError

C 1

D 3

💡 Hint

Count unique values in the original Series.

Categorical categories are the unique values in the Series. Here, 'red', 'blue', and 'green' are the unique values, so 3 categories.

🔧 Debug

advanced

Why does this code raise a TypeError?

This code tries to assign a new value to a categorical column but raises a TypeError. Why?

Pandas

import pandas as pd
s = pd.Series(['small', 'medium', 'large'], dtype='category')
s[0] = 'extra large'

A Because 'extra large' is not in the categories, assignment fails

B Because categorical columns are immutable and cannot be changed

C Because 'extra large' is a string and categorical only accepts integers

D Because pandas Series do not support item assignment

💡 Hint

Think about allowed values in categorical data.

You cannot assign a value not in the categories to a categorical Series. It raises a TypeError.

🚀 Application

advanced

Choosing categorical type for performance

You have a DataFrame with a column 'city' containing 1 million rows but only 50 unique city names. Which benefit will you get by converting 'city' to categorical type?

A Slower filtering but faster sorting on 'city'

B Reduced memory usage and faster groupby operations on 'city'

C No benefit because 50 unique values is too many for categorical

D Increased memory usage due to category overhead

💡 Hint

Think about how categorical stores repeated values and how groupby uses categories.

Categorical reduces memory by storing repeated values as codes and speeds up groupby by working on integer codes instead of strings.

🧠 Conceptual

expert

Why does sorting a categorical column with ordered=True differ from unordered?

What is the main difference in sorting behavior between a pandas categorical column with ordered=True and one with ordered=False?

A With ordered=True, sorting respects the category order; with ordered=False, sorting is by category codes

B With ordered=True, sorting is faster; with ordered=False, sorting raises an error

C With ordered=True, sorting is random; with ordered=False, sorting respects category order

D There is no difference; both sort alphabetically

💡 Hint

Think about what ordered categories mean for comparison.

Ordered categorical means categories have a defined order used in sorting. Unordered categorical sorts by the underlying codes.

Practice

1. Why is using the categorical data type in pandas helpful when you have a column with many repeated text values?

easy

A. It converts text to numbers for calculations.

B. It automatically sorts the data alphabetically.

C. It saves memory by storing repeated values efficiently.

D. It removes duplicate rows from the DataFrame.

Solution

1. Step 1: Understand memory usage of text data

   Repeated text values take more memory because each string is stored separately.

2. Step 2: How categorical type optimizes memory

   Categorical type stores unique values once and uses codes to represent them, saving memory.

3. Final Answer:

   It saves memory by storing repeated values efficiently. -> Option C

4. Quick Check:

   Memory saving = A [OK]

Hint: Categorical stores unique values once, saving memory [OK]

Common Mistakes:

• Thinking categorical sorts data automatically
• Believing categorical removes duplicates
• Assuming categorical converts text to numbers for math

2. Which of the following is the correct way to convert a pandas column df['color'] to categorical type?

easy

A. df['color'] = df['color'].convert('category')

B. df['color'] = pd.to_numeric(df['color'])

C. df['color'] = df['color'].astype('string')

D. df['color'] = df['color'].astype('category')

Solution

1. Step 1: Identify correct pandas method for type conversion

   The astype() method is used to change data types in pandas.

2. Step 2: Check correct argument for categorical type

   Passing the string 'category' to astype() converts the column to categorical type.

3. Final Answer:

   df['color'] = df['color'].astype('category') -> Option D

4. Quick Check:

   astype('category') = D [OK]

Hint: Use astype('category') to convert column type [OK]

Common Mistakes:

• Using pd.to_numeric for text columns
• Trying astype('string') instead of 'category'
• Using a non-existent convert() method

3. Given this code:

import pandas as pd
colors = ['red', 'blue', 'red', 'green', 'blue', 'blue']
df = pd.DataFrame({'color': colors})
df['color'] = df['color'].astype('category')
print(df['color'].cat.codes.tolist())

What is the output?

medium

A. [2, 0, 2, 1, 0, 0]

B. [1, 2, 1, 3, 2, 2]

C. [0, 1, 0, 2, 1, 1]

D. [0, 0, 0, 0, 0, 0]

Solution

1. Step 1: Understand category codes assignment

   Categories are assigned codes in alphabetical order: 'blue' = 0, 'green' = 1, 'red' = 2.

2. Step 2: Check actual categories order

   By default, pandas assigns categories in alphabetical order, not order of appearance.

3. Step 3: Map colors to codes

   Mapping list: 'red'->2, 'blue'->0, 'red'->2, 'green'->1, 'blue'->0, 'blue'->0.

4. Final Answer:

   [2, 0, 2, 1, 0, 0] -> Option A

5. Quick Check:

   Category codes = A [OK]

Hint: Category codes start at 0 in alphabetical order [OK]

Common Mistakes:

• Assuming order of appearance for codes
• Thinking codes start at 1
• Confusing category codes with original values

4. You run this code:

df['size'] = ['small', 'medium', 'large', 'medium']
df['size'] = df['size'].astype('category')
df['size'] = df['size'].cat.set_categories(['small', 'medium', 'large'], ordered=True)
print(df['size'].sort_values())

But the output is not sorted as expected. What is the likely problem?

medium

A. The set_categories method does not sort the data.

B. The categories were not set as ordered before conversion.

C. You must convert to string type before sorting.

D. The sort_values() method does not work on categorical data.

Solution

1. Step 1: Understand what set_categories does

   It sets the categories and order but does not reorder the actual data.

2. Step 2: How sorting works on ordered categorical

   Sorting uses the category order, but sort_values() sorts rows, not the category levels themselves.

3. Final Answer:

   The set_categories method does not sort the data. -> Option A

4. Quick Check:

   set_categories() sets order, doesn't sort data [OK]

Hint: set_categories sets order, sort_values sorts rows [OK]

Common Mistakes:

• Expecting set_categories to reorder data automatically
• Thinking sort_values fails on categorical columns
• Trying to convert to string before sorting

5. You have a large DataFrame with a column 'city' containing many repeated city names. You want to reduce memory usage and speed up filtering for cities in a specific order: 'New York', 'Los Angeles', 'Chicago'. Which approach is best?

hard

A. Keep 'city' as string type and sort the DataFrame by 'city' alphabetically

B. Convert 'city' to categorical with ordered categories: ['New York', 'Los Angeles', 'Chicago']

C. Convert 'city' to numeric codes without setting categories

D. Remove duplicate city names from the DataFrame

Solution

1. Step 1: Why use categorical with order

   Ordered categorical saves memory and allows meaningful sorting/filtering by specified city order.

2. Step 2: Why other options are less suitable

   Keeping strings uses more memory; numeric codes without categories lose order meaning; removing duplicates loses data.

3. Final Answer:

   Convert 'city' to categorical with ordered categories: ['New York', 'Los Angeles', 'Chicago'] -> Option B

4. Quick Check:

   Ordered categorical = A [OK]

Hint: Use ordered categorical for memory and custom order [OK]

Common Mistakes:

• Not setting categories order
• Using numeric codes without categories
• Removing duplicates instead of optimizing type