Why categorical type matters in Pandas - Performance Analysis

Time Complexity: Why categorical type matters

O(n)

Understanding Time Complexity

We want to see how using the categorical type in pandas affects the time it takes to do operations.

How does changing data type change the work pandas must do?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

# Create a large DataFrame with repeated categories
n = 100000
categories = ['apple', 'banana', 'cherry']
data = pd.DataFrame({
    'fruit': ['apple', 'banana', 'cherry'] * (n // 3)
})

# Convert to categorical type
cat_data = data['fruit'].astype('category')

# Count the occurrences of each category
counts = cat_data.value_counts()

This code creates a large list of fruits, converts it to a categorical type, and counts how many times each fruit appears.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Counting occurrences by scanning all items in the column.
• How many times: Once over all n rows to count.

How Execution Grows With Input

Counting needs to look at each item once, so work grows as the list grows.

Input Size (n)	Approx. Operations
10	10 checks
100	100 checks
1000	1000 checks

Pattern observation: The work grows directly with the number of rows.

Final Time Complexity

Time Complexity: O(n)

This means the time to count grows in a straight line as the data size grows.

Common Mistake

[X] Wrong: "Using categorical type makes counting instant or constant time."

[OK] Correct: Even with categories, pandas must still look at every item once to count them all.

Interview Connect

Understanding how data types affect operation speed helps you write faster code and explain your choices clearly.

Self-Check

"What if we had many more unique categories instead of just a few? How would the time complexity change?"

Practice

1. Why is using the categorical data type in pandas helpful when you have a column with many repeated text values?

easy

A. It converts text to numbers for calculations.

B. It automatically sorts the data alphabetically.

C. It saves memory by storing repeated values efficiently.

D. It removes duplicate rows from the DataFrame.

Solution

1. Step 1: Understand memory usage of text data

   Repeated text values take more memory because each string is stored separately.

2. Step 2: How categorical type optimizes memory

   Categorical type stores unique values once and uses codes to represent them, saving memory.

3. Final Answer:

   It saves memory by storing repeated values efficiently. -> Option C

4. Quick Check:

   Memory saving = A [OK]

Hint: Categorical stores unique values once, saving memory [OK]

Common Mistakes:

• Thinking categorical sorts data automatically
• Believing categorical removes duplicates
• Assuming categorical converts text to numbers for math

2. Which of the following is the correct way to convert a pandas column df['color'] to categorical type?

easy

A. df['color'] = df['color'].convert('category')

B. df['color'] = pd.to_numeric(df['color'])

C. df['color'] = df['color'].astype('string')

D. df['color'] = df['color'].astype('category')

Solution

1. Step 1: Identify correct pandas method for type conversion

   The astype() method is used to change data types in pandas.

2. Step 2: Check correct argument for categorical type

   Passing the string 'category' to astype() converts the column to categorical type.

3. Final Answer:

   df['color'] = df['color'].astype('category') -> Option D

4. Quick Check:

   astype('category') = D [OK]

Hint: Use astype('category') to convert column type [OK]

Common Mistakes:

• Using pd.to_numeric for text columns
• Trying astype('string') instead of 'category'
• Using a non-existent convert() method

3. Given this code:

import pandas as pd
colors = ['red', 'blue', 'red', 'green', 'blue', 'blue']
df = pd.DataFrame({'color': colors})
df['color'] = df['color'].astype('category')
print(df['color'].cat.codes.tolist())

What is the output?

medium

A. [2, 0, 2, 1, 0, 0]

B. [1, 2, 1, 3, 2, 2]

C. [0, 1, 0, 2, 1, 1]

D. [0, 0, 0, 0, 0, 0]

Solution

1. Step 1: Understand category codes assignment

   Categories are assigned codes in alphabetical order: 'blue' = 0, 'green' = 1, 'red' = 2.

2. Step 2: Check actual categories order

   By default, pandas assigns categories in alphabetical order, not order of appearance.

3. Step 3: Map colors to codes

   Mapping list: 'red'->2, 'blue'->0, 'red'->2, 'green'->1, 'blue'->0, 'blue'->0.

4. Final Answer:

   [2, 0, 2, 1, 0, 0] -> Option A

5. Quick Check:

   Category codes = A [OK]

Hint: Category codes start at 0 in alphabetical order [OK]

Common Mistakes:

• Assuming order of appearance for codes
• Thinking codes start at 1
• Confusing category codes with original values

4. You run this code:

df['size'] = ['small', 'medium', 'large', 'medium']
df['size'] = df['size'].astype('category')
df['size'] = df['size'].cat.set_categories(['small', 'medium', 'large'], ordered=True)
print(df['size'].sort_values())

But the output is not sorted as expected. What is the likely problem?

medium

A. The set_categories method does not sort the data.

B. The categories were not set as ordered before conversion.

C. You must convert to string type before sorting.

D. The sort_values() method does not work on categorical data.

Solution

1. Step 1: Understand what set_categories does

   It sets the categories and order but does not reorder the actual data.

2. Step 2: How sorting works on ordered categorical

   Sorting uses the category order, but sort_values() sorts rows, not the category levels themselves.

3. Final Answer:

   The set_categories method does not sort the data. -> Option A

4. Quick Check:

   set_categories() sets order, doesn't sort data [OK]

Hint: set_categories sets order, sort_values sorts rows [OK]

Common Mistakes:

• Expecting set_categories to reorder data automatically
• Thinking sort_values fails on categorical columns
• Trying to convert to string before sorting

5. You have a large DataFrame with a column 'city' containing many repeated city names. You want to reduce memory usage and speed up filtering for cities in a specific order: 'New York', 'Los Angeles', 'Chicago'. Which approach is best?

hard

A. Keep 'city' as string type and sort the DataFrame by 'city' alphabetically

B. Convert 'city' to categorical with ordered categories: ['New York', 'Los Angeles', 'Chicago']

C. Convert 'city' to numeric codes without setting categories

D. Remove duplicate city names from the DataFrame

Solution

1. Step 1: Why use categorical with order

   Ordered categorical saves memory and allows meaningful sorting/filtering by specified city order.

2. Step 2: Why other options are less suitable

   Keeping strings uses more memory; numeric codes without categories lose order meaning; removing duplicates loses data.

3. Final Answer:

   Convert 'city' to categorical with ordered categories: ['New York', 'Los Angeles', 'Chicago'] -> Option B

4. Quick Check:

   Ordered categorical = A [OK]

Hint: Use ordered categorical for memory and custom order [OK]

Common Mistakes:

• Not setting categories order
• Using numeric codes without categories
• Removing duplicates instead of optimizing type