Memory savings with categoricals in Pandas - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a categorical data type in pandas?

A categorical data type in pandas is a way to store data that has a limited set of possible values, called categories. It saves memory by storing the data as integers internally and mapping them to the categories.

beginner

How does using categorical data types save memory compared to using strings?

Using categorical data types saves memory because it stores each unique value only once and uses integer codes to represent the data, instead of storing full strings repeatedly.

beginner

Which pandas method converts a column to a categorical type?

The pandas method to convert a column to categorical type is astype('category').

beginner

What is one common use case for converting data to categorical type?

A common use case is when a column has repeated values from a small set, like days of the week or product categories, to reduce memory and speed up operations.

beginner

True or False: Categorical data types can only be used with text data.

False. Categorical data types can be used with any data that has a limited set of values, including numbers and strings.

What does pandas store internally when using a categorical data type?

A Floating point numbers

B Integer codes representing categories

C Full string values for each row

D Boolean values

Pandas stores integer codes internally for categorical data, mapping them to the category values.

Which pandas function converts a column to categorical?

A astype('category')

B to_categorical()

C convert_category()

D category_convert()

The correct method is astype('category') to convert a column to categorical type.

Why do categorical types save memory?

A They remove missing values

B They compress data using zip

C They store unique values once and use codes

D They convert data to integers only

Categorical types save memory by storing unique values once and using integer codes for data.

Which data is best suited for categorical type?

A Unique IDs with many different values

B Continuous numerical data

C Random floating point numbers

D Repeated values from a small set

Categorical types are best for repeated values from a small set, like categories.

Can categorical types be used for numeric data?

A Yes, if the numbers represent categories

B No, only for text data

C Only for floating point numbers

D Only for boolean data

Categorical types can be used for numeric data if the numbers represent categories.

Explain how converting a pandas column to categorical type helps save memory.

Think about how repeated values are stored in normal strings versus categorical.

Describe a real-life example where using categorical data type would be beneficial.

Consider data with many repeated labels.

Practice

1.

What is the main benefit of converting a pandas column to category type?

easy

A. It sorts the data automatically

B. It saves memory by storing repeated values efficiently

C. It converts numbers to strings

D. It removes missing values

Solution

1. Step 1: Understand what category type does

   The category type stores repeated text values once and uses codes internally.

2. Step 2: Identify the main benefit

   This reduces memory usage because repeated strings are not stored multiple times.

3. Final Answer:

   It saves memory by storing repeated values efficiently -> Option B

4. Quick Check:

   Memory savings = A [OK]

Hint: Remember: categories store repeated text once [OK]

Common Mistakes:

• Thinking it sorts data automatically
• Confusing category with string conversion
• Assuming it removes missing values

2.

Which of the following is the correct way to convert a pandas DataFrame column df['color'] to a categorical type?

import pandas as pd
df = pd.DataFrame({'color': ['red', 'blue', 'red', 'green']})

easy

A. df['color'] = df['color'].astype('category')

B. df['color'] = df['color'].to_category()

C. df['color'] = pd.category(df['color'])

D. df['color'] = df['color'].convert('category')

Solution

1. Step 1: Recall pandas syntax for type conversion

   Use astype('category') to convert a column to categorical type.

2. Step 2: Check each option

   Only df['color'] = df['color'].astype('category') uses the correct astype method with 'category' string.

3. Final Answer:

   df['color'] = df['color'].astype('category') -> Option A

4. Quick Check:

   astype('category') = A [OK]

Hint: Use astype('category') to convert columns [OK]

Common Mistakes:

• Using non-existent methods like to_category()
• Trying to call pd.category() which does not exist
• Using convert() instead of astype()

3.

What will be the output of the following code?

import pandas as pd
import sys

df = pd.DataFrame({'fruit': ['apple', 'banana', 'apple', 'banana', 'cherry']})
mem_before = df['fruit'].memory_usage()
df['fruit'] = df['fruit'].astype('category')
mem_after = df['fruit'].memory_usage()
print(mem_before, mem_after)

medium

A. 112 80

B. 80 80

C. 80 112

D. 112 112

Solution

1. Step 1: Calculate memory before conversion

   Strings take more memory; here memory_usage() returns 112 bytes before conversion.

2. Step 2: Calculate memory after conversion to category

   Category stores codes and unique values separately, reducing memory to 80 bytes.

3. Final Answer:

   112 80 -> Option A

4. Quick Check:

   Memory before > after = C [OK]

Hint: Memory decreases after astype('category') [OK]

Common Mistakes:

• Assuming memory stays the same
• Thinking memory increases after conversion
• Confusing order of printed values

4.

Identify the error in the following code that tries to convert a column to categorical and save memory:

import pandas as pd
df = pd.DataFrame({'city': ['NY', 'LA', 'NY', 'SF']})
df['city'] = df['city'].astype(category)

medium

A. DataFrame creation syntax is wrong

B. astype cannot convert to category type

C. Missing quotes around 'category' in astype

D. Column 'city' does not exist

Solution

1. Step 1: Check astype argument

   The argument to astype must be a string 'category', but here category is unquoted, causing a NameError.

2. Step 2: Verify other parts

   DataFrame creation and column name are correct; astype supports 'category'.

3. Final Answer:

   Missing quotes around 'category' in astype -> Option C

4. Quick Check:

   Quotes needed for 'category' string [OK]

Hint: Always quote 'category' in astype('category') [OK]

Common Mistakes:

• Forgetting quotes around 'category'
• Thinking astype can't convert to category
• Assuming column name is wrong

5.

You have a DataFrame with a column df['status'] containing 1 million rows with only 3 unique values: 'new', 'in_progress', 'done'. How can you reduce memory usage effectively?

hard

A. Replace values with integers manually without category

B. Convert df['status'] to string type

C. Drop the df['status'] column

D. Convert df['status'] to category type

Solution

1. Step 1: Understand the data characteristics

   With only 3 unique values repeated 1 million times, storing as category saves memory by storing unique values once.

2. Step 2: Compare options

   Converting to string keeps large memory; dropping column loses data; manual integer replacement is error-prone and less efficient.

3. Final Answer:

   Convert df['status'] to category type -> Option D

4. Quick Check:

   Few unique values + large rows = category [OK]

Hint: Use category for few unique values in large data [OK]

Common Mistakes:

• Using string type which uses more memory
• Dropping column losing important data
• Manual integer replacement without category benefits