Memory savings with categoricals in Pandas - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Memory usage difference between object and categorical

Given a pandas DataFrame column with repeated string values, what is the approximate memory usage difference when converting it from object dtype to categorical dtype?

Pandas

import pandas as pd

col = pd.Series(['apple', 'banana', 'apple', 'banana', 'apple'] * 1000)
mem_obj = col.memory_usage(deep=True)
mem_cat = col.astype('category').memory_usage(deep=True)
print(mem_obj, mem_cat)

A mem_cat uses zero memory

B mem_obj is roughly 5 times larger than mem_cat

C mem_cat is roughly 5 times larger than mem_obj

D mem_obj and mem_cat have about the same memory usage

💡 Hint

Think about how categorical stores repeated strings once, while object stores each string separately.

Categorical dtype stores unique values once and uses integer codes for each entry, saving memory compared to object dtype which stores each string separately.

❓ data_output

intermediate

Memory usage after converting multiple columns to categorical

You have a DataFrame with 3 columns of repeated string categories. After converting all 3 columns to categorical dtype, what is the expected effect on total memory usage?

Pandas

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'A': ['cat', 'dog', 'cat', 'dog'] * 1000,
    'B': ['red', 'blue', 'red', 'blue'] * 1000,
    'C': ['small', 'large', 'small', 'large'] * 1000
})
mem_before = df.memory_usage(deep=True).sum()
df_cat = df.astype('category')
mem_after = df_cat.memory_usage(deep=True).sum()
print(mem_before, mem_after)

A Total memory usage decreases significantly after conversion

B Total memory usage increases after conversion

C Total memory usage stays the same

D Total memory usage becomes zero

💡 Hint

Each categorical column stores unique values once, reducing memory.

Converting multiple repeated string columns to categorical reduces memory usage significantly because unique values are stored once and data is stored as integer codes.

🔧 Debug

advanced

Why does this categorical conversion not reduce memory?

Consider this code snippet: import pandas as pd col = pd.Series(['unique' + str(i) for i in range(10000)]) col_cat = col.astype('category') print(col.memory_usage(deep=true), col_cat.memory_usage(deep=true)) Why does converting to categorical not reduce memory usage here?

Pandas

import pandas as pd
col = pd.Series(['unique' + str(i) for i in range(10000)])
col_cat = col.astype('category')
print(col.memory_usage(deep=True), col_cat.memory_usage(deep=True))

A Because pandas does not support categorical for string data

B Because categorical dtype always uses more memory than object dtype

C Because all values are unique, categorical stores all strings plus codes, increasing memory

D Because the code has a syntax error and does not run

💡 Hint

Think about how many unique values there are and what categorical stores.

When all values are unique, categorical stores all unique strings plus integer codes, so memory usage can be higher than object dtype which stores strings once each.

🧠 Conceptual

advanced

Understanding categorical dtype internals

Which statement best describes how pandas stores data internally when using categorical dtype?

A It converts all strings to numeric floats

B It stores all string values repeatedly as in object dtype

C It compresses strings using zip compression internally

D It stores an array of integer codes and a separate array of unique categories

💡 Hint

Think about how categorical reduces memory by avoiding repeated strings.

Categorical dtype stores data as integer codes referencing a separate array of unique categories, avoiding repeated storage of strings.

🚀 Application

expert

Optimizing memory for a large DataFrame with mixed data types

You have a large DataFrame with 1 million rows and these columns: - 'city' (string, 100 unique values) - 'temperature' (float) - 'date' (datetime) - 'status' (string, 3 unique values) Which memory optimization strategy will save the most memory without losing data?

A Convert 'city' and 'status' to categorical dtype, keep 'temperature' as float, and 'date' as datetime

B Convert 'temperature' to categorical dtype, keep others as is

C Convert all columns to categorical dtype

D Convert 'date' to string dtype, keep others as is

💡 Hint

Only columns with repeated string values benefit from categorical dtype.

Converting 'city' and 'status' to categorical saves memory because they have few unique values. 'temperature' should stay float for numeric operations. 'date' is best kept as datetime for time operations.

Practice

1.

What is the main benefit of converting a pandas column to category type?

easy

A. It sorts the data automatically

B. It saves memory by storing repeated values efficiently

C. It converts numbers to strings

D. It removes missing values

Solution

1. Step 1: Understand what category type does

   The category type stores repeated text values once and uses codes internally.

2. Step 2: Identify the main benefit

   This reduces memory usage because repeated strings are not stored multiple times.

3. Final Answer:

   It saves memory by storing repeated values efficiently -> Option B

4. Quick Check:

   Memory savings = A [OK]

Hint: Remember: categories store repeated text once [OK]

Common Mistakes:

• Thinking it sorts data automatically
• Confusing category with string conversion
• Assuming it removes missing values

2.

Which of the following is the correct way to convert a pandas DataFrame column df['color'] to a categorical type?

import pandas as pd
df = pd.DataFrame({'color': ['red', 'blue', 'red', 'green']})

easy

A. df['color'] = df['color'].astype('category')

B. df['color'] = df['color'].to_category()

C. df['color'] = pd.category(df['color'])

D. df['color'] = df['color'].convert('category')

Solution

1. Step 1: Recall pandas syntax for type conversion

   Use astype('category') to convert a column to categorical type.

2. Step 2: Check each option

   Only df['color'] = df['color'].astype('category') uses the correct astype method with 'category' string.

3. Final Answer:

   df['color'] = df['color'].astype('category') -> Option A

4. Quick Check:

   astype('category') = A [OK]

Hint: Use astype('category') to convert columns [OK]

Common Mistakes:

• Using non-existent methods like to_category()
• Trying to call pd.category() which does not exist
• Using convert() instead of astype()

3.

What will be the output of the following code?

import pandas as pd
import sys

df = pd.DataFrame({'fruit': ['apple', 'banana', 'apple', 'banana', 'cherry']})
mem_before = df['fruit'].memory_usage()
df['fruit'] = df['fruit'].astype('category')
mem_after = df['fruit'].memory_usage()
print(mem_before, mem_after)

medium

A. 112 80

B. 80 80

C. 80 112

D. 112 112

Solution

1. Step 1: Calculate memory before conversion

   Strings take more memory; here memory_usage() returns 112 bytes before conversion.

2. Step 2: Calculate memory after conversion to category

   Category stores codes and unique values separately, reducing memory to 80 bytes.

3. Final Answer:

   112 80 -> Option A

4. Quick Check:

   Memory before > after = C [OK]

Hint: Memory decreases after astype('category') [OK]

Common Mistakes:

• Assuming memory stays the same
• Thinking memory increases after conversion
• Confusing order of printed values

4.

Identify the error in the following code that tries to convert a column to categorical and save memory:

import pandas as pd
df = pd.DataFrame({'city': ['NY', 'LA', 'NY', 'SF']})
df['city'] = df['city'].astype(category)

medium

A. DataFrame creation syntax is wrong

B. astype cannot convert to category type

C. Missing quotes around 'category' in astype

D. Column 'city' does not exist

Solution

1. Step 1: Check astype argument

   The argument to astype must be a string 'category', but here category is unquoted, causing a NameError.

2. Step 2: Verify other parts

   DataFrame creation and column name are correct; astype supports 'category'.

3. Final Answer:

   Missing quotes around 'category' in astype -> Option C

4. Quick Check:

   Quotes needed for 'category' string [OK]

Hint: Always quote 'category' in astype('category') [OK]

Common Mistakes:

• Forgetting quotes around 'category'
• Thinking astype can't convert to category
• Assuming column name is wrong

5.

You have a DataFrame with a column df['status'] containing 1 million rows with only 3 unique values: 'new', 'in_progress', 'done'. How can you reduce memory usage effectively?

hard

A. Replace values with integers manually without category

B. Convert df['status'] to string type

C. Drop the df['status'] column

D. Convert df['status'] to category type

Solution

1. Step 1: Understand the data characteristics

   With only 3 unique values repeated 1 million times, storing as category saves memory by storing unique values once.

2. Step 2: Compare options

   Converting to string keeps large memory; dropping column loses data; manual integer replacement is error-prone and less efficient.

3. Final Answer:

   Convert df['status'] to category type -> Option D

4. Quick Check:

   Few unique values + large rows = category [OK]

Hint: Use category for few unique values in large data [OK]

Common Mistakes:

• Using string type which uses more memory
• Dropping column losing important data
• Manual integer replacement without category benefits