Memory savings with categoricals in Pandas - Mini Project: Build & Apply

Memory savings with categoricals

📖 Scenario: You work in a company that collects survey data about favorite fruits from many people. The data is stored in a table with many repeated fruit names. You want to save computer memory by using a special data type called categorical.

🎯 Goal: You will create a pandas DataFrame with fruit names, convert the fruit column to a categorical type, and compare memory usage before and after. This helps you understand how categoricals save memory.

📋 What You'll Learn

Create a pandas DataFrame with a column named fruit containing repeated fruit names.

Create a variable called memory_before to store the memory usage of the fruit column before conversion.

Convert the fruit column to categorical type and store it back in the DataFrame.

Create a variable called memory_after to store the memory usage of the fruit column after conversion.

Print the values of memory_before and memory_after.

💡 Why This Matters

🌍 Real World

In real data projects, large datasets often have repeated text values. Using categorical data types saves memory and speeds up analysis.

💼 Career

Data scientists and analysts use categoricals to optimize memory and performance when working with big data in pandas.

1

Create the initial DataFrame

Import pandas as pd and create a DataFrame called df with one column named fruit containing these exact values in order: 'apple', 'banana', 'apple', 'orange', 'banana', 'apple'.

Pandas

# Import pandas and create the DataFrame df with the fruit column
# Your code here

Hint

Use pd.DataFrame and pass a dictionary with key 'fruit' and the list of fruits as value.

2

Measure memory usage before conversion

Create a variable called memory_before and set it to the memory usage of the fruit column in df using the memory_usage(deep=True).sum() method.

Pandas

import pandas as pd

df = pd.DataFrame({
    'fruit': ['apple', 'banana', 'apple', 'orange', 'banana', 'apple']
})

# Create memory_before variable to store memory usage of fruit column
# Your code here

Hint

Use df['fruit'].memory_usage(deep=True) to get memory in bytes.

3

Convert the fruit column to categorical

Convert the fruit column in df to categorical type using astype('category') and assign it back to df['fruit']. Then create a variable called memory_after and set it to the memory usage of the new categorical fruit column using memory_usage(deep=True).sum().

Pandas

import pandas as pd

df = pd.DataFrame({
    'fruit': ['apple', 'banana', 'apple', 'orange', 'banana', 'apple']
})

memory_before = df['fruit'].memory_usage(deep=True)

# Convert fruit column to categorical and create memory_after variable
# Your code here

Hint

Use astype('category') to convert the column. Then measure memory usage again.

4

Print memory usage before and after

Print the values of memory_before and memory_after on separate lines using two print statements.

Pandas

import pandas as pd

df = pd.DataFrame({
    'fruit': ['apple', 'banana', 'apple', 'orange', 'banana', 'apple']
})

memory_before = df['fruit'].memory_usage(deep=True)

df['fruit'] = df['fruit'].astype('category')

memory_after = df['fruit'].memory_usage(deep=True)

# Print memory_before and memory_after
# Your code here

Hint

Use two print statements: print(memory_before) and print(memory_after).

Practice

1.

What is the main benefit of converting a pandas column to category type?

easy

A. It sorts the data automatically

B. It saves memory by storing repeated values efficiently

C. It converts numbers to strings

D. It removes missing values

Solution

1. Step 1: Understand what category type does

   The category type stores repeated text values once and uses codes internally.

2. Step 2: Identify the main benefit

   This reduces memory usage because repeated strings are not stored multiple times.

3. Final Answer:

   It saves memory by storing repeated values efficiently -> Option B

4. Quick Check:

   Memory savings = A [OK]

Hint: Remember: categories store repeated text once [OK]

Common Mistakes:

• Thinking it sorts data automatically
• Confusing category with string conversion
• Assuming it removes missing values

2.

Which of the following is the correct way to convert a pandas DataFrame column df['color'] to a categorical type?

import pandas as pd
df = pd.DataFrame({'color': ['red', 'blue', 'red', 'green']})

easy

A. df['color'] = df['color'].astype('category')

B. df['color'] = df['color'].to_category()

C. df['color'] = pd.category(df['color'])

D. df['color'] = df['color'].convert('category')

Solution

1. Step 1: Recall pandas syntax for type conversion

   Use astype('category') to convert a column to categorical type.

2. Step 2: Check each option

   Only df['color'] = df['color'].astype('category') uses the correct astype method with 'category' string.

3. Final Answer:

   df['color'] = df['color'].astype('category') -> Option A

4. Quick Check:

   astype('category') = A [OK]

Hint: Use astype('category') to convert columns [OK]

Common Mistakes:

• Using non-existent methods like to_category()
• Trying to call pd.category() which does not exist
• Using convert() instead of astype()

3.

What will be the output of the following code?

import pandas as pd
import sys

df = pd.DataFrame({'fruit': ['apple', 'banana', 'apple', 'banana', 'cherry']})
mem_before = df['fruit'].memory_usage()
df['fruit'] = df['fruit'].astype('category')
mem_after = df['fruit'].memory_usage()
print(mem_before, mem_after)

medium

A. 112 80

B. 80 80

C. 80 112

D. 112 112

Solution

1. Step 1: Calculate memory before conversion

   Strings take more memory; here memory_usage() returns 112 bytes before conversion.

2. Step 2: Calculate memory after conversion to category

   Category stores codes and unique values separately, reducing memory to 80 bytes.

3. Final Answer:

   112 80 -> Option A

4. Quick Check:

   Memory before > after = C [OK]

Hint: Memory decreases after astype('category') [OK]

Common Mistakes:

• Assuming memory stays the same
• Thinking memory increases after conversion
• Confusing order of printed values

4.

Identify the error in the following code that tries to convert a column to categorical and save memory:

import pandas as pd
df = pd.DataFrame({'city': ['NY', 'LA', 'NY', 'SF']})
df['city'] = df['city'].astype(category)

medium

A. DataFrame creation syntax is wrong

B. astype cannot convert to category type

C. Missing quotes around 'category' in astype

D. Column 'city' does not exist

Solution

1. Step 1: Check astype argument

   The argument to astype must be a string 'category', but here category is unquoted, causing a NameError.

2. Step 2: Verify other parts

   DataFrame creation and column name are correct; astype supports 'category'.

3. Final Answer:

   Missing quotes around 'category' in astype -> Option C

4. Quick Check:

   Quotes needed for 'category' string [OK]

Hint: Always quote 'category' in astype('category') [OK]

Common Mistakes:

• Forgetting quotes around 'category'
• Thinking astype can't convert to category
• Assuming column name is wrong

5.

You have a DataFrame with a column df['status'] containing 1 million rows with only 3 unique values: 'new', 'in_progress', 'done'. How can you reduce memory usage effectively?

hard

A. Replace values with integers manually without category

B. Convert df['status'] to string type

C. Drop the df['status'] column

D. Convert df['status'] to category type

Solution

1. Step 1: Understand the data characteristics

   With only 3 unique values repeated 1 million times, storing as category saves memory by storing unique values once.

2. Step 2: Compare options

   Converting to string keeps large memory; dropping column loses data; manual integer replacement is error-prone and less efficient.

3. Final Answer:

   Convert df['status'] to category type -> Option D

4. Quick Check:

   Few unique values + large rows = category [OK]

Hint: Use category for few unique values in large data [OK]

Common Mistakes:

• Using string type which uses more memory
• Dropping column losing important data
• Manual integer replacement without category benefits