Data type optimization in Data Analysis Python - Mini Project: Build & Apply

Data type optimization

📖 Scenario: You work as a data analyst for a retail company. You have a dataset of product sales with product IDs, quantities sold, and prices. The dataset is large, and you want to optimize the data types to save memory and improve performance.

🎯 Goal: You will create a dictionary with product sales data, add a configuration for the maximum quantity threshold, optimize the data types by converting values to smaller types where possible, and finally print the optimized data.

📋 What You'll Learn

Create a dictionary called sales_data with exact entries for product IDs as strings and their quantities and prices as integers and floats.

Create a variable called max_quantity set to 1000.

Use a dictionary comprehension to create a new dictionary called optimized_sales where quantities greater than max_quantity are converted to float and others to int, and prices are converted to float with two decimals.

Print the optimized_sales dictionary.

💡 Why This Matters

🌍 Real World

Optimizing data types helps reduce memory use and speeds up data processing in real-world data analysis tasks.

💼 Career

Data analysts and scientists often optimize data types to handle large datasets efficiently and improve performance.

1

Create the sales data dictionary

Create a dictionary called sales_data with these exact entries: 'P001': {'quantity': 500, 'price': 19.99}, 'P002': {'quantity': 1500, 'price': 29.95}, 'P003': {'quantity': 300, 'price': 9.99}.

Data Analysis Python

# Create the sales_data dictionary with exact entries
# Your code here

Hint

Use a dictionary with product IDs as keys and dictionaries as values containing 'quantity' and 'price'.

2

Set the maximum quantity threshold

Create a variable called max_quantity and set it to 1000.

Data Analysis Python

sales_data = {
    'P001': {'quantity': 500, 'price': 19.99},
    'P002': {'quantity': 1500, 'price': 29.95},
    'P003': {'quantity': 300, 'price': 9.99}
}
# Create max_quantity variable and set it to 1000
# Your code here

Hint

Just assign the number 1000 to the variable max_quantity.

3

Optimize data types with dictionary comprehension

Use a dictionary comprehension to create a new dictionary called optimized_sales. For each product in sales_data, convert quantity to float if it is greater than max_quantity, otherwise keep it as int. Convert price to float rounded to two decimals.

Data Analysis Python

sales_data = {
    'P001': {'quantity': 500, 'price': 19.99},
    'P002': {'quantity': 1500, 'price': 29.95},
    'P003': {'quantity': 300, 'price': 9.99}
}
max_quantity = 1000
# Create optimized_sales dictionary using dictionary comprehension
# Your code here

Hint

Use a dictionary comprehension with for product, info in sales_data.items(). Use a conditional expression to convert quantity.

4

Print the optimized sales data

Print the optimized_sales dictionary.

Data Analysis Python

sales_data = {
    'P001': {'quantity': 500, 'price': 19.99},
    'P002': {'quantity': 1500, 'price': 29.95},
    'P003': {'quantity': 300, 'price': 9.99}
}
max_quantity = 1000
optimized_sales = {
    product: {
        'quantity': float(info['quantity']) if info['quantity'] > max_quantity else int(info['quantity']),
        'price': round(float(info['price']), 2)
    }
    for product, info in sales_data.items()
}
# Print optimized_sales dictionary
# Your code here

Hint

Use print(optimized_sales) to display the dictionary.

Practice

1. Why is data type optimization important when working with large datasets in pandas?

easy

A. It sorts the data in ascending order.

B. It changes the data values to strings.

C. It deletes unnecessary columns automatically.

D. It reduces memory usage and speeds up data processing.

Solution

1. Step 1: Understand data type optimization purpose

   Data type optimization aims to reduce the memory used by data and improve processing speed.

2. Step 2: Identify the correct benefit

   Reducing memory and speeding up processing matches the purpose, unlike changing values or sorting.

3. Final Answer:

   It reduces memory usage and speeds up data processing. -> Option D

4. Quick Check:

   Data type optimization = reduce memory and speed up [OK]

Hint: Think about saving memory and faster work [OK]

Common Mistakes:

• Confusing optimization with sorting or deleting data
• Thinking it changes data values instead of types
• Assuming it automatically cleans data

2. Which pandas method is used to change the data type of a DataFrame column?

easy

A. astype()

B. convert()

C. change_type()

D. modify_dtype()

Solution

1. Step 1: Recall pandas method for type conversion

   The pandas method to change a column's data type is astype().

2. Step 2: Verify other options

   convert(), change_type(), and modify_dtype() are not valid pandas methods.

3. Final Answer:

   astype() -> Option A

4. Quick Check:

   pandas type change = astype() [OK]

Hint: Remember pandas astype() for type changes [OK]

Common Mistakes:

• Using non-existent methods like convert()
• Confusing with methods for data cleaning
• Trying to assign types directly without astype()

3. Given the code below, what will be the memory usage of the DataFrame after optimization?

import pandas as pd
import numpy as np

df = pd.DataFrame({
  'A': np.arange(1000, dtype='int64'),
  'B': ['cat', 'dog', 'mouse', 'cat'] * 250
})

memory_before = df.memory_usage(deep=True).sum()
df['B'] = df['B'].astype('category')
memory_after = df.memory_usage(deep=True).sum()

print(memory_after < memory_before)

medium

A. True, memory usage decreases after converting to category.

B. False, memory usage increases after converting to category.

C. True, memory usage stays the same after converting to category.

D. Error, astype('category') is invalid.

Solution

1. Step 1: Understand memory usage before optimization

   Column 'B' is object type with repeated strings, using more memory.

2. Step 2: Effect of converting to category

   Converting 'B' to category stores unique values once, reducing memory.

3. Final Answer:

   True, memory usage decreases after converting to category. -> Option A

4. Quick Check:

   Category type reduces memory = True [OK]

Hint: Categorical reduces repeated string memory [OK]

Common Mistakes:

• Thinking category increases memory
• Assuming memory stays same after astype
• Believing astype('category') causes error

4. Identify the error in the following code snippet that tries to optimize data types:

import pandas as pd

df = pd.DataFrame({'col': [1, 2, 3, 4]})
df['col'] = df['col'].astype('int8')
df['col'] = df['col'].astype('category')
df['col'] = df['col'].astype('int16')

medium

A. int8 is not a valid data type in pandas.

B. Cannot convert from category back to int16 directly.

C. astype() cannot be used multiple times on the same column.

D. DataFrame creation syntax is incorrect.

Solution

1. Step 1: Check data type conversions

   Converting from int8 to category is valid, but category stores codes, not original ints.

2. Step 2: Conversion from category to int16

   Directly converting category to int16 causes error because category is not numeric type.

3. Final Answer:

   Cannot convert from category back to int16 directly. -> Option B

4. Quick Check:

   Category to int conversion invalid [OK]

Hint: Category to int needs .cat.codes first [OK]

Common Mistakes:

• Assuming int8 is invalid
• Thinking astype can chain without issues
• Believing DataFrame creation is wrong

5. You have a DataFrame with a column 'status' containing only three unique string values repeated many times. Which is the best way to optimize memory usage for this column while keeping it easy to analyze?

hard

A. Leave 'status' as object type because conversion wastes time.

B. Convert 'status' to integer type directly with astype('int32').

C. Convert 'status' to a categorical type using astype('category').

D. Convert 'status' to float type with astype('float64').

Solution

1. Step 1: Analyze the column data

   'status' has few unique strings repeated many times, ideal for categorical type.

2. Step 2: Choose the best optimization

   astype('category') reduces memory by storing unique values once and codes for repeats.

3. Final Answer:

   Convert 'status' to a categorical type using astype('category'). -> Option C

4. Quick Check:

   Few unique strings = use category [OK]

Hint: Few unique strings? Use category type [OK]

Common Mistakes:

• Trying to convert strings directly to int or float
• Ignoring memory benefits of categorical
• Thinking object type is always best