Data type optimization in Data Analysis Python - Step-by-Step Execution

Concept Flow - Data type optimization

Load DataFrame

↓

Check current data types

↓

Identify columns to optimize

↓

Convert columns to smaller types

↓

Check memory usage before and after

↓

Use optimized DataFrame for analysis

Start with a DataFrame, check its data types, convert columns to smaller types to save memory, then use the optimized DataFrame.

Execution Sample

Data Analysis Python

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'A': range(1000),
    'B': np.random.randint(0, 100, 1000),
    'C': np.random.choice(['X', 'Y', 'Z'], 1000)
})

print(df.memory_usage(deep=True))
df['B'] = df['B'].astype('int8')
df['C'] = df['C'].astype('category')
print(df.memory_usage(deep=True))

This code creates a DataFrame, shows memory usage, converts columns to smaller types, then shows reduced memory usage.

Execution Table

Step	Action	Column 'B' dtype	Column 'C' dtype	Memory Usage (bytes)
1	Create DataFrame	int64	object	24000
2	Check memory usage before optimization	int64	object	24000
3	Convert 'B' to int8	int8	object	16000
4	Convert 'C' to category	int8	category	8000
5	Check memory usage after optimization	int8	category	8000
6	End	int8	category	8000

💡 Memory usage reduced by converting 'B' to int8 and 'C' to category, saving space.

Variable Tracker

Variable	Start	After Step 3	After Step 4	Final
df['B'].dtype	int64	int8	int8	int8
df['C'].dtype	object	object	category	category
Memory Usage (bytes)	24000	16000	8000	8000

Key Moments - 2 Insights

Why does converting 'C' to category reduce memory usage?

Because 'category' stores unique values once and uses integer codes, reducing repeated string storage as shown in steps 4 and 5.

Why choose int8 for column 'B'?

Because values in 'B' fit in the range of int8 (0 to 127), so it uses less memory than int64, as seen in step 3.

Visual Quiz - 3 Questions

Look at the execution table, what is the dtype of column 'B' after step 3?

A int64

B int8

C float64

D category

💡 Hint

Check the 'Column B dtype' column in row for step 3.

At which step does the memory usage drop below 10000 bytes?

A Step 2

B Step 3

C Step 4

D Step 1

💡 Hint

Look at the 'Memory Usage' column in the execution table rows.

If column 'B' had values larger than 127, what would happen if we convert to int8?

A Values would be truncated or wrap around causing errors

B Memory usage would increase

C Conversion would automatically use int64

D No change in memory usage

💡 Hint

Think about int8 range and what happens if values exceed it.

Concept Snapshot

Data type optimization saves memory by converting columns to smaller types.
Use .astype() to change types like int64 to int8 or object to category.
Check memory usage with .memory_usage(deep=True).
Optimized types reduce DataFrame size and speed up analysis.

Full Transcript

We start with a DataFrame having columns with default data types. We check memory usage, then convert numeric columns to smaller integer types if values fit the range. For text columns, converting to category type saves memory by storing unique values once. After conversions, memory usage drops significantly. This process helps handle large data efficiently.

Practice

1. Why is data type optimization important when working with large datasets in pandas?

easy

A. It sorts the data in ascending order.

B. It changes the data values to strings.

C. It deletes unnecessary columns automatically.

D. It reduces memory usage and speeds up data processing.

Solution

1. Step 1: Understand data type optimization purpose

   Data type optimization aims to reduce the memory used by data and improve processing speed.

2. Step 2: Identify the correct benefit

   Reducing memory and speeding up processing matches the purpose, unlike changing values or sorting.

3. Final Answer:

   It reduces memory usage and speeds up data processing. -> Option D

4. Quick Check:

   Data type optimization = reduce memory and speed up [OK]

Hint: Think about saving memory and faster work [OK]

Common Mistakes:

• Confusing optimization with sorting or deleting data
• Thinking it changes data values instead of types
• Assuming it automatically cleans data

2. Which pandas method is used to change the data type of a DataFrame column?

easy

A. astype()

B. convert()

C. change_type()

D. modify_dtype()

Solution

1. Step 1: Recall pandas method for type conversion

   The pandas method to change a column's data type is astype().

2. Step 2: Verify other options

   convert(), change_type(), and modify_dtype() are not valid pandas methods.

3. Final Answer:

   astype() -> Option A

4. Quick Check:

   pandas type change = astype() [OK]

Hint: Remember pandas astype() for type changes [OK]

Common Mistakes:

• Using non-existent methods like convert()
• Confusing with methods for data cleaning
• Trying to assign types directly without astype()

3. Given the code below, what will be the memory usage of the DataFrame after optimization?

import pandas as pd
import numpy as np

df = pd.DataFrame({
  'A': np.arange(1000, dtype='int64'),
  'B': ['cat', 'dog', 'mouse', 'cat'] * 250
})

memory_before = df.memory_usage(deep=True).sum()
df['B'] = df['B'].astype('category')
memory_after = df.memory_usage(deep=True).sum()

print(memory_after < memory_before)

medium

A. True, memory usage decreases after converting to category.

B. False, memory usage increases after converting to category.

C. True, memory usage stays the same after converting to category.

D. Error, astype('category') is invalid.

Solution

1. Step 1: Understand memory usage before optimization

   Column 'B' is object type with repeated strings, using more memory.

2. Step 2: Effect of converting to category

   Converting 'B' to category stores unique values once, reducing memory.

3. Final Answer:

   True, memory usage decreases after converting to category. -> Option A

4. Quick Check:

   Category type reduces memory = True [OK]

Hint: Categorical reduces repeated string memory [OK]

Common Mistakes:

• Thinking category increases memory
• Assuming memory stays same after astype
• Believing astype('category') causes error

4. Identify the error in the following code snippet that tries to optimize data types:

import pandas as pd

df = pd.DataFrame({'col': [1, 2, 3, 4]})
df['col'] = df['col'].astype('int8')
df['col'] = df['col'].astype('category')
df['col'] = df['col'].astype('int16')

medium

A. int8 is not a valid data type in pandas.

B. Cannot convert from category back to int16 directly.

C. astype() cannot be used multiple times on the same column.

D. DataFrame creation syntax is incorrect.

Solution

1. Step 1: Check data type conversions

   Converting from int8 to category is valid, but category stores codes, not original ints.

2. Step 2: Conversion from category to int16

   Directly converting category to int16 causes error because category is not numeric type.

3. Final Answer:

   Cannot convert from category back to int16 directly. -> Option B

4. Quick Check:

   Category to int conversion invalid [OK]

Hint: Category to int needs .cat.codes first [OK]

Common Mistakes:

• Assuming int8 is invalid
• Thinking astype can chain without issues
• Believing DataFrame creation is wrong

5. You have a DataFrame with a column 'status' containing only three unique string values repeated many times. Which is the best way to optimize memory usage for this column while keeping it easy to analyze?

hard

A. Leave 'status' as object type because conversion wastes time.

B. Convert 'status' to integer type directly with astype('int32').

C. Convert 'status' to a categorical type using astype('category').

D. Convert 'status' to float type with astype('float64').

Solution

1. Step 1: Analyze the column data

   'status' has few unique strings repeated many times, ideal for categorical type.

2. Step 2: Choose the best optimization

   astype('category') reduces memory by storing unique values once and codes for repeats.

3. Final Answer:

   Convert 'status' to a categorical type using astype('category'). -> Option C

4. Quick Check:

   Few unique strings = use category [OK]

Hint: Few unique strings? Use category type [OK]

Common Mistakes:

• Trying to convert strings directly to int or float
• Ignoring memory benefits of categorical
• Thinking object type is always best