Data type optimization in Data Analysis Python - Step-by-Step Execution

Concept Flow - Data type optimization

Load DataFrame

↓

Check current data types

↓

Identify columns to optimize

↓

Convert columns to smaller types

↓

Check memory usage before and after

↓

Use optimized DataFrame for analysis

Start with a DataFrame, check its data types, convert columns to smaller types to save memory, then use the optimized DataFrame.

Execution Sample

Data Analysis Python

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'A': range(1000),
    'B': np.random.randint(0, 100, 1000),
    'C': np.random.choice(['X', 'Y', 'Z'], 1000)
})

print(df.memory_usage(deep=True))
df['B'] = df['B'].astype('int8')
df['C'] = df['C'].astype('category')
print(df.memory_usage(deep=True))

This code creates a DataFrame, shows memory usage, converts columns to smaller types, then shows reduced memory usage.

Execution Table

Step	Action	Column 'B' dtype	Column 'C' dtype	Memory Usage (bytes)
1	Create DataFrame	int64	object	24000
2	Check memory usage before optimization	int64	object	24000
3	Convert 'B' to int8	int8	object	16000
4	Convert 'C' to category	int8	category	8000
5	Check memory usage after optimization	int8	category	8000
6	End	int8	category	8000

💡 Memory usage reduced by converting 'B' to int8 and 'C' to category, saving space.

Variable Tracker

Variable	Start	After Step 3	After Step 4	Final
df['B'].dtype	int64	int8	int8	int8
df['C'].dtype	object	object	category	category
Memory Usage (bytes)	24000	16000	8000	8000

Key Moments - 2 Insights

Why does converting 'C' to category reduce memory usage?

Because 'category' stores unique values once and uses integer codes, reducing repeated string storage as shown in steps 4 and 5.

Why choose int8 for column 'B'?

Because values in 'B' fit in the range of int8 (0 to 127), so it uses less memory than int64, as seen in step 3.

Visual Quiz - 3 Questions

Look at the execution table, what is the dtype of column 'B' after step 3?

A int64

B int8

C float64

D category

💡 Hint

Check the 'Column B dtype' column in row for step 3.

At which step does the memory usage drop below 10000 bytes?

A Step 2

B Step 3

C Step 4

D Step 1

💡 Hint

Look at the 'Memory Usage' column in the execution table rows.

If column 'B' had values larger than 127, what would happen if we convert to int8?

A Values would be truncated or wrap around causing errors

B Memory usage would increase

C Conversion would automatically use int64

D No change in memory usage

💡 Hint

Think about int8 range and what happens if values exceed it.

Concept Snapshot

Data type optimization saves memory by converting columns to smaller types.
Use .astype() to change types like int64 to int8 or object to category.
Check memory usage with .memory_usage(deep=True).
Optimized types reduce DataFrame size and speed up analysis.

Full Transcript

We start with a DataFrame having columns with default data types. We check memory usage, then convert numeric columns to smaller integer types if values fit the range. For text columns, converting to category type saves memory by storing unique values once. After conversions, memory usage drops significantly. This process helps handle large data efficiently.