Using appropriate dtypes in Pandas - Step-by-Step Execution

Concept Flow - Using appropriate dtypes

Load data with default dtypes

↓

Check memory usage

↓

Identify columns to optimize

↓

Convert columns to appropriate dtypes

↓

Check memory usage again

↓

Use optimized data for analysis

This flow shows loading data, checking memory, converting columns to better types, and then using optimized data.

Execution Sample

Pandas

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, 3],
    'B': ['x', 'y', 'z'],
    'C': [1.0, 2.5, 3.1]
})
df['A'] = df['A'].astype('int8')
df['B'] = df['B'].astype('category')

This code creates a DataFrame and changes column types to use less memory.

Execution Table

Step	Action	Column	Original dtype	New dtype	Memory Usage (approx)
1	Create DataFrame	A	int64	int64	24 bytes
2	Create DataFrame	B	object	object	72 bytes
3	Create DataFrame	C	float64	float64	24 bytes
4	Convert dtype	A	int64	int8	3 bytes
5	Convert dtype	B	object	category	33 bytes
6	No change	C	float64	float64	24 bytes
7	Total memory before optimization	-	-	-	120 bytes
8	Total memory after optimization	-	-	-	60 bytes

💡 Memory usage reduced by converting columns to smaller or categorical dtypes.

Variable Tracker

Variable	Start	After Step 4	After Step 5	Final
df['A'].dtype	int64	int8	int8	int8
df['B'].dtype	object	object	category	category
df['C'].dtype	float64	float64	float64	float64
Memory usage (approx)	120 bytes	99 bytes	60 bytes	60 bytes

Key Moments - 3 Insights

Why do we convert 'B' from object to category dtype?

Because 'B' contains repeated strings, converting to category saves memory by storing codes instead of full strings, as shown in steps 2 and 5 in the execution_table.

Why does converting 'A' from int64 to int8 reduce memory?

int8 uses 1 byte per value instead of 8 bytes for int64, so memory drops significantly as seen between steps 1 and 4 in the execution_table.

Why didn't we change the dtype of column 'C'?

Because 'C' contains float values that need precision, and no smaller float dtype was used here, so it remains float64 as in steps 3 and 6.

Visual Quiz - 3 Questions

Look at the execution_table at step 4, what is the new dtype of column 'A'?

A int8

B int64

C float64

D category

💡 Hint

Check the 'New dtype' column for step 4 in the execution_table.

At which step does the memory usage reduce to about half of the original?

A Step 5

B Step 8

C Step 6

D Step 3

💡 Hint

Look at the 'Memory Usage (approx)' column and compare total memory before and after optimization.

If column 'B' was not converted to category, what would be the approximate memory usage after step 5?

A About 33 bytes

B About 24 bytes

C About 72 bytes

D About 3 bytes

💡 Hint

Refer to the original memory usage of column 'B' at step 2 in the execution_table.

Concept Snapshot

Using appropriate dtypes in pandas:
- Load data with default types
- Check memory usage
- Convert columns to smaller types (e.g., int64 to int8)
- Convert string columns to 'category' if repeated
- Check memory again to confirm savings
- Use optimized DataFrame for faster, lighter analysis

Full Transcript

This visual execution shows how to use appropriate data types in pandas to save memory. We start by creating a DataFrame with default types: integers as int64, strings as object, and floats as float64. We check memory usage, then convert the integer column to int8 to use less space. We convert the string column to category because it has repeated values, which saves memory by storing codes instead of full strings. The float column remains unchanged. After conversions, memory usage is about half the original. This process helps make data analysis faster and more efficient by reducing memory load.

Practice

1. Why is it important to use appropriate data types in a pandas DataFrame?

easy

A. It automatically fixes missing data values.

B. It reduces memory usage and speeds up data processing.

C. It makes the DataFrame look colorful in output.

D. It changes the order of rows in the DataFrame.

Solution

1. Step 1: Understand data types impact on memory

   Using smaller or correct data types reduces the memory needed to store data.

2. Step 2: Understand data types impact on speed

   Operations on smaller data types run faster, improving performance.

3. Final Answer:

   It reduces memory usage and speeds up data processing. -> Option B

4. Quick Check:

   Appropriate dtypes = less memory and faster [OK]

Hint: Smaller data types save memory and speed up work [OK]

Common Mistakes:

• Thinking data types change data values automatically
• Believing data types affect DataFrame colors
• Assuming data types reorder rows

2. Which of the following is the correct way to change a column named age to integer type in pandas?

easy

A. df['age'] = df['age'].astype('int')

B. df.astype('age', int)

C. df['age'].convert(int)

D. df.change_type('age', 'int')

Solution

1. Step 1: Recall pandas syntax for changing dtype

   The correct method is to use astype() on the column.

2. Step 2: Check method usage

   Use df['age'] = df['age'].astype('int') to convert the 'age' column to integer.

3. Final Answer:

   df['age'] = df['age'].astype('int') -> Option A

4. Quick Check:

   astype() method changes dtype [OK]

Hint: df[col] = df[col].astype(dtype) to change dtype [OK]

Common Mistakes:

• Forgetting to assign the result of astype() back to the column
• Using non-existent methods like convert() or change_type()
• Passing arguments in wrong order or syntax

3. Given the code below, what will be the output of df.dtypes?

import pandas as pd
data = {'score': [90, 85, 88], 'passed': ['True', 'False', 'True']}
df = pd.DataFrame(data)
df['passed'] = df['passed'].astype('bool')
print(df.dtypes)

medium

A. score object
passed bool
dtype: object

B. score int64
passed object
dtype: object

C. score int64
passed bool
dtype: object

D. score float64
passed bool
dtype: object

Solution

1. Step 1: Check initial data types

   'score' is a list of integers, so pandas infers int64. 'passed' is string initially.

2. Step 2: Convert 'passed' to boolean

   Using astype('bool') converts 'passed' column to boolean type.

3. Final Answer:

   score int64 passed bool dtype: object -> Option C

4. Quick Check:

   astype('bool') converts string to bool [OK]

Hint: astype('bool') converts strings like 'True' to boolean [OK]

Common Mistakes:

• Assuming 'passed' stays as object after astype('bool')
• Thinking 'score' becomes float automatically
• Confusing bool with string type

4. What is wrong with this code snippet?

df['price'] = df['price'].astype('int64')
df['price'] = df['price'].astype('category')

Why might this cause an error or unexpected behavior?

medium

A. Converting numeric to category without checking unique values may increase memory usage or cause unexpected behavior.

B. You cannot convert a column twice in pandas.

C. astype('category') only works on string columns.

D. int64 is not a valid dtype in pandas.

Solution

1. Step 1: Understand dtype conversion to category

   Converting numeric to category is allowed but categories are based on unique values.

2. Step 2: Consider memory and behavior risks

   If the numeric column has many unique values, category conversion may increase memory or cause unexpected grouping.

3. Final Answer:

   Converting numeric to category without checking unique values may increase memory usage or cause unexpected behavior. -> Option A

4. Quick Check:

   Category dtype needs careful unique value check [OK]

Hint: Check unique values before converting to category [OK]

Common Mistakes:

• Thinking multiple astype() calls are invalid
• Believing astype('category') only works on strings
• Assuming int64 is invalid dtype

5. You have a large DataFrame with a column status containing only three unique string values: 'new', 'in_progress', 'done'. Which dtype change will save memory and keep data efficient?

hard

A. Keep status as string without changes.

B. Convert status to object dtype.

C. Convert status to int64 dtype directly.

D. Convert status to category dtype.

Solution

1. Step 1: Identify unique values in 'status'

   Only three unique strings exist, ideal for categorical dtype.

2. Step 2: Understand category dtype benefits

   Category stores unique values once and uses integer codes internally, saving memory.

3. Step 3: Compare other options

   Object dtype stores full strings repeatedly; int64 conversion is invalid without mapping; no change wastes memory.

4. Final Answer:

   Convert status to category dtype. -> Option D

5. Quick Check:

   Few unique strings? Use category dtype [OK]

Hint: Few unique strings? Use category dtype for memory [OK]

Common Mistakes:

• Trying to convert strings directly to int64 without mapping
• Keeping object dtype wastes memory
• Ignoring category dtype benefits