Combining multiple cleaning steps in Pandas - Step-by-Step Execution

Concept Flow - Combining multiple cleaning steps

Load raw data

↓

Step 1: Handle missing values

↓

Step 2: Remove duplicates

↓

Step 3: Fix data types

↓

Step 4: Rename columns

↓

Cleaned data ready for analysis

Data cleaning involves applying several steps one after another to prepare data for analysis.

Execution Sample

Pandas

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, None, 4, 4],
    'B': ['x', 'y', 'y', None, 'y']
})

# Clean data
cleaned = (df.dropna()
             .drop_duplicates()
             .astype({'A': 'int'})
             .rename(columns={'A': 'Alpha', 'B': 'Beta'}))

This code loads data, drops missing values, removes duplicates, converts column A to int, and renames columns.

Execution Table

Step	Action	Data Snapshot	Resulting DataFrame Shape
1	Original DataFrame	[{'A':1,'B':'x'},{'A':2,'B':'y'},{'A':None,'B':'y'},{'A':4,'B':None},{'A':4,'B':'y'}]	5 rows, 2 cols
2	Drop rows with missing values	[{'A':1,'B':'x'},{'A':2,'B':'y'},{'A':4,'B':'y'},{'A':4,'B':'y'}]	4 rows, 2 cols
3	Drop duplicate rows	[{'A':1,'B':'x'},{'A':2,'B':'y'},{'A':4,'B':'y'}]	3 rows, 2 cols
4	Convert column 'A' to int	[{'A':1,'B':'x'},{'A':2,'B':'y'},{'A':4,'B':'y'}]	3 rows, 2 cols
5	Rename columns 'A'->'Alpha', 'B'->'Beta'	[{'Alpha':1,'Beta':'x'},{'Alpha':2,'Beta':'y'},{'Alpha':4,'Beta':'y'}]	3 rows, 2 cols
6	End of cleaning steps	Final cleaned DataFrame	3 rows, 2 cols

💡 All cleaning steps applied; data ready for analysis.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	Final
df	[{'A':1,'B':'x'},{'A':2,'B':'y'},{'A':None,'B':'y'},{'A':4,'B':None},{'A':4,'B':'y'}]	[{'A':1,'B':'x'},{'A':2,'B':'y'},{'A':4,'B':'y'},{'A':4,'B':'y'}]	[{'A':1,'B':'x'},{'A':2,'B':'y'},{'A':4,'B':'y'}]	[{'A':1,'B':'x'},{'A':2,'B':'y'},{'A':4,'B':'y'}]	[{'Alpha':1,'Beta':'x'},{'Alpha':2,'Beta':'y'},{'Alpha':4,'Beta':'y'}]
cleaned	N/A	N/A	N/A	N/A	[{'Alpha':1,'Beta':'x'},{'Alpha':2,'Beta':'y'},{'Alpha':4,'Beta':'y'}]

Key Moments - 3 Insights

Why does dropna() remove rows with any missing value instead of just one column?

dropna() by default removes rows where any column has missing data, as shown in execution_table step 2 where rows with None in 'A' or 'B' are removed.

Why does drop_duplicates() not remove any rows after dropna()?

After dropna(), the remaining rows are unique, so drop_duplicates() in step 3 finds no duplicates to remove.

Why do we convert column 'A' to int after dropping missing values?

Because missing values (None) prevent conversion to int, so we drop them first (step 2), then convert safely in step 4.

Visual Quiz - 3 Questions

Look at the execution table at step 2. How many rows remain after dropna()?

A 4

B 3

C 5

D 2

💡 Hint

Check the 'Data Snapshot' and 'Resulting DataFrame Shape' columns at step 2.

At which step does the DataFrame columns get renamed?

A Step 3

B Step 5

C Step 4

D Step 6

💡 Hint

Look for the action mentioning renaming columns in the execution table.

If we skip dropna(), what problem will occur at step 4?

A Duplicates will increase

B No problem, conversion works fine

C Conversion to int will fail due to missing values

D Columns will not rename

💡 Hint

Refer to key moment about why dropna() is needed before converting types.

Concept Snapshot

Combining multiple cleaning steps in pandas:
- Chain methods like dropna(), drop_duplicates(), astype(), rename()
- Each step cleans data progressively
- Order matters: e.g., drop missing before type conversion
- Result is clean data ready for analysis

Full Transcript

This example shows how to combine multiple data cleaning steps using pandas. We start with raw data containing missing values and duplicates. First, we remove rows with missing values using dropna(). Next, we remove duplicate rows with drop_duplicates(). Then, we convert the data type of column 'A' to integer, which requires no missing values. Finally, we rename columns for clarity. Each step changes the data progressively, resulting in a clean DataFrame ready for analysis.

Practice

1. What is the main benefit of combining multiple cleaning steps in pandas using method chaining?

easy

A. It creates multiple copies of the data.

B. It slows down the data processing.

C. It makes the code shorter and easier to read.

D. It hides errors in the data.

Solution

1. Step 1: Understand method chaining

   Method chaining lets you apply many cleaning steps one after another in one line or block.

2. Step 2: Benefits of chaining

   This approach reduces code length and improves readability by avoiding intermediate variables.

3. Final Answer:

   It makes the code shorter and easier to read. -> Option C

4. Quick Check:

   Method chaining = shorter, readable code [OK]

Hint: Chaining keeps code neat and easy to follow [OK]

Common Mistakes:

• Thinking chaining slows down processing
• Believing it creates multiple data copies
• Assuming it hides data errors

2. Which of the following is the correct syntax to chain pandas cleaning methods to drop missing values and then reset the index?

easy

A. df.dropna().reset_index(drop=True

B. df.dropna.reset_index(drop=True)

C. df.dropna().reset_index(drop=True))

D. df.dropna().reset_index(drop=True)

Solution

1. Step 1: Check method calls

   Each method must be called with parentheses: dropna() and reset_index().

2. Step 2: Verify parentheses balance

   df.dropna().reset_index(drop=True) has balanced parentheses and correct chaining syntax.

3. Final Answer:

   df.dropna().reset_index(drop=True) -> Option D

4. Quick Check:

   Method calls need parentheses [OK]

Hint: Always use () after pandas methods when chaining [OK]

Common Mistakes:

• Missing parentheses after methods
• Unbalanced parentheses causing syntax errors
• Using incorrect method names

3. Given the DataFrame df below, what will be the output after applying the chained cleaning steps?

import pandas as pd
df = pd.DataFrame({
  'A': [1, 2, None, 4],
  'B': ['x', None, 'y', 'z']
})
cleaned = df.dropna().reset_index(drop=True)

medium

A. {'A': [1.0, 4.0], 'B': ['x', 'y']}

B. {'A': [1.0, 4.0], 'B': ['x', 'z']}

C. {'A': [1.0, 2.0, 4.0], 'B': ['x', 'y', 'z']}

D. {'A': [1.0, 2.0, 4.0], 'B': ['x', None, 'z']}

Solution

1. Step 1: Apply dropna()

   Rows with any missing values are removed. Row 1 and 2 have missing values, so only rows 0 and 3 remain.

2. Step 2: Reset index

   Resetting index with drop=True gives new index 0 and 1 for the remaining rows.

3. Final Answer:

   {'A': [1.0, 4.0], 'B': ['x', 'z']} -> Option B

4. Quick Check:

   dropna removes rows with NaN, reset_index fixes index [OK]

Hint: dropna removes rows with NaN, reset_index fixes numbering [OK]

Common Mistakes:

• Forgetting dropna removes rows with any NaN
• Not resetting index after dropping rows
• Assuming missing values are filled automatically

4. The following code is intended to clean a DataFrame by filling missing values with 0 and then dropping duplicates. What is the error?

cleaned = df.fillna(0).drop_duplicates

medium

A. Missing parentheses after drop_duplicates method.

B. fillna should be called after drop_duplicates.

C. drop_duplicates does not exist in pandas.

D. fillna cannot be chained with drop_duplicates.

Solution

1. Step 1: Check method calls

   fillna(0) is called correctly with parentheses, but drop_duplicates is missing parentheses.

2. Step 2: Understand method chaining syntax

   All methods must be called with parentheses to execute them.

3. Final Answer:

   Missing parentheses after drop_duplicates method. -> Option A

4. Quick Check:

   Methods need () to run [OK]

Hint: Always add () after pandas methods when chaining [OK]

Common Mistakes:

• Forgetting parentheses after methods
• Changing method order unnecessarily
• Thinking drop_duplicates is not a pandas method

5. You have a DataFrame df with columns 'Name', 'Age', and 'Score'. Some 'Age' values are missing, and some rows are duplicates. You want to fill missing 'Age' with the average age, remove duplicates, and reset the index. Which chained code correctly does this?

hard

A. df.fillna({'Age': df['Age'].mean()}).drop_duplicates().reset_index(drop=True)

B. df.drop_duplicates().fillna(df['Age'].mean()).reset_index(drop=True)

C. df.fillna(df['Age'].mean()).drop_duplicates.reset_index(drop=True)

D. df.reset_index(drop=True).fillna({'Age': df['Age'].mean()}).drop_duplicates()

Solution

1. Step 1: Fill missing 'Age' with average

   Use fillna with a dictionary to fill only 'Age' column with its mean.

2. Step 2: Remove duplicates and reset index

   Then drop duplicates and reset index with drop=True to clean the DataFrame fully.

3. Final Answer:

   df.fillna({'Age': df['Age'].mean()}).drop_duplicates().reset_index(drop=True) -> Option A

4. Quick Check:

   Fill missing, drop duplicates, reset index in order [OK]

Hint: Fill missing with dict, then drop duplicates, reset index last [OK]

Common Mistakes:

• Filling all columns instead of only 'Age'
• Missing parentheses on drop_duplicates
• Resetting index before cleaning data