Why systematic cleaning matters in Pandas - See It in Action

Why Systematic Cleaning Matters

📖 Scenario: Imagine you work in a small online store. You have a list of customer orders, but the data is messy. Some orders have missing prices, some have wrong product names, and some have extra spaces. To understand your sales, you need to clean this data carefully.

🎯 Goal: You will create a small dataset of orders, set a rule to identify invalid prices, clean the data by fixing or removing bad entries, and finally show the cleaned data. This will teach you why cleaning data step-by-step is important before analysis.

📋 What You'll Learn

Create a pandas DataFrame with given order data

Set a price threshold to identify invalid prices

Use pandas methods to clean the data systematically

Print the cleaned DataFrame as the final output

💡 Why This Matters

🌍 Real World

Cleaning data is a crucial first step in any data analysis or business decision. Messy data can lead to wrong conclusions.

💼 Career

Data scientists and analysts spend a lot of time cleaning data before they can analyze it. This project shows why systematic cleaning is important.

1

Create the initial orders DataFrame

Import pandas as pd and create a DataFrame called orders with these exact columns and rows:
OrderID: [101, 102, 103, 104, 105]
Product: [' apple', 'banana', 'Orange ', 'banana', 'apple']
Price: [1.2, 0, 0.8, -1, 1.5]

Pandas

# Import pandas and create the DataFrame orders with the given data
# Your code here

Hint

Use pd.DataFrame with a dictionary of lists for columns.

2

Set a price threshold for invalid prices

Create a variable called min_price and set it to 0.5. This will help us find prices that are too low or invalid.

Pandas

import pandas as pd
orders = pd.DataFrame({
    'OrderID': [101, 102, 103, 104, 105],
    'Product': ['  apple', 'banana', 'Orange ', 'banana', 'apple'],
    'Price': [1.2, 0, 0.8, -1, 1.5]
})
# Create min_price variable and set it to 0.5
# Your code here

Hint

Just create a variable named min_price and assign 0.5.

3

Clean the data by fixing product names and filtering prices

Create a new DataFrame called clean_orders by doing these steps:
1. Remove extra spaces from the Product column using str.strip().
2. Keep only rows where Price is greater than or equal to min_price.

Pandas

import pandas as pd
orders = pd.DataFrame({
    'OrderID': [101, 102, 103, 104, 105],
    'Product': ['  apple', 'banana', 'Orange ', 'banana', 'apple'],
    'Price': [1.2, 0, 0.8, -1, 1.5]
})
min_price = 0.5
# Create clean_orders by stripping spaces and filtering prices
# Your code here

Hint

Use str.strip() on the Product column and filter rows with orders['Price'] >= min_price.

4

Print the cleaned orders DataFrame

Print the clean_orders DataFrame to see the cleaned data.

Pandas

import pandas as pd
orders = pd.DataFrame({
    'OrderID': [101, 102, 103, 104, 105],
    'Product': ['  apple', 'banana', 'Orange ', 'banana', 'apple'],
    'Price': [1.2, 0, 0.8, -1, 1.5]
})
min_price = 0.5
orders['Product'] = orders['Product'].str.strip()
clean_orders = orders[orders['Price'] >= min_price]
# Print the clean_orders DataFrame
# Your code here

Hint

Use print(clean_orders) to show the cleaned data.

Practice

1. Why is systematic cleaning important when working with data in pandas?

easy

A. It increases the file size of the dataset.

B. It makes the data look colorful and pretty.

C. It helps ensure the analysis results are accurate and reliable.

D. It automatically creates new data columns.

Solution

1. Step 1: Understand the purpose of cleaning data

   Cleaning removes errors and inconsistencies that can mislead analysis.

2. Step 2: Connect cleaning to analysis trustworthiness

   Systematic cleaning ensures results are based on correct and consistent data.

3. Final Answer:

   It helps ensure the analysis results are accurate and reliable. -> Option C

4. Quick Check:

   Cleaning = Accurate results [OK]

Hint: Cleaning fixes errors to trust your results [OK]

Common Mistakes:

• Thinking cleaning only changes data appearance
• Believing cleaning adds new data automatically
• Assuming cleaning increases data size

2. Which pandas function is commonly used to remove rows with missing values?

easy

A. df.replace()

B. df.dropna()

C. df.drop_duplicates()

D. df.fillna()

Solution

1. Step 1: Identify function to remove missing data

   df.dropna() removes rows with any missing (NaN) values.

2. Step 2: Differentiate from other functions

   df.fillna() fills missing values, drop_duplicates() removes duplicate rows, replace() changes values.

3. Final Answer:

   df.dropna() -> Option B

4. Quick Check:

   Remove missing rows = dropna() [OK]

Hint: dropna removes missing rows, fillna fills them [OK]

Common Mistakes:

• Confusing fillna() with dropna()
• Using drop_duplicates() to remove missing values
• Trying replace() to drop rows

3. Given the code below, what will be the output?

import pandas as pd
data = {'Name': ['Anna', 'Bob', None, 'Diana'], 'Age': [25, None, 30, 22]}
df = pd.DataFrame(data)
df_clean = df.dropna()
print(df_clean)

medium

A. DataFrame with only rows where 'Name' and 'Age' are not None

B. DataFrame with all rows including those with None values

C. Empty DataFrame with no rows

D. Error because dropna() needs parameters

Solution

1. Step 1: Understand dropna() behavior

   dropna() removes rows with any missing (None/NaN) values in any column.

2. Step 2: Apply dropna() to given data

   Rows with None in 'Name' or 'Age' are removed, leaving only rows with complete data.

3. Final Answer:

   DataFrame with only rows where 'Name' and 'Age' are not None -> Option A

4. Quick Check:

   dropna() removes rows with missing values [OK]

Hint: dropna() keeps only complete rows [OK]

Common Mistakes:

• Expecting dropna() to fill missing values
• Thinking dropna() removes columns instead of rows
• Assuming dropna() causes an error without parameters

4. The following code is intended to remove duplicate rows from a DataFrame, but it does not work as expected. What is the error?

import pandas as pd
data = {'A': [1, 2, 2, 3], 'B': [4, 5, 5, 6]}
df = pd.DataFrame(data)
df.drop_duplicates()
print(df)

medium

A. drop_duplicates() cannot be used on numeric columns.

B. drop_duplicates() requires a column name as argument.

C. drop_duplicates() needs to be called before creating the DataFrame.

D. drop_duplicates() does not remove duplicates without inplace=True or reassignment.

Solution

1. Step 1: Check drop_duplicates() usage

   By default, drop_duplicates() returns a new DataFrame and does not change the original unless inplace=True or reassigned.

2. Step 2: Identify why duplicates remain

   Since df.drop_duplicates() is called without assignment or inplace=True, df remains unchanged.

3. Final Answer:

   drop_duplicates() does not remove duplicates without inplace=True or reassignment. -> Option D

4. Quick Check:

   drop_duplicates() needs inplace or assignment [OK]

Hint: Use inplace=True or assign result to remove duplicates [OK]

Common Mistakes:

• Assuming drop_duplicates() modifies original DataFrame by default
• Thinking drop_duplicates() needs column argument always
• Believing drop_duplicates() cannot work on numbers

5. You have a DataFrame with messy text data in a column 'City' that contains extra spaces and inconsistent capitalization. Which sequence of pandas operations best cleans this column systematically?

hard

A. df['City'] = df['City'].str.strip().str.title()
# No further cleaning needed

B. df['City'] = df['City'].str.strip().str.title()
df['City'] = df['City'].str.replace(' ', '')

C. df['City'] = df['City'].str.strip().str.title()
df['City'] = df['City'].str.replace(' ', ' ')
df['City'] = df['City'].str.replace(' ', ' ')

D. df['City'] = df['City'].str.strip().str.lower()
df['City'] = df['City'].str.capitalize()
df['City'] = df['City'].str.replace(' ', '')

Solution

1. Step 1: Remove extra spaces

   str.strip() removes leading and trailing spaces effectively.

2. Step 2: Fix capitalization

   str.title() converts text to title case, making capitalization consistent.

3. Step 3: Avoid unnecessary replacements

   Removing all spaces or replacing spaces incorrectly can harm data integrity.

4. Final Answer:

   df['City'] = df['City'].str.strip().str.title() -> Option A

5. Quick Check:

   Strip spaces + title case = clean text [OK]

Hint: Strip spaces then title case for clean text [OK]

Common Mistakes:

• Removing all spaces inside city names
• Applying inconsistent capitalization methods
• Overcomplicating cleaning with unnecessary replacements