Why systematic cleaning matters in Pandas - Challenge Your Understanding

Challenge - 5 Problems

❓ data_output

intermediate

Effect of Missing Data on Mean Calculation

Consider a dataset with missing values. What is the output of the mean calculation without cleaning?

Pandas

import pandas as pd

data = {'score': [10, 20, None, 40, 50]}
df = pd.DataFrame(data)
mean_score = df['score'].mean()
print(mean_score)

A TypeError

B None

C SyntaxError

D 30.0

💡 Hint

Think about how pandas handles missing values in calculations.

Pandas automatically ignores None or NaN values when calculating the mean, so the mean is calculated over the available numbers only.

❓ data_output

intermediate

Impact of Duplicate Rows on Data Analysis

Given a DataFrame with duplicate rows, what is the number of unique rows after removing duplicates?

Pandas

import pandas as pd

data = {'id': [1, 2, 2, 3, 4, 4, 4], 'value': [10, 20, 20, 30, 40, 40, 40]}
df = pd.DataFrame(data)
unique_rows = df.drop_duplicates().shape[0]
print(unique_rows)

A 4

B 5

C 3

D 7

💡 Hint

Removing duplicates keeps only one instance of each repeated row.

Rows with ids 2 and 4 are duplicated. After dropping duplicates, only unique rows remain, totaling 4.

🧠 Conceptual

advanced

Why Systematic Cleaning Prevents Analysis Errors

Which statement best explains why systematic data cleaning is important before analysis?

A It helps identify and fix inconsistencies that could lead to wrong conclusions.

B It ensures all data is removed to avoid errors.

C It speeds up the analysis by deleting half the data randomly.

D It replaces all numbers with zeros to simplify calculations.

💡 Hint

Think about the role of data quality in making good decisions.

Systematic cleaning finds and fixes problems like missing or wrong data, so the analysis results are trustworthy.

❓ Predict Output

advanced

Result of Cleaning and Filtering Data

What is the output DataFrame after cleaning missing values and filtering scores above 25?

Pandas

import pandas as pd

data = {'name': ['Anna', 'Bob', 'Cara', 'Dan'], 'score': [20, None, 30, 40]}
df = pd.DataFrame(data)
df_clean = df.dropna()
df_filtered = df_clean[df_clean['score'] > 25]
print(df_filtered)

A

   name  score
0  Anna   20.0
2  Cara   30.0
3   Dan   40.0

B

   name  score
2  Cara   30.0
3   Dan   40.0

C

   name  score
1   Bob    NaN
2  Cara   30.0
3   Dan   40.0

D

Empty DataFrame
Columns: [name, score]
Index: []

💡 Hint

First remove rows with missing scores, then keep only scores above 25.

Row with Bob is removed due to missing score. Anna's score is 20, below 25, so only Cara and Dan remain.

🚀 Application

expert

Identify the Impact of Unclean Data on Grouped Aggregation

Given this dataset, what is the average score per group if missing values are NOT cleaned?

Pandas

import pandas as pd
import numpy as np

data = {'group': ['A', 'A', 'B', 'B', 'C', 'C'], 'score': [10, np.nan, 20, 30, np.nan, 50]}
df = pd.DataFrame(data)
grouped_mean = df.groupby('group')['score'].mean()
print(grouped_mean)

A

group
A    NaN
B    25.0
C    NaN
Name: score, dtype: float64

B

group
A    10.0
B    25.0
C    0.0
Name: score, dtype: float64

C

group
A    10.0
B    25.0
C    50.0
Name: score, dtype: float64

D

group
A    20.0
B    25.0
C    50.0
Name: score, dtype: float64

💡 Hint

Remember how pandas handles missing values in group calculations.

Pandas ignores missing values in mean calculations per group. Group A has one valid score 10, group B has 20 and 30 averaging 25, group C has one valid score 50.

Practice

1. Why is systematic cleaning important when working with data in pandas?

easy

A. It increases the file size of the dataset.

B. It makes the data look colorful and pretty.

C. It helps ensure the analysis results are accurate and reliable.

D. It automatically creates new data columns.

Solution

1. Step 1: Understand the purpose of cleaning data

   Cleaning removes errors and inconsistencies that can mislead analysis.

2. Step 2: Connect cleaning to analysis trustworthiness

   Systematic cleaning ensures results are based on correct and consistent data.

3. Final Answer:

   It helps ensure the analysis results are accurate and reliable. -> Option C

4. Quick Check:

   Cleaning = Accurate results [OK]

Hint: Cleaning fixes errors to trust your results [OK]

Common Mistakes:

• Thinking cleaning only changes data appearance
• Believing cleaning adds new data automatically
• Assuming cleaning increases data size

2. Which pandas function is commonly used to remove rows with missing values?

easy

A. df.replace()

B. df.dropna()

C. df.drop_duplicates()

D. df.fillna()

Solution

1. Step 1: Identify function to remove missing data

   df.dropna() removes rows with any missing (NaN) values.

2. Step 2: Differentiate from other functions

   df.fillna() fills missing values, drop_duplicates() removes duplicate rows, replace() changes values.

3. Final Answer:

   df.dropna() -> Option B

4. Quick Check:

   Remove missing rows = dropna() [OK]

Hint: dropna removes missing rows, fillna fills them [OK]

Common Mistakes:

• Confusing fillna() with dropna()
• Using drop_duplicates() to remove missing values
• Trying replace() to drop rows

3. Given the code below, what will be the output?

import pandas as pd
data = {'Name': ['Anna', 'Bob', None, 'Diana'], 'Age': [25, None, 30, 22]}
df = pd.DataFrame(data)
df_clean = df.dropna()
print(df_clean)

medium

A. DataFrame with only rows where 'Name' and 'Age' are not None

B. DataFrame with all rows including those with None values

C. Empty DataFrame with no rows

D. Error because dropna() needs parameters

Solution

1. Step 1: Understand dropna() behavior

   dropna() removes rows with any missing (None/NaN) values in any column.

2. Step 2: Apply dropna() to given data

   Rows with None in 'Name' or 'Age' are removed, leaving only rows with complete data.

3. Final Answer:

   DataFrame with only rows where 'Name' and 'Age' are not None -> Option A

4. Quick Check:

   dropna() removes rows with missing values [OK]

Hint: dropna() keeps only complete rows [OK]

Common Mistakes:

• Expecting dropna() to fill missing values
• Thinking dropna() removes columns instead of rows
• Assuming dropna() causes an error without parameters

4. The following code is intended to remove duplicate rows from a DataFrame, but it does not work as expected. What is the error?

import pandas as pd
data = {'A': [1, 2, 2, 3], 'B': [4, 5, 5, 6]}
df = pd.DataFrame(data)
df.drop_duplicates()
print(df)

medium

A. drop_duplicates() cannot be used on numeric columns.

B. drop_duplicates() requires a column name as argument.

C. drop_duplicates() needs to be called before creating the DataFrame.

D. drop_duplicates() does not remove duplicates without inplace=True or reassignment.

Solution

1. Step 1: Check drop_duplicates() usage

   By default, drop_duplicates() returns a new DataFrame and does not change the original unless inplace=True or reassigned.

2. Step 2: Identify why duplicates remain

   Since df.drop_duplicates() is called without assignment or inplace=True, df remains unchanged.

3. Final Answer:

   drop_duplicates() does not remove duplicates without inplace=True or reassignment. -> Option D

4. Quick Check:

   drop_duplicates() needs inplace or assignment [OK]

Hint: Use inplace=True or assign result to remove duplicates [OK]

Common Mistakes:

• Assuming drop_duplicates() modifies original DataFrame by default
• Thinking drop_duplicates() needs column argument always
• Believing drop_duplicates() cannot work on numbers

5. You have a DataFrame with messy text data in a column 'City' that contains extra spaces and inconsistent capitalization. Which sequence of pandas operations best cleans this column systematically?

hard

A. df['City'] = df['City'].str.strip().str.title()
# No further cleaning needed

B. df['City'] = df['City'].str.strip().str.title()
df['City'] = df['City'].str.replace(' ', '')

C. df['City'] = df['City'].str.strip().str.title()
df['City'] = df['City'].str.replace(' ', ' ')
df['City'] = df['City'].str.replace(' ', ' ')

D. df['City'] = df['City'].str.strip().str.lower()
df['City'] = df['City'].str.capitalize()
df['City'] = df['City'].str.replace(' ', '')

Solution

1. Step 1: Remove extra spaces

   str.strip() removes leading and trailing spaces effectively.

2. Step 2: Fix capitalization

   str.title() converts text to title case, making capitalization consistent.

3. Step 3: Avoid unnecessary replacements

   Removing all spaces or replacing spaces incorrectly can harm data integrity.

4. Final Answer:

   df['City'] = df['City'].str.strip().str.title() -> Option A

5. Quick Check:

   Strip spaces + title case = clean text [OK]

Hint: Strip spaces then title case for clean text [OK]

Common Mistakes:

• Removing all spaces inside city names
• Applying inconsistent capitalization methods
• Overcomplicating cleaning with unnecessary replacements