Why data cleaning consumes most analysis time in Data Analysis Python - See It in Action

Why Data Cleaning Consumes Most Analysis Time

📖 Scenario: You are a data analyst working with a small dataset from a customer survey. The data has some missing values and inconsistent entries. You want to understand why cleaning this data takes most of your time before you can analyze it.

🎯 Goal: Build a simple Python script that shows how to identify and handle missing and inconsistent data entries in a dataset.

📋 What You'll Learn

Create a dictionary called survey_data with customer names as keys and their ratings as values, including some missing and inconsistent entries.

Create a variable called missing_value set to None to represent missing data.

Use a for loop with variables customer and rating to iterate over survey_data.items() and create a new dictionary cleaned_data that replaces missing or invalid ratings with the average rating.

Print the cleaned_data dictionary to see the cleaned results.

💡 Why This Matters

🌍 Real World

In real life, data from surveys, sensors, or databases often have missing or wrong values. Cleaning this data is essential before any meaningful analysis.

💼 Career

Data scientists and analysts spend a large part of their work cleaning data to ensure accurate results and insights.

1

Create the initial survey data

Create a dictionary called survey_data with these exact entries: 'Alice': 5, 'Bob': None, 'Charlie': 3, 'David': 'N/A', 'Eva': 4.

Data Analysis Python

# Create the survey_data dictionary with given entries
# Your code here

Hint

Use curly braces to create a dictionary. Use None for missing values and the string 'N/A' for inconsistent entries.

2

Set the missing value indicator

Create a variable called missing_value and set it to None to represent missing data.

Data Analysis Python

survey_data = {'Alice': 5, 'Bob': None, 'Charlie': 3, 'David': 'N/A', 'Eva': 4}
# Create missing_value variable set to None
# Your code here

Hint

Just assign None to the variable missing_value.

3

Clean the data by replacing missing and invalid entries

Use a for loop with variables customer and rating to iterate over survey_data.items(). Calculate the average of valid ratings first. Then create a new dictionary called cleaned_data where missing or invalid ratings (like None or 'N/A') are replaced by the average rating.

Data Analysis Python

survey_data = {'Alice': 5, 'Bob': None, 'Charlie': 3, 'David': 'N/A', 'Eva': 4}
missing_value = None
# Calculate average rating of valid entries
# Create cleaned_data dictionary replacing missing or invalid ratings
# Your code here

Hint

First find the average of valid ratings. Then loop through each entry. If the rating is None or 'N/A', replace it with the average. Otherwise, keep the original rating.

4

Print the cleaned data

Write a print statement to display the cleaned_data dictionary.

Data Analysis Python

survey_data = {'Alice': 5, 'Bob': None, 'Charlie': 3, 'David': 'N/A', 'Eva': 4}
missing_value = None

valid_ratings = [rating for rating in survey_data.values() if isinstance(rating, int)]
average_rating = sum(valid_ratings) / len(valid_ratings)

cleaned_data = {}
for customer, rating in survey_data.items():
    if rating == missing_value or rating == 'N/A':
        cleaned_data[customer] = average_rating
    else:
        cleaned_data[customer] = rating
# Print the cleaned_data dictionary
# Your code here

Hint

Use print(cleaned_data) to show the final cleaned dictionary.

Practice

1. Why does data cleaning usually take the most time in data analysis?

easy

A. Because data cleaning is only about deleting data

B. Because data cleaning requires writing complex machine learning models

C. Because real-world data often has missing or incorrect values that need fixing

D. Because data cleaning is about creating visualizations

Solution

1. Step 1: Understand real-world data quality

   Real-world data is often messy with missing or wrong values.

2. Step 2: Recognize cleaning tasks

   Cleaning fixes these issues to prepare data for analysis.

3. Final Answer:

   Because real-world data often has missing or incorrect values that need fixing -> Option C

4. Quick Check:

   Messy data = cleaning time [OK]

Hint: Messy data means more cleaning time [OK]

Common Mistakes:

• Thinking cleaning is about building models
• Confusing cleaning with visualization
• Assuming cleaning just deletes data

2. Which Python code snippet correctly replaces missing values in a DataFrame column 'age' with the mean age?

easy

A. df['age'].dropna().mean()

B. df['age'].replace('NaN', df['age'].mean())

C. df.fillna('age', df.mean())

D. df['age'].fillna(df['age'].mean(), inplace=True)

Solution

1. Step 1: Identify correct method to fill missing values

   The fillna() method replaces missing values; inplace=True updates the DataFrame directly.

2. Step 2: Check correct usage of mean for replacement

   df['age'].mean() calculates the mean of the 'age' column correctly.

3. Final Answer:

   df['age'].fillna(df['age'].mean(), inplace=True) -> Option D

4. Quick Check:

   fillna with mean = correct syntax [OK]

Hint: Use fillna with mean() and inplace=True [OK]

Common Mistakes:

• Using replace with string 'NaN' instead of np.nan
• Wrong method fillna arguments
• Dropping missing values instead of filling

3. Given this code snippet cleaning a DataFrame:

import pandas as pd

df = pd.DataFrame({'score': [10, None, 20, None, 30]})
df['score'] = df['score'].fillna(int(df['score'].median()))
print(df['score'].tolist())

What is the output list?

medium

A. [10, 20, 20, 20, 30]

B. [10, 15, 20, 15, 30]

C. [10, null, 20, null, 30]

D. [10, 20, 20, null, 30]

Solution

1. Step 1: Calculate median of non-missing scores

   Non-missing values are [10, 20, 30]. Median is 20.

2. Step 2: Replace null with median

   Missing values replaced by 20, so list becomes [10, 20, 20, 20, 30].

3. Final Answer:

   [10, 20, 20, 20, 30] -> Option A

4. Quick Check:

   Median fill replaces null with 20 [OK]

Hint: Median of [10,20,30] is 20; fill null with 20 [OK]

Common Mistakes:

• Using mean instead of median
• Not replacing null values
• Confusing null with string 'null'

4. This code tries to remove duplicate rows from a DataFrame but does not work as expected:

df.drop_duplicates()
print(len(df))

What is the main issue?

medium

A. drop_duplicates() does not remove duplicates without inplace=True or reassignment

B. drop_duplicates() cannot remove duplicates in pandas

C. len(df) counts duplicates, so it is always wrong

D. drop_duplicates() requires a column name argument

Solution

1. Step 1: Understand drop_duplicates behavior

   By default, drop_duplicates() returns a new DataFrame and does not change original unless inplace=True or reassigned.

2. Step 2: Check code effect on original DataFrame

   Since result is not saved or inplace=True used, original df remains unchanged, so length stays same.

3. Final Answer:

   drop_duplicates() does not remove duplicates without inplace=True or reassignment -> Option A

4. Quick Check:

   Use inplace=True or assign result [OK]

Hint: Use inplace=True or assign result to remove duplicates [OK]

Common Mistakes:

• Assuming drop_duplicates changes df without inplace
• Thinking len(df) excludes duplicates automatically
• Believing drop_duplicates needs column argument always

5. You have a dataset with columns ['name', 'age', 'income']. Some age values are missing, and some income values are negative due to data entry errors. Which sequence of cleaning steps is best to prepare data for analysis?

hard

A. Delete all rows with missing 'age', then fill negative 'income' with mean

B. Fill missing 'age' with median, then replace negative 'income' with zero

C. Fill missing 'age' with zero, then delete rows with negative 'income'

D. Replace negative 'income' with median, then fill missing 'age' with mean

Solution

1. Step 1: Handle missing 'age' values

   Filling missing 'age' with median preserves data without bias from zeros or deletion.

2. Step 2: Fix negative 'income' errors

   Replacing negative 'income' with zero corrects errors without deleting data.

3. Final Answer:

   Fill missing 'age' with median, then replace negative 'income' with zero -> Option B

4. Quick Check:

   Median fill + zero fix = best cleaning [OK]

Hint: Fill missing with median; fix errors with zero [OK]

Common Mistakes:

• Deleting rows unnecessarily
• Filling missing with zero causing bias
• Replacing errors with mean without fixing negatives