Why data cleaning consumes most analysis time in Data Analysis Python - See It in Action

Why Data Cleaning Consumes Most Analysis Time

📖 Scenario: You are a data analyst working with a small dataset from a customer survey. The data has some missing values and inconsistent entries. You want to understand why cleaning this data takes most of your time before you can analyze it.

🎯 Goal: Build a simple Python script that shows how to identify and handle missing and inconsistent data entries in a dataset.

📋 What You'll Learn

Create a dictionary called survey_data with customer names as keys and their ratings as values, including some missing and inconsistent entries.

Create a variable called missing_value set to None to represent missing data.

Use a for loop with variables customer and rating to iterate over survey_data.items() and create a new dictionary cleaned_data that replaces missing or invalid ratings with the average rating.

Print the cleaned_data dictionary to see the cleaned results.

💡 Why This Matters

🌍 Real World

In real life, data from surveys, sensors, or databases often have missing or wrong values. Cleaning this data is essential before any meaningful analysis.

💼 Career

Data scientists and analysts spend a large part of their work cleaning data to ensure accurate results and insights.

1

Create the initial survey data

Create a dictionary called survey_data with these exact entries: 'Alice': 5, 'Bob': None, 'Charlie': 3, 'David': 'N/A', 'Eva': 4.

Data Analysis Python

# Create the survey_data dictionary with given entries
# Your code here

Hint

Use curly braces to create a dictionary. Use None for missing values and the string 'N/A' for inconsistent entries.

2

Set the missing value indicator

Create a variable called missing_value and set it to None to represent missing data.

Data Analysis Python

survey_data = {'Alice': 5, 'Bob': None, 'Charlie': 3, 'David': 'N/A', 'Eva': 4}
# Create missing_value variable set to None
# Your code here

Hint

Just assign None to the variable missing_value.

3

Clean the data by replacing missing and invalid entries

Use a for loop with variables customer and rating to iterate over survey_data.items(). Calculate the average of valid ratings first. Then create a new dictionary called cleaned_data where missing or invalid ratings (like None or 'N/A') are replaced by the average rating.

Data Analysis Python

survey_data = {'Alice': 5, 'Bob': None, 'Charlie': 3, 'David': 'N/A', 'Eva': 4}
missing_value = None
# Calculate average rating of valid entries
# Create cleaned_data dictionary replacing missing or invalid ratings
# Your code here

Hint

First find the average of valid ratings. Then loop through each entry. If the rating is None or 'N/A', replace it with the average. Otherwise, keep the original rating.

4

Print the cleaned data

Write a print statement to display the cleaned_data dictionary.

Data Analysis Python

survey_data = {'Alice': 5, 'Bob': None, 'Charlie': 3, 'David': 'N/A', 'Eva': 4}
missing_value = None

valid_ratings = [rating for rating in survey_data.values() if isinstance(rating, int)]
average_rating = sum(valid_ratings) / len(valid_ratings)

cleaned_data = {}
for customer, rating in survey_data.items():
    if rating == missing_value or rating == 'N/A':
        cleaned_data[customer] = average_rating
    else:
        cleaned_data[customer] = rating
# Print the cleaned_data dictionary
# Your code here

Hint

Use print(cleaned_data) to show the final cleaned dictionary.