Why handling missing data matters in Pandas - See It in Action

Why handling missing data matters

📖 Scenario: Imagine you work in a health clinic. You have a small table of patient data with their ages and blood pressure readings. Some readings are missing. You want to understand why missing data can cause problems before fixing it.

🎯 Goal: You will create a small patient data table with missing values, count how many values are missing, and see how missing data affects calculations like average blood pressure.

📋 What You'll Learn

Create a pandas DataFrame with patient names, ages, and blood pressure readings including missing values

Create a variable to count missing blood pressure values

Calculate the average blood pressure without handling missing data

Print the count of missing values and the average blood pressure

💡 Why This Matters

🌍 Real World

In real health data, missing measurements happen often. Knowing how to find and handle them helps keep analysis accurate.

💼 Career

Data scientists must detect and manage missing data to build reliable models and reports.

1

Create patient data with missing values

Create a pandas DataFrame called patients with these exact columns and values:
'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eva']
'Age': [25, 30, 35, 40, 45]
'BloodPressure': [120, None, 130, None, 110]
Use import pandas as pd before creating the DataFrame.

Pandas

import pandas as pd
# Create the patients DataFrame with missing blood pressure values
# Your code here

Hint

Use pd.DataFrame with a dictionary of lists. Use None for missing values.

2

Count missing blood pressure values

Create a variable called missing_count that counts how many missing values are in the 'BloodPressure' column of patients. Use the isna() method and sum().

Pandas

import pandas as pd
patients = pd.DataFrame({
    'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eva'],
    'Age': [25, 30, 35, 40, 45],
    'BloodPressure': [120, None, 130, None, 110]
})
# Count missing values in the 'BloodPressure' column
# Your code here

Hint

Use patients['BloodPressure'].isna() to get True for missing values, then sum them.

3

Calculate average blood pressure without handling missing data

Create a variable called average_bp that calculates the mean of the 'BloodPressure' column in patients without removing missing values explicitly. Use the mean() method directly.

Pandas

import pandas as pd
patients = pd.DataFrame({
    'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eva'],
    'Age': [25, 30, 35, 40, 45],
    'BloodPressure': [120, None, 130, None, 110]
})
missing_count = patients['BloodPressure'].isna().sum()
# Calculate the average blood pressure without handling missing data
# Your code here

Hint

Use patients['BloodPressure'].mean() directly. It ignores missing values automatically.

4

Print missing count and average blood pressure

Print the text Missing blood pressure values: followed by the value of missing_count. Then print Average blood pressure: followed by the value of average_bp. Use two separate print() statements.

Pandas

import pandas as pd
patients = pd.DataFrame({
    'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eva'],
    'Age': [25, 30, 35, 40, 45],
    'BloodPressure': [120, None, 130, None, 110]
})
missing_count = patients['BloodPressure'].isna().sum()
average_bp = patients['BloodPressure'].mean()
# Print the missing count and average blood pressure
# Your code here

Hint

Use print("Missing blood pressure values:", missing_count) and print("Average blood pressure:", average_bp).

Practice

1. Why is it important to handle missing data in a pandas DataFrame?

easy

A. Because missing data can cause errors or lead to wrong conclusions.

B. Because pandas does not allow any missing data by default.

C. Because missing data always improves model accuracy.

D. Because missing data automatically fills with zeros.

Solution

1. Step 1: Understand the impact of missing data

   Missing data can cause errors in calculations or models and lead to incorrect results.

2. Step 2: Recognize the need for handling missing data

   Handling missing data ensures the analysis is accurate and reliable.

3. Final Answer:

   Because missing data can cause errors or lead to wrong conclusions. -> Option A

4. Quick Check:

   Missing data impact = errors/wrong conclusions [OK]

Hint: Missing data can break analysis; always check first [OK]

Common Mistakes:

• Assuming pandas never allows missing data
• Thinking missing data improves accuracy
• Believing missing data fills automatically

2. Which pandas function is used to find missing values in a DataFrame?

easy

A. replace()

B. fillna()

C. dropna()

D. isnull()

Solution

1. Step 1: Identify function to detect missing data

   The isnull() function returns True for missing values in a DataFrame.

2. Step 2: Differentiate from other functions

   fillna() fills missing data, dropna() removes rows/columns with missing data, and replace() replaces specified values.

3. Final Answer:

   isnull() -> Option D

4. Quick Check:

   Find missing = isnull() [OK]

Hint: Use isnull() to spot missing data quickly [OK]

Common Mistakes:

• Confusing fillna() with isnull()
• Using dropna() to find missing data
• Thinking replace() detects missing values

3. What will be the output of this code?

import pandas as pd
df = pd.DataFrame({'A': [1, None, 3], 'B': [4, 5, None]})
print(df.isnull().sum())

medium

A. A 2
B 2
dtype: int64

B. A 0
B 0
dtype: int64

C. A 1
B 1
dtype: int64

D. A 1
B 2
dtype: int64

Solution

1. Step 1: Understand the DataFrame content

   Column 'A' has one None (missing), column 'B' has one None (missing).

2. Step 2: Apply isnull() and sum()

   isnull() marks missing as True, sum() counts True values per column, so both columns have 1 missing value.

3. Final Answer:

   A 1 B 1 dtype: int64 -> Option C

4. Quick Check:

   Count missing per column = 1 each [OK]

Hint: sum() on isnull() counts missing per column [OK]

Common Mistakes:

• Counting total missing instead of per column
• Confusing None with zero
• Expecting different counts for columns

4. Identify the error in this code snippet for handling missing data:

import pandas as pd
df = pd.DataFrame({'X': [1, None, 3]})
df.fillna(df.mean())

medium

A. fillna() returns a new DataFrame; df is unchanged without assignment.

B. df.mean() returns a Series, which cannot be used in fillna().

C. fillna() modifies df in place by default, no error here.

D. None values cannot be replaced using fillna().

Solution

1. Step 1: Understand fillna() behavior

   fillna() returns a new DataFrame by default and does not change df unless inplace=True or reassigned.

2. Step 2: Check code effect

   Since df.fillna(df.mean()) is not assigned back, df remains unchanged, so missing values are not filled.

3. Final Answer:

   fillna() returns a new DataFrame; df is unchanged without assignment. -> Option A

4. Quick Check:

   fillna() needs assignment or inplace=True [OK]

Hint: Assign fillna() result or use inplace=True to update df [OK]

Common Mistakes:

• Assuming fillna() changes df without assignment
• Thinking df.mean() is invalid input
• Believing None cannot be replaced

5. You have a DataFrame with many missing values scattered across columns. Which approach best preserves data while handling missing values for analysis?

hard

A. Use dropna() to remove all rows with any missing data.

B. Use fillna() with column means to replace missing values.

C. Ignore missing data and proceed with analysis.

D. Replace missing values with zeros regardless of context.

Solution

1. Step 1: Evaluate dropna() impact

   Dropping all rows with missing data can remove too much data, losing valuable information.

2. Step 2: Consider fillna() with column means

   Replacing missing values with column means keeps data size and reduces bias from missingness.

3. Step 3: Assess ignoring or zero replacement

   Ignoring missing data risks errors; zero replacement may distort data if zeros are not meaningful.

4. Final Answer:

   Use fillna() with column means to replace missing values. -> Option B

5. Quick Check:

   fillna() with means preserves data best [OK]

Hint: Fill missing with means to keep data and reduce bias [OK]

Common Mistakes:

• Dropping too many rows with dropna()
• Ignoring missing data effects
• Replacing missing with zeros blindly