Why Filling missing values (fillna) in Data Analysis Python? - Purpose & Use Cases

The Big Idea

What if you could fix all missing data in seconds instead of hours?

The Scenario

Imagine you have a big table of customer data with some empty spots where information is missing. You try to fix it by looking at each empty spot and guessing what should be there.

The Problem

Doing this by hand is slow and tiring. You might make mistakes or miss some spots. It's hard to keep track and update all missing pieces correctly.

The Solution

Using fillna lets you quickly fill all missing spots with a value you choose or a calculated one. It works fast and keeps your data clean without errors.

Before vs After

✗ Before

for i in range(len(data)):
    if data[i] is None:
        data[i] = 0

✓ After

data.fillna(0, inplace=True)

What It Enables

You can handle incomplete data easily and prepare it for analysis or machine learning without wasting time.

Real Life Example

A store's sales data has missing prices for some products. Filling those missing prices with the average price helps calculate total sales correctly.

Key Takeaways

Manual filling is slow and error-prone.

fillna fills all missing values quickly and safely.

This makes data ready for better analysis and decisions.

Practice

1. What does the fillna() function do in a pandas DataFrame?

easy

A. It converts all values to strings.

B. It replaces missing values with a specified value or method.

C. It sorts the DataFrame by missing values.

D. It deletes rows with missing values.

Solution

1. Step 1: Understand the purpose of fillna()

   The fillna() function is designed to handle missing data by replacing NaN values.

2. Step 2: Identify what fillna() replaces missing values with

   It can replace missing values with a fixed value or use methods like forward fill or backward fill.

3. Final Answer:

   It replaces missing values with a specified value or method. -> Option B

4. Quick Check:

   fillna() replaces NaN values = D [OK]

Hint: fillna() fills missing spots, not deletes or sorts [OK]

Common Mistakes:

• Confusing fillna() with dropna() which deletes rows
• Thinking fillna() sorts data
• Assuming fillna() changes data types

2. Which of the following is the correct syntax to fill missing values in column 'Age' with 30 in a pandas DataFrame df?

easy

A. df['Age'].fill(30)

B. df.fillna['Age'](30)

C. df.fillna('Age', 30)

D. df['Age'].fillna(30)

Solution

1. Step 1: Identify the correct method call on a DataFrame column

   To fill missing values in a specific column, call fillna() on that column like df['Age'].fillna(30).

2. Step 2: Check syntax correctness

   df['Age'].fillna(30) uses correct syntax: column selection with brackets and method call with parentheses and argument.

3. Final Answer:

   df['Age'].fillna(30) -> Option D

4. Quick Check:

   Correct method call syntax = A [OK]

Hint: Use brackets for column, parentheses for fillna() [OK]

Common Mistakes:

• Using square brackets instead of parentheses for fillna()
• Calling fillna on DataFrame with wrong syntax
• Using fill() instead of fillna()

3. Given the DataFrame df below, what will be the output of df['Score'].fillna(0)?

import pandas as pd
df = pd.DataFrame({'Score': [10, None, 20, None, 30]})
print(df['Score'].fillna(0))

medium

A. [0, 0, 0, 0, 0]

B. [10, None, 20, None, 30]

C. [10, 0, 20, 0, 30]

D. [10, 10, 20, 20, 30]

Solution

1. Step 1: Understand fillna(0) effect on NaN values

   Calling fillna(0) replaces all missing (NaN) values with 0.

2. Step 2: Apply fillna(0) to the 'Score' column

   The original column has values [10, NaN, 20, NaN, 30]. After fillna(0), NaNs become 0, so result is [10, 0, 20, 0, 30].

3. Final Answer:

   [10, 0, 20, 0, 30] -> Option C

4. Quick Check:

   NaN replaced by 0 = C [OK]

Hint: fillna(value) replaces NaN with value directly [OK]

Common Mistakes:

• Expecting fillna() to remove rows
• Thinking fillna() copies previous values by default
• Confusing fillna(0) with fillna(method='ffill')

4. What is wrong with this code snippet to fill missing values in DataFrame df?

df.fillna(method='forward')

medium

A. The method name should be 'ffill', not 'forward'.

B. fillna() cannot use methods to fill missing values.

C. The method argument should be a number, not a string.

D. fillna() must be called on a column, not the whole DataFrame.

Solution

1. Step 1: Check valid method names for fillna()

   fillna() supports method='ffill' for forward fill and method='bfill' for backward fill.

2. Step 2: Identify the incorrect method name

   The code uses method='forward', which is invalid. The correct name is 'ffill'.

3. Final Answer:

   The method name should be 'ffill', not 'forward'. -> Option A

4. Quick Check:

   Correct method name = 'ffill' [OK]

Hint: Use 'ffill' or 'bfill' for method in fillna() [OK]

Common Mistakes:

• Using 'forward' instead of 'ffill'
• Thinking fillna() can't use methods
• Calling fillna() only on columns, not DataFrame

5. You have a DataFrame df with missing values in multiple columns. You want to fill missing values in column 'A' with the mean of 'A', and in column 'B' with the previous valid value. Which code correctly does this?

hard

A. df['A'] = df['A'].fillna(df['A'].mean())
df['B'] = df['B'].fillna(method='ffill')

B. df.fillna({'A': df['A'].mean(), 'B': 'ffill'})

C. df['A'].fillna(df['A'].mean())
df['B'].fillna(method='bfill')

D. df.fillna(method={'A': 'mean', 'B': 'ffill'})

Solution

1. Step 1: Fill column 'A' with its mean

   Calculate mean with df['A'].mean() and fill missing values using fillna() on column 'A'. Assign back to df['A'].

2. Step 2: Fill column 'B' with previous valid value

   Use fillna(method='ffill') on column 'B' to forward fill missing values. Assign back to df['B'].

3. Final Answer:

   df['A'] = df['A'].fillna(df['A'].mean()) df['B'] = df['B'].fillna(method='ffill') -> Option A

4. Quick Check:

   Mean fill + forward fill = B [OK]

Hint: Fill columns separately with correct methods, assign back [OK]

Common Mistakes:

• Using fillna() without assignment, so no change saved
• Passing 'ffill' as string value instead of method argument
• Trying to use method='mean' which is invalid