Filling missing values (fillna) in Data Analysis Python

Introduction

Sometimes data has empty spots called missing values. Filling them helps us keep the data complete and ready for analysis.

You have a table with some empty cells and want to replace them with a number or word.

You want to fill missing dates in a timeline with the last known value.

You need to prepare data for a machine learning model that can't handle empty values.

You want to replace missing survey answers with a default response.

Syntax

Data Analysis Python

DataFrame.fillna(value=None, method=None, inplace=False, limit=None)

value: The value to replace missing spots with (like 0 or 'unknown').

method: Use 'ffill' to fill forward or 'bfill' to fill backward from nearby values.

Examples

Replace all missing values with zero.

Data Analysis Python

df.fillna(0)

Fill missing 'Age' with 30 and missing 'Name' with 'Unknown'.

Data Analysis Python

df.fillna({'Age': 30, 'Name': 'Unknown'})

Fill missing values by copying the previous row's value down.

Data Analysis Python

df.fillna(method='ffill')

Fill missing values by copying the next row's value up, but only for one missing spot in a row.

Data Analysis Python

df.fillna(method='bfill', limit=1)

Sample Program

This code creates a table with some missing spots. It fills missing ages with 0, missing names with 'Unknown', and fills missing cities by copying the previous city down.

Data Analysis Python

import pandas as pd

# Create a simple table with missing values
 data = {'Name': ['Alice', None, 'Charlie', 'David'],
         'Age': [25, None, 30, None],
         'City': ['New York', 'Los Angeles', None, 'Chicago']}

df = pd.DataFrame(data)

print('Original DataFrame:')
print(df)

# Fill missing values: Age with 0, Name with 'Unknown', City forward fill
filled_df = df.fillna({'Age': 0, 'Name': 'Unknown'}).fillna(method='ffill')

print('\nDataFrame after filling missing values:')
print(filled_df)

Important Notes

Using inplace=True changes the original table directly without making a new one.

Filling missing values helps avoid errors in calculations or models that don't accept empty spots.

Summary

Missing values can be filled with a fixed value or by copying nearby values.

Use fillna() to replace missing spots in your data.

Filling missing data keeps your analysis smooth and error-free.

Practice

1. What does the fillna() function do in a pandas DataFrame?

easy

A. It converts all values to strings.

B. It replaces missing values with a specified value or method.

C. It sorts the DataFrame by missing values.

D. It deletes rows with missing values.

Solution

1. Step 1: Understand the purpose of fillna()

   The fillna() function is designed to handle missing data by replacing NaN values.

2. Step 2: Identify what fillna() replaces missing values with

   It can replace missing values with a fixed value or use methods like forward fill or backward fill.

3. Final Answer:

   It replaces missing values with a specified value or method. -> Option B

4. Quick Check:

   fillna() replaces NaN values = D [OK]

Hint: fillna() fills missing spots, not deletes or sorts [OK]

Common Mistakes:

• Confusing fillna() with dropna() which deletes rows
• Thinking fillna() sorts data
• Assuming fillna() changes data types

2. Which of the following is the correct syntax to fill missing values in column 'Age' with 30 in a pandas DataFrame df?

easy

A. df['Age'].fill(30)

B. df.fillna['Age'](30)

C. df.fillna('Age', 30)

D. df['Age'].fillna(30)

Solution

1. Step 1: Identify the correct method call on a DataFrame column

   To fill missing values in a specific column, call fillna() on that column like df['Age'].fillna(30).

2. Step 2: Check syntax correctness

   df['Age'].fillna(30) uses correct syntax: column selection with brackets and method call with parentheses and argument.

3. Final Answer:

   df['Age'].fillna(30) -> Option D

4. Quick Check:

   Correct method call syntax = A [OK]

Hint: Use brackets for column, parentheses for fillna() [OK]

Common Mistakes:

• Using square brackets instead of parentheses for fillna()
• Calling fillna on DataFrame with wrong syntax
• Using fill() instead of fillna()

3. Given the DataFrame df below, what will be the output of df['Score'].fillna(0)?

import pandas as pd
df = pd.DataFrame({'Score': [10, None, 20, None, 30]})
print(df['Score'].fillna(0))

medium

A. [0, 0, 0, 0, 0]

B. [10, None, 20, None, 30]

C. [10, 0, 20, 0, 30]

D. [10, 10, 20, 20, 30]

Solution

1. Step 1: Understand fillna(0) effect on NaN values

   Calling fillna(0) replaces all missing (NaN) values with 0.

2. Step 2: Apply fillna(0) to the 'Score' column

   The original column has values [10, NaN, 20, NaN, 30]. After fillna(0), NaNs become 0, so result is [10, 0, 20, 0, 30].

3. Final Answer:

   [10, 0, 20, 0, 30] -> Option C

4. Quick Check:

   NaN replaced by 0 = C [OK]

Hint: fillna(value) replaces NaN with value directly [OK]

Common Mistakes:

• Expecting fillna() to remove rows
• Thinking fillna() copies previous values by default
• Confusing fillna(0) with fillna(method='ffill')

4. What is wrong with this code snippet to fill missing values in DataFrame df?

df.fillna(method='forward')

medium

A. The method name should be 'ffill', not 'forward'.

B. fillna() cannot use methods to fill missing values.

C. The method argument should be a number, not a string.

D. fillna() must be called on a column, not the whole DataFrame.

Solution

1. Step 1: Check valid method names for fillna()

   fillna() supports method='ffill' for forward fill and method='bfill' for backward fill.

2. Step 2: Identify the incorrect method name

   The code uses method='forward', which is invalid. The correct name is 'ffill'.

3. Final Answer:

   The method name should be 'ffill', not 'forward'. -> Option A

4. Quick Check:

   Correct method name = 'ffill' [OK]

Hint: Use 'ffill' or 'bfill' for method in fillna() [OK]

Common Mistakes:

• Using 'forward' instead of 'ffill'
• Thinking fillna() can't use methods
• Calling fillna() only on columns, not DataFrame

5. You have a DataFrame df with missing values in multiple columns. You want to fill missing values in column 'A' with the mean of 'A', and in column 'B' with the previous valid value. Which code correctly does this?

hard

A. df['A'] = df['A'].fillna(df['A'].mean())
df['B'] = df['B'].fillna(method='ffill')

B. df.fillna({'A': df['A'].mean(), 'B': 'ffill'})

C. df['A'].fillna(df['A'].mean())
df['B'].fillna(method='bfill')

D. df.fillna(method={'A': 'mean', 'B': 'ffill'})

Solution

1. Step 1: Fill column 'A' with its mean

   Calculate mean with df['A'].mean() and fill missing values using fillna() on column 'A'. Assign back to df['A'].

2. Step 2: Fill column 'B' with previous valid value

   Use fillna(method='ffill') on column 'B' to forward fill missing values. Assign back to df['B'].

3. Final Answer:

   df['A'] = df['A'].fillna(df['A'].mean()) df['B'] = df['B'].fillna(method='ffill') -> Option A

4. Quick Check:

   Mean fill + forward fill = B [OK]

Hint: Fill columns separately with correct methods, assign back [OK]

Common Mistakes:

• Using fillna() without assignment, so no change saved
• Passing 'ffill' as string value instead of method argument
• Trying to use method='mean' which is invalid