Filling missing values (fillna) in Data Analysis Python - Step-by-Step Execution

Concept Flow - Filling missing values (fillna)

Start with DataFrame

↓

Identify missing values

↓

Choose fill value or method

↓

Apply fillna()

↓

Get DataFrame with no missing values

↓

End

We start with data that has missing values, decide how to fill them, apply fillna(), and get a complete dataset.

Execution Sample

Data Analysis Python

import pandas as pd

df = pd.DataFrame({'A': [1, None, 3], 'B': [4, 5, None]})
df_filled = df.fillna(0)
print(df_filled)

This code fills missing values in a DataFrame with 0.

Execution Table

Step	DataFrame State	fillna() Action	Resulting DataFrame
1	{'A': [1, None, 3], 'B': [4, 5, None]}	Identify missing values at positions A[1], B[2]	{'A': [1, None, 3], 'B': [4, 5, None]}
2	Same as step 1	Apply fillna(0) to replace None with 0	{'A': [1, 0, 3], 'B': [4, 5, 0]}
3	No missing values remain	Print filled DataFrame	A B 0 1 4 1 0 5 2 3 0
4	Execution ends	No more steps	End of process

💡 All missing values replaced by 0, no None left in DataFrame

Variable Tracker

Variable	Start	After fillna	Final
df	{'A': [1, None, 3], 'B': [4, 5, None]}	No change	Same as start
df_filled	Not defined	{'A': [1, 0, 3], 'B': [4, 5, 0]}	Same as after fillna

Key Moments - 2 Insights

Why does fillna(0) replace None with 0 in the DataFrame?

fillna(0) looks for missing values (None or NaN) and replaces them with 0, as shown in execution_table step 2.

Does fillna() change the original DataFrame?

No, fillna() returns a new DataFrame with filled values. The original stays the same unless you use inplace=True, as seen in variable_tracker.

Visual Quiz - 3 Questions

Look at the execution table at step 2, what value replaces the missing value in column 'A'?

A None

B 0

C 1

D NaN

💡 Hint

Check the 'Resulting DataFrame' column in step 2 of the execution_table.

At which step does the DataFrame have no missing values?

A Step 2

B Step 3

C Step 1

D Step 4

💡 Hint

Look at the 'DataFrame State' and 'Resulting DataFrame' columns in the execution_table.

If we used fillna(method='ffill') instead of fillna(0), what would happen to the missing value in 'B' at index 2?

A It would remain None

B It would be replaced by 0

C It would be replaced by 5

D It would be replaced by 4

💡 Hint

Forward fill (ffill) replaces missing values with the previous non-missing value, check variable_tracker for values in 'B'.

Concept Snapshot

fillna(value) replaces missing values in a DataFrame with the given value.
It returns a new DataFrame unless inplace=True is used.
Missing values include None or NaN.
You can also use methods like 'ffill' or 'bfill' to fill.
Useful to clean data before analysis.

Full Transcript

We start with a DataFrame that has missing values represented by None. We identify these missing spots. Then, we choose a value to fill them, here zero. Using fillna(0), we replace all missing values with zero, creating a new DataFrame. The original DataFrame remains unchanged. Finally, we print the filled DataFrame which has no missing values. This process helps prepare data for analysis by removing gaps.

Practice

1. What does the fillna() function do in a pandas DataFrame?

easy

A. It converts all values to strings.

B. It replaces missing values with a specified value or method.

C. It sorts the DataFrame by missing values.

D. It deletes rows with missing values.

Solution

1. Step 1: Understand the purpose of fillna()

   The fillna() function is designed to handle missing data by replacing NaN values.

2. Step 2: Identify what fillna() replaces missing values with

   It can replace missing values with a fixed value or use methods like forward fill or backward fill.

3. Final Answer:

   It replaces missing values with a specified value or method. -> Option B

4. Quick Check:

   fillna() replaces NaN values = D [OK]

Hint: fillna() fills missing spots, not deletes or sorts [OK]

Common Mistakes:

• Confusing fillna() with dropna() which deletes rows
• Thinking fillna() sorts data
• Assuming fillna() changes data types

2. Which of the following is the correct syntax to fill missing values in column 'Age' with 30 in a pandas DataFrame df?

easy

A. df['Age'].fill(30)

B. df.fillna['Age'](30)

C. df.fillna('Age', 30)

D. df['Age'].fillna(30)

Solution

1. Step 1: Identify the correct method call on a DataFrame column

   To fill missing values in a specific column, call fillna() on that column like df['Age'].fillna(30).

2. Step 2: Check syntax correctness

   df['Age'].fillna(30) uses correct syntax: column selection with brackets and method call with parentheses and argument.

3. Final Answer:

   df['Age'].fillna(30) -> Option D

4. Quick Check:

   Correct method call syntax = A [OK]

Hint: Use brackets for column, parentheses for fillna() [OK]

Common Mistakes:

• Using square brackets instead of parentheses for fillna()
• Calling fillna on DataFrame with wrong syntax
• Using fill() instead of fillna()

3. Given the DataFrame df below, what will be the output of df['Score'].fillna(0)?

import pandas as pd
df = pd.DataFrame({'Score': [10, None, 20, None, 30]})
print(df['Score'].fillna(0))

medium

A. [0, 0, 0, 0, 0]

B. [10, None, 20, None, 30]

C. [10, 0, 20, 0, 30]

D. [10, 10, 20, 20, 30]

Solution

1. Step 1: Understand fillna(0) effect on NaN values

   Calling fillna(0) replaces all missing (NaN) values with 0.

2. Step 2: Apply fillna(0) to the 'Score' column

   The original column has values [10, NaN, 20, NaN, 30]. After fillna(0), NaNs become 0, so result is [10, 0, 20, 0, 30].

3. Final Answer:

   [10, 0, 20, 0, 30] -> Option C

4. Quick Check:

   NaN replaced by 0 = C [OK]

Hint: fillna(value) replaces NaN with value directly [OK]

Common Mistakes:

• Expecting fillna() to remove rows
• Thinking fillna() copies previous values by default
• Confusing fillna(0) with fillna(method='ffill')

4. What is wrong with this code snippet to fill missing values in DataFrame df?

df.fillna(method='forward')

medium

A. The method name should be 'ffill', not 'forward'.

B. fillna() cannot use methods to fill missing values.

C. The method argument should be a number, not a string.

D. fillna() must be called on a column, not the whole DataFrame.

Solution

1. Step 1: Check valid method names for fillna()

   fillna() supports method='ffill' for forward fill and method='bfill' for backward fill.

2. Step 2: Identify the incorrect method name

   The code uses method='forward', which is invalid. The correct name is 'ffill'.

3. Final Answer:

   The method name should be 'ffill', not 'forward'. -> Option A

4. Quick Check:

   Correct method name = 'ffill' [OK]

Hint: Use 'ffill' or 'bfill' for method in fillna() [OK]

Common Mistakes:

• Using 'forward' instead of 'ffill'
• Thinking fillna() can't use methods
• Calling fillna() only on columns, not DataFrame

5. You have a DataFrame df with missing values in multiple columns. You want to fill missing values in column 'A' with the mean of 'A', and in column 'B' with the previous valid value. Which code correctly does this?

hard

A. df['A'] = df['A'].fillna(df['A'].mean())
df['B'] = df['B'].fillna(method='ffill')

B. df.fillna({'A': df['A'].mean(), 'B': 'ffill'})

C. df['A'].fillna(df['A'].mean())
df['B'].fillna(method='bfill')

D. df.fillna(method={'A': 'mean', 'B': 'ffill'})

Solution

1. Step 1: Fill column 'A' with its mean

   Calculate mean with df['A'].mean() and fill missing values using fillna() on column 'A'. Assign back to df['A'].

2. Step 2: Fill column 'B' with previous valid value

   Use fillna(method='ffill') on column 'B' to forward fill missing values. Assign back to df['B'].

3. Final Answer:

   df['A'] = df['A'].fillna(df['A'].mean()) df['B'] = df['B'].fillna(method='ffill') -> Option A

4. Quick Check:

   Mean fill + forward fill = B [OK]

Hint: Fill columns separately with correct methods, assign back [OK]

Common Mistakes:

• Using fillna() without assignment, so no change saved
• Passing 'ffill' as string value instead of method argument
• Trying to use method='mean' which is invalid