Filling missing values (fillna) in Data Analysis Python - Time & Space Complexity

Time Complexity: Filling missing values (fillna)

O(n)

Understanding Time Complexity

When we fill missing values in data, we want to know how long it takes as the data grows.

We ask: How does the time to fill missing spots change when the data gets bigger?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

data = pd.Series([1, None, 3, None, 5])
filled_data = data.fillna(0)

This code replaces missing values in a list of numbers with zero.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Checking each element in the data to see if it is missing.
• How many times: Once for every item in the data list.

How Execution Grows With Input

As the data list grows, the time to fill missing values grows in a straight line.

Input Size (n)	Approx. Operations
10	10 checks and fills
100	100 checks and fills
1000	1000 checks and fills

Pattern observation: Doubling the data doubles the work needed.

Final Time Complexity

Time Complexity: O(n)

This means the time to fill missing values grows directly with the size of the data.

Common Mistake

[X] Wrong: "Filling missing values happens instantly no matter how big the data is."

[OK] Correct: The code must check each item to find missing spots, so bigger data takes more time.

Interview Connect

Understanding how filling missing data scales helps you explain data cleaning steps clearly and confidently.

Self-Check

"What if we filled missing values using the previous value instead of a fixed number? How would the time complexity change?"

Practice

1. What does the fillna() function do in a pandas DataFrame?

easy

A. It converts all values to strings.

B. It replaces missing values with a specified value or method.

C. It sorts the DataFrame by missing values.

D. It deletes rows with missing values.

Solution

1. Step 1: Understand the purpose of fillna()

   The fillna() function is designed to handle missing data by replacing NaN values.

2. Step 2: Identify what fillna() replaces missing values with

   It can replace missing values with a fixed value or use methods like forward fill or backward fill.

3. Final Answer:

   It replaces missing values with a specified value or method. -> Option B

4. Quick Check:

   fillna() replaces NaN values = D [OK]

Hint: fillna() fills missing spots, not deletes or sorts [OK]

Common Mistakes:

• Confusing fillna() with dropna() which deletes rows
• Thinking fillna() sorts data
• Assuming fillna() changes data types

2. Which of the following is the correct syntax to fill missing values in column 'Age' with 30 in a pandas DataFrame df?

easy

A. df['Age'].fill(30)

B. df.fillna['Age'](30)

C. df.fillna('Age', 30)

D. df['Age'].fillna(30)

Solution

1. Step 1: Identify the correct method call on a DataFrame column

   To fill missing values in a specific column, call fillna() on that column like df['Age'].fillna(30).

2. Step 2: Check syntax correctness

   df['Age'].fillna(30) uses correct syntax: column selection with brackets and method call with parentheses and argument.

3. Final Answer:

   df['Age'].fillna(30) -> Option D

4. Quick Check:

   Correct method call syntax = A [OK]

Hint: Use brackets for column, parentheses for fillna() [OK]

Common Mistakes:

• Using square brackets instead of parentheses for fillna()
• Calling fillna on DataFrame with wrong syntax
• Using fill() instead of fillna()

3. Given the DataFrame df below, what will be the output of df['Score'].fillna(0)?

import pandas as pd
df = pd.DataFrame({'Score': [10, None, 20, None, 30]})
print(df['Score'].fillna(0))

medium

A. [0, 0, 0, 0, 0]

B. [10, None, 20, None, 30]

C. [10, 0, 20, 0, 30]

D. [10, 10, 20, 20, 30]

Solution

1. Step 1: Understand fillna(0) effect on NaN values

   Calling fillna(0) replaces all missing (NaN) values with 0.

2. Step 2: Apply fillna(0) to the 'Score' column

   The original column has values [10, NaN, 20, NaN, 30]. After fillna(0), NaNs become 0, so result is [10, 0, 20, 0, 30].

3. Final Answer:

   [10, 0, 20, 0, 30] -> Option C

4. Quick Check:

   NaN replaced by 0 = C [OK]

Hint: fillna(value) replaces NaN with value directly [OK]

Common Mistakes:

• Expecting fillna() to remove rows
• Thinking fillna() copies previous values by default
• Confusing fillna(0) with fillna(method='ffill')

4. What is wrong with this code snippet to fill missing values in DataFrame df?

df.fillna(method='forward')

medium

A. The method name should be 'ffill', not 'forward'.

B. fillna() cannot use methods to fill missing values.

C. The method argument should be a number, not a string.

D. fillna() must be called on a column, not the whole DataFrame.

Solution

1. Step 1: Check valid method names for fillna()

   fillna() supports method='ffill' for forward fill and method='bfill' for backward fill.

2. Step 2: Identify the incorrect method name

   The code uses method='forward', which is invalid. The correct name is 'ffill'.

3. Final Answer:

   The method name should be 'ffill', not 'forward'. -> Option A

4. Quick Check:

   Correct method name = 'ffill' [OK]

Hint: Use 'ffill' or 'bfill' for method in fillna() [OK]

Common Mistakes:

• Using 'forward' instead of 'ffill'
• Thinking fillna() can't use methods
• Calling fillna() only on columns, not DataFrame

5. You have a DataFrame df with missing values in multiple columns. You want to fill missing values in column 'A' with the mean of 'A', and in column 'B' with the previous valid value. Which code correctly does this?

hard

A. df['A'] = df['A'].fillna(df['A'].mean())
df['B'] = df['B'].fillna(method='ffill')

B. df.fillna({'A': df['A'].mean(), 'B': 'ffill'})

C. df['A'].fillna(df['A'].mean())
df['B'].fillna(method='bfill')

D. df.fillna(method={'A': 'mean', 'B': 'ffill'})

Solution

1. Step 1: Fill column 'A' with its mean

   Calculate mean with df['A'].mean() and fill missing values using fillna() on column 'A'. Assign back to df['A'].

2. Step 2: Fill column 'B' with previous valid value

   Use fillna(method='ffill') on column 'B' to forward fill missing values. Assign back to df['B'].

3. Final Answer:

   df['A'] = df['A'].fillna(df['A'].mean()) df['B'] = df['B'].fillna(method='ffill') -> Option A

4. Quick Check:

   Mean fill + forward fill = B [OK]

Hint: Fill columns separately with correct methods, assign back [OK]

Common Mistakes:

• Using fillna() without assignment, so no change saved
• Passing 'ffill' as string value instead of method argument
• Trying to use method='mean' which is invalid