Dropping missing values with dropna() in Pandas - Time & Space Complexity

Time Complexity: Dropping missing values with dropna()

O(n x m)

Understanding Time Complexity

We want to understand how the time to remove missing values grows as the data gets bigger.

How does the work change when we have more rows in our data?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

data = pd.DataFrame({
    'A': [1, 2, None, 4],
    'B': [None, 2, 3, 4],
    'C': [1, None, 3, 4]
})

clean_data = data.dropna()

This code creates a small table with some missing values and removes all rows that have any missing value.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Checking each cell in the table to find missing values.
• How many times: Once for every cell in the data (rows x columns).

How Execution Grows With Input

As the number of rows grows, the work grows roughly in direct proportion to the number of cells.

Input Size (rows)	Approx. Operations (cells checked)
10	10 x columns
100	100 x columns
1000	1000 x columns

Pattern observation: Doubling rows doubles the work, since every cell is checked once.

Final Time Complexity

Time Complexity: O(n * m)

This means the time grows proportionally with the number of rows (n) times the number of columns (m).

Common Mistake

[X] Wrong: "dropna() only looks at rows, so time grows with rows only."

[OK] Correct: dropna() must check every cell to find missing values, so columns also affect time.

Interview Connect

Understanding how data cleaning steps like dropna() scale helps you explain your code choices clearly and confidently.

Self-Check

"What if we used dropna(axis=1) to drop columns with missing values? How would the time complexity change?"

Practice

1. What does the dropna() function do in pandas?

easy

A. Sorts the DataFrame by index

B. Fills missing values with zeros

C. Removes rows or columns with missing values

D. Calculates the mean of columns

Solution

1. Step 1: Understand the purpose of dropna()

   The dropna() function is designed to remove data that contains missing values (NaN) from a DataFrame.

2. Step 2: Identify what dropna() removes

   It removes either rows or columns that have missing values, depending on the parameters used.

3. Final Answer:

   Removes rows or columns with missing values -> Option C

4. Quick Check:

   dropna() removes missing data [OK]

Hint: Remember: dropna() drops missing data, not fills it [OK]

Common Mistakes:

• Confusing dropna() with fillna()
• Thinking dropna() sorts data
• Assuming dropna() calculates statistics

2. Which of the following is the correct syntax to drop rows with any missing values in a DataFrame df?

easy

A. df.dropna(axis=0, how='any')

B. df.dropna(axis=1, how='any')

C. df.dropna(axis=0, how='all')

D. df.dropna(axis=1, how='all')

Solution

1. Step 1: Identify axis for rows

   In pandas, axis=0 means rows, so to drop rows, use axis=0.

2. Step 2: Use how='any' to drop rows with any missing values

   The how='any' parameter drops rows if any value is missing in that row.

3. Final Answer:

   df.dropna(axis=0, how='any') -> Option A

4. Quick Check:

   Drop rows with any NaN = axis=0, how='any' [OK]

Hint: axis=0 drops rows; how='any' drops if any NaN [OK]

Common Mistakes:

• Using axis=1 to drop rows
• Using how='all' to drop rows with any NaN
• Confusing axis and how parameters

3. Given the DataFrame:

import pandas as pd
data = {'A': [1, 2, None], 'B': [4, None, 6], 'C': [7, 8, 9]}
df = pd.DataFrame(data)

What is the result of df.dropna(axis=0, how='any')?

medium

A. Rows with index 0 and 1 only

B. Rows with index 0 and 2 only

C. All rows dropped

D. Only row with index 0

Solution

1. Step 1: Identify rows with any missing values

   Row 0: A=1, B=4, C=7 (no NaN)
   Row 1: A=2, B=NaN, C=8 (has NaN)
   Row 2: A=NaN, B=6, C=9 (has NaN)

2. Step 2: Drop rows with any NaN

   Only row 0 has no missing values, so only it remains after dropna(axis=0, how='any').

3. Final Answer:

   Only row with index 0 -> Option D

4. Quick Check:

   Drop rows with any NaN leaves only complete rows [OK]

Hint: Check which rows have any NaN to keep them [OK]

Common Mistakes:

• Keeping rows with NaN
• Confusing axis=0 and axis=1
• Misreading how='any' vs how='all'

4. You run the code:

df.dropna(axis=2)

and get an error. What is the problem?

medium

A. axis=2 is invalid; only 0 (rows) or 1 (columns) allowed

B. dropna() cannot be used on DataFrames

C. Missing values must be filled before dropping

D. You must specify how='any' or how='all'

Solution

1. Step 1: Understand valid axis values for dropna()

   In pandas, axis can only be 0 (rows) or 1 (columns) for dropna().

2. Step 2: Identify the error cause

   Using axis=2 is invalid and causes an error because it is not a supported axis.

3. Final Answer:

   axis=2 is invalid; only 0 (rows) or 1 (columns) allowed -> Option A

4. Quick Check:

   Valid axis values = 0 or 1 only [OK]

Hint: axis must be 0 or 1 for dropna() [OK]

Common Mistakes:

• Using axis=2 mistakenly
• Thinking dropna() needs fillna() first
• Assuming how parameter is mandatory

5. You have a DataFrame df with some rows having all values missing and some rows having only some missing values. Which dropna() call will remove only rows where all values are missing, keeping rows with partial data?

hard

A. df.dropna(axis=0, how='any')

B. df.dropna(axis=0, how='all')

C. df.dropna(axis=1, how='any')

D. df.dropna(axis=1, how='all')

Solution

1. Step 1: Understand how='all' parameter

   Using how='all' drops rows only if all values in that row are missing.

2. Step 2: Use axis=0 to drop rows

   Since we want to drop rows, axis=0 is correct.

3. Step 3: Confirm this keeps rows with some data

   Rows with some non-missing values are kept because not all values are missing.

4. Final Answer:

   df.dropna(axis=0, how='all') -> Option B

5. Quick Check:

   Drop rows with all NaN = axis=0, how='all' [OK]

Hint: Use how='all' to drop only fully empty rows [OK]

Common Mistakes:

• Using how='any' which drops partial rows
• Using axis=1 which drops columns
• Confusing how parameter effects