Why Dropping missing values with dropna() in Pandas? - Purpose & Use Cases

The Big Idea

What if you could clean messy data in seconds instead of hours?

The Scenario

Imagine you have a big table of data with some empty spots where information is missing. You try to clean it by looking at each row and deleting the ones with missing data by hand.

The Problem

Doing this manually is slow and tiring. You might miss some rows or delete the wrong ones. It's easy to make mistakes, especially when the table is large or changes often.

The Solution

The dropna() function in pandas quickly removes all rows or columns with missing values. It does this perfectly every time, saving you hours of work and avoiding errors.

Before vs After

✗ Before

for index, row in data.iterrows():
    if row.isnull().any():
        data.drop(index, inplace=True)

✓ After

clean_data = data.dropna()

What It Enables

With dropna(), you can instantly clean your data and focus on finding insights without worrying about missing values.

Real Life Example

A health researcher collects patient data but some tests are missing. Using dropna(), they quickly remove incomplete records to analyze only full data sets.

Key Takeaways

Manual cleaning of missing data is slow and error-prone.

dropna() automates removing missing values perfectly.

This lets you focus on analysis, not data fixing.

Practice

1. What does the dropna() function do in pandas?

easy

A. Sorts the DataFrame by index

B. Fills missing values with zeros

C. Removes rows or columns with missing values

D. Calculates the mean of columns

Solution

1. Step 1: Understand the purpose of dropna()

   The dropna() function is designed to remove data that contains missing values (NaN) from a DataFrame.

2. Step 2: Identify what dropna() removes

   It removes either rows or columns that have missing values, depending on the parameters used.

3. Final Answer:

   Removes rows or columns with missing values -> Option C

4. Quick Check:

   dropna() removes missing data [OK]

Hint: Remember: dropna() drops missing data, not fills it [OK]

Common Mistakes:

• Confusing dropna() with fillna()
• Thinking dropna() sorts data
• Assuming dropna() calculates statistics

2. Which of the following is the correct syntax to drop rows with any missing values in a DataFrame df?

easy

A. df.dropna(axis=0, how='any')

B. df.dropna(axis=1, how='any')

C. df.dropna(axis=0, how='all')

D. df.dropna(axis=1, how='all')

Solution

1. Step 1: Identify axis for rows

   In pandas, axis=0 means rows, so to drop rows, use axis=0.

2. Step 2: Use how='any' to drop rows with any missing values

   The how='any' parameter drops rows if any value is missing in that row.

3. Final Answer:

   df.dropna(axis=0, how='any') -> Option A

4. Quick Check:

   Drop rows with any NaN = axis=0, how='any' [OK]

Hint: axis=0 drops rows; how='any' drops if any NaN [OK]

Common Mistakes:

• Using axis=1 to drop rows
• Using how='all' to drop rows with any NaN
• Confusing axis and how parameters

3. Given the DataFrame:

import pandas as pd
data = {'A': [1, 2, None], 'B': [4, None, 6], 'C': [7, 8, 9]}
df = pd.DataFrame(data)

What is the result of df.dropna(axis=0, how='any')?

medium

A. Rows with index 0 and 1 only

B. Rows with index 0 and 2 only

C. All rows dropped

D. Only row with index 0

Solution

1. Step 1: Identify rows with any missing values

   Row 0: A=1, B=4, C=7 (no NaN)
   Row 1: A=2, B=NaN, C=8 (has NaN)
   Row 2: A=NaN, B=6, C=9 (has NaN)

2. Step 2: Drop rows with any NaN

   Only row 0 has no missing values, so only it remains after dropna(axis=0, how='any').

3. Final Answer:

   Only row with index 0 -> Option D

4. Quick Check:

   Drop rows with any NaN leaves only complete rows [OK]

Hint: Check which rows have any NaN to keep them [OK]

Common Mistakes:

• Keeping rows with NaN
• Confusing axis=0 and axis=1
• Misreading how='any' vs how='all'

4. You run the code:

df.dropna(axis=2)

and get an error. What is the problem?

medium

A. axis=2 is invalid; only 0 (rows) or 1 (columns) allowed

B. dropna() cannot be used on DataFrames

C. Missing values must be filled before dropping

D. You must specify how='any' or how='all'

Solution

1. Step 1: Understand valid axis values for dropna()

   In pandas, axis can only be 0 (rows) or 1 (columns) for dropna().

2. Step 2: Identify the error cause

   Using axis=2 is invalid and causes an error because it is not a supported axis.

3. Final Answer:

   axis=2 is invalid; only 0 (rows) or 1 (columns) allowed -> Option A

4. Quick Check:

   Valid axis values = 0 or 1 only [OK]

Hint: axis must be 0 or 1 for dropna() [OK]

Common Mistakes:

• Using axis=2 mistakenly
• Thinking dropna() needs fillna() first
• Assuming how parameter is mandatory

5. You have a DataFrame df with some rows having all values missing and some rows having only some missing values. Which dropna() call will remove only rows where all values are missing, keeping rows with partial data?

hard

A. df.dropna(axis=0, how='any')

B. df.dropna(axis=0, how='all')

C. df.dropna(axis=1, how='any')

D. df.dropna(axis=1, how='all')

Solution

1. Step 1: Understand how='all' parameter

   Using how='all' drops rows only if all values in that row are missing.

2. Step 2: Use axis=0 to drop rows

   Since we want to drop rows, axis=0 is correct.

3. Step 3: Confirm this keeps rows with some data

   Rows with some non-missing values are kept because not all values are missing.

4. Final Answer:

   df.dropna(axis=0, how='all') -> Option B

5. Quick Check:

   Drop rows with all NaN = axis=0, how='all' [OK]

Hint: Use how='all' to drop only fully empty rows [OK]

Common Mistakes:

• Using how='any' which drops partial rows
• Using axis=1 which drops columns
• Confusing how parameter effects