Identifying missing values (isnull, isna) in Data Analysis Python

Introduction

We use these functions to find missing or empty spots in our data. This helps us clean and understand the data better.

When checking if survey answers are missing before analysis.

When cleaning a sales dataset to find missing prices or quantities.

When preparing data for machine learning and need to handle empty values.

When exploring a dataset to see which columns have missing information.

Syntax

Data Analysis Python

DataFrame.isnull()
DataFrame.isna()

Both isnull() and isna() do the same thing in pandas.

They return a DataFrame of the same shape with True where values are missing and False otherwise.

Examples

This shows True where values are missing in the DataFrame.

Data Analysis Python

import pandas as pd

data = {'Name': ['Anna', 'Bob', None], 'Age': [25, None, 30]}
df = pd.DataFrame(data)

print(df.isnull())

This does the same as isnull(), showing missing values.

Data Analysis Python

print(df.isna())

Check missing values only in the 'Age' column.

Data Analysis Python

print(df['Age'].isnull())

Sample Program

This program creates a small table with some missing values. Then it finds and prints where the missing values are.

Data Analysis Python

import pandas as pd

data = {'Product': ['Book', 'Pen', 'Notebook', None], 'Price': [12.99, None, 5.49, 7.99]}
df = pd.DataFrame(data)

missing_values = df.isnull()
print(missing_values)

Important Notes

Missing values can be None, NaN, or NaT in pandas.

Use these functions before filling or dropping missing data to understand where they are.

Summary

isnull() and isna() help find missing data in tables.

They return True for missing spots and False for filled spots.

Use them to check data quality before analysis or cleaning.

Practice

1. What do the isnull() and isna() functions do in pandas?

easy

A. They identify missing or empty values in a DataFrame or Series.

B. They fill missing values with zeros.

C. They remove rows with missing values.

D. They convert data types to numeric.

Solution

1. Step 1: Understand the purpose of isnull() and isna()

   Both functions check for missing data and return True where data is missing.

2. Step 2: Differentiate from other functions

   They do not fill, remove, or convert data but only detect missing values.

3. Final Answer:

   They identify missing or empty values in a DataFrame or Series. -> Option A

4. Quick Check:

   Missing data detection = C [OK]

Hint: Think: True means missing, False means present [OK]

Common Mistakes:

• Confusing detection with filling missing values
• Thinking they remove data instead of marking it
• Assuming they change data types

2. Which of the following is the correct way to check for missing values in a pandas DataFrame named df?

easy

A. df.isempty()

B. df.checknull()

C. df.isnull()

D. df.missing()

Solution

1. Step 1: Recall pandas function names for missing data

   The correct pandas function to detect missing values is isnull().

2. Step 2: Verify other options

   Functions like checknull(), missing(), and isempty() do not exist in pandas for this purpose.

3. Final Answer:

   df.isnull() -> Option C

4. Quick Check:

   Correct pandas method = D [OK]

Hint: Remember: pandas uses isnull() or isna() [OK]

Common Mistakes:

• Using non-existent function names
• Confusing with methods that fill or drop data
• Misspelling function names

3. Given the following code, what is the output?

import pandas as pd
data = {'A': [1, None, 3], 'B': [4, 5, None]}
df = pd.DataFrame(data)
print(df.isna())

medium

A. A DataFrame with all False values.

B. A DataFrame showing True where values are missing and False otherwise.

C. A DataFrame with all True values.

D. An error because isna() is not a valid function.

Solution

1. Step 1: Understand the input DataFrame

   The DataFrame has missing values at df['A'][1] and df['B'][2].

2. Step 2: Apply isna() method

   This method returns True at missing positions and False elsewhere.

3. Final Answer:

   A DataFrame showing True where values are missing and False otherwise. -> Option B

4. Quick Check:

   Missing spots = True, others = False [OK]

Hint: Look for None or NaN spots turning True [OK]

Common Mistakes:

• Thinking isna() fills missing values
• Expecting all True or all False output
• Confusing isna() with dropna()

4. What is wrong with this code snippet to detect missing values?

import pandas as pd
data = {'X': [10, 20, None]}
df = pd.DataFrame(data)
print(df.isnull)

medium

A. The variable name df is invalid.

B. The DataFrame is not created correctly.

C. The isnull function does not exist in pandas.

D. The code is missing parentheses after isnull to call the function.

Solution

1. Step 1: Check the usage of isnull

   isnull is a function and needs parentheses to execute, like df.isnull().

2. Step 2: Verify other parts of the code

   DataFrame creation and variable names are correct; isnull exists in pandas.

3. Final Answer:

   The code is missing parentheses after isnull to call the function. -> Option D

4. Quick Check:

   Function call needs () = A [OK]

Hint: Remember to add () to call functions [OK]

Common Mistakes:

• Forgetting parentheses on function calls
• Assuming isnull is an attribute, not a method
• Misnaming variables or functions

5. You have a DataFrame df with columns ['Name', 'Age', 'Score']. Some Age and Score values are missing. How do you create a new column 'MissingCount' that counts how many missing values each row has?

hard

A. df['MissingCount'] = df.isnull().sum(axis=1)

B. df['MissingCount'] = df.isnull().sum()

C. df['MissingCount'] = df.isnull().count(axis=1)

D. df['MissingCount'] = df.isnull().mean(axis=1)

Solution

1. Step 1: Use isnull() to detect missing values

   This returns True/False for each cell indicating missing data.

2. Step 2: Sum missing values per row with sum(axis=1)

   Summing True values (counted as 1) across columns gives missing count per row.

3. Final Answer:

   df['MissingCount'] = df.isnull().sum(axis=1) -> Option A

4. Quick Check:

   Sum True per row = B [OK]

Hint: Sum isnull() across columns with axis=1 [OK]

Common Mistakes:

• Summing without axis=1 sums all missing in DataFrame
• Using count() instead of sum() to count missing
• Using mean() which gives fraction, not count