NaN and None in Pandas - Step-by-Step Execution

Concept Flow - NaN and None in Pandas

Create DataFrame with None and NaN

↓

Pandas stores None as NaN in float columns

↓

Check for missing values using isna()

↓

Operations handle NaN and None similarly

↓

Fill or drop missing values as needed

↓

Result DataFrame

Shows how pandas treats None and NaN as missing values and how to detect and handle them.

Execution Sample

Pandas

import pandas as pd
import numpy as np
data = {'A': [1, None, 3], 'B': [4, np.nan, 6]}
df = pd.DataFrame(data)
df.isna()

Creates a DataFrame with None and NaN, then checks which values are missing.

Execution Table

Step	Action	DataFrame 'df'	isna() Output
1	Create DataFrame with None and NaN	{'A': [1, None, 3], 'B': [4, nan, 6]}	N/A
2	Pandas converts None to NaN in column 'A'	{'A': [1.0, NaN, 3.0], 'B': [4.0, NaN, 6.0]}	N/A
3	Call df.isna() to detect missing values	Same as step 2	{'A': [False, True, False], 'B': [False, True, False]}
4	Use fillna() to replace NaN with 0	Same as step 2	N/A
5	After fillna(0), missing values replaced	{'A': [1.0, 0.0, 3.0], 'B': [4.0, 0.0, 6.0]}	N/A
6	Drop rows with missing values using dropna()	Same as step 2	N/A
7	After dropna(), rows with NaN removed	{'A': [1.0, 3.0], 'B': [4.0, 6.0]}	N/A
8	End of execution	Final DataFrame after operations	Final isna() output if checked

💡 All missing values detected and handled; execution ends after fill or drop operations.

Variable Tracker

Variable	Start	After Step 2	After Step 5	After Step 7
df	{'A': [1, None, 3], 'B': [4, nan, 6]}	{'A': [1.0, NaN, 3.0], 'B': [4.0, NaN, 6.0]}	{'A': [1.0, 0.0, 3.0], 'B': [4.0, 0.0, 6.0]}	{'A': [1.0, 3.0], 'B': [4.0, 6.0]}
isna_output	N/A	N/A	{'A': [False, False, False], 'B': [False, False, False]}	N/A

Key Moments - 3 Insights

Why does pandas convert None to NaN in the DataFrame?

Pandas uses NaN to represent missing values in numeric columns because None is a Python object and cannot be stored in numeric arrays. This is shown in execution_table step 2 where None becomes NaN.

Does isna() detect both None and NaN as missing values?

Yes, isna() treats both None and NaN as missing values. In the execution_table step 3, both None (converted to NaN) and NaN are detected as True.

What happens if you try to fill missing values with fillna()?

fillna() replaces all NaN (and None converted to NaN) with the specified value. See execution_table step 5 where NaNs become 0.

Visual Quiz - 3 Questions

Look at the execution_table step 3, what is the isna() output for df['A'][1]?

A False

B True

C None

D 0

💡 Hint

Check the 'isna() Output' column at step 3 for column 'A' row 1.

At which step does the DataFrame replace missing values with 0?

A Step 5

B Step 7

C Step 3

D Step 2

💡 Hint

Look at the 'Action' column describing fillna() usage.

If you drop rows with missing values, what happens to the number of rows in df?

A It stays the same

B It increases

C It decreases

D It becomes zero

💡 Hint

See execution_table step 7 where dropna() removes rows with NaN.

Concept Snapshot

NaN and None in Pandas:
- None is converted to NaN in numeric columns.
- Both represent missing data.
- Use df.isna() to detect missing values.
- Use fillna() to replace missing values.
- Use dropna() to remove missing data rows.

Full Transcript

This visual execution shows how pandas handles missing data using None and NaN. When creating a DataFrame with None and NaN, pandas converts None to NaN in numeric columns. The isna() function detects both as missing values. You can replace missing values using fillna() or remove rows with missing data using dropna(). The variable tracker shows how the DataFrame changes after each operation. Key moments clarify why None becomes NaN and how missing values are detected and handled. The quiz tests understanding of these steps.

Practice

1. What do NaN and None represent in a pandas DataFrame?

easy

A. Missing or null data values

B. Valid numeric data

C. Column headers

D. Index labels

Solution

1. Step 1: Understand the role of NaN and None in pandas

   Both NaN and None are used to mark missing or null data in pandas DataFrames.

2. Step 2: Differentiate from other DataFrame elements

   They do not represent valid data, headers, or indexes but indicate absence of data.

3. Final Answer:

   Missing or null data values -> Option A

4. Quick Check:

   NaN and None = Missing data [OK]

Hint: NaN and None always mean missing data in pandas [OK]

Common Mistakes:

• Thinking NaN is a valid number
• Confusing None with column names
• Assuming NaN is an index label

2. Which of the following is the correct way to check for missing values in a pandas DataFrame df?

easy

A. df.isna()

B. df.notna()

C. df.isnan()

D. df.ismissing()

Solution

1. Step 1: Recall pandas functions for missing data

   The correct function to check for missing values is isna(), which returns a boolean DataFrame indicating missing entries.

2. Step 2: Identify incorrect options

   notna() returns a boolean DataFrame indicating non-missing entries; isnan() and ismissing() do not exist in pandas.

3. Final Answer:

   df.isna() -> Option A

4. Quick Check:

   Use isna() to find missing values [OK]

Hint: Use df.isna() to find missing values in pandas [OK]

Common Mistakes:

• Using non-existent df.isnan()
• Using df.notna() which checks non-missing values
• Trying df.ismissing() which is invalid

3. Given the code:

import pandas as pd
import numpy as np
df = pd.DataFrame({'A': [1, np.nan, 3], 'B': [None, 2, 3]})
result = df.isna().sum().to_dict()

What is the value of result?

medium

A. {'A': 0, 'B': 0}

B. {'A': 0, 'B': 1}

C. {'A': 1, 'B': 0}

D. {'A': 1, 'B': 1}

Solution

1. Step 1: Identify missing values in each column

   Column 'A' has one np.nan (missing), column 'B' has one None (missing).

2. Step 2: Use isna() and sum() to count missing per column

   df.isna() marks missing as True, sum() counts Trues per column, resulting in {'A': 1, 'B': 1}.

3. Final Answer:

   {'A': 1, 'B': 1} -> Option D

4. Quick Check:

   Count missing with isna().sum() = {'A':1,'B':1} [OK]

Hint: Count missing with df.isna().sum() per column [OK]

Common Mistakes:

• Ignoring None as missing in column B
• Confusing sum() to count non-missing
• Assuming np.nan and None are treated differently here

4. Consider this code snippet:

import pandas as pd
df = pd.DataFrame({'X': [1, None, 3]})
df['X'] = df['X'].fillna(0)
print(df)

What is the output?

medium

A. X
0 1
1 3
2 0

B. X
0 1
1 None
2 3

C. X
0 1
1 0
2 3

D. X
0 1
1 NaN
2 3

Solution

1. Step 1: Identify missing value in column 'X'

   The second row has None, which pandas treats as missing (NaN).

2. Step 2: Use fillna(0) to replace missing with zero

   The fillna(0) replaces the missing value with 0, so the DataFrame becomes 1, 0, 3.

3. Final Answer:

   X 0 1 1 0 2 3 -> Option C

4. Quick Check:

   fillna(0) replaces None with 0 [OK]

Hint: fillna(0) replaces missing values with zero [OK]

Common Mistakes:

• Expecting None to remain after fillna
• Confusing fillna with dropna
• Assuming fillna changes non-missing values

5. You have a DataFrame df with columns 'A' and 'B' containing some NaN and None values. You want to replace all missing values in 'A' with the mean of 'A' and all missing values in 'B' with the string 'missing'. Which code correctly does this?

hard

A. df.fillna({'A': df['A'].mean(), 'B': 'missing'})

B. df['A'] = df['A'].fillna(df['A'].mean())
df['B'] = df['B'].fillna('missing')

C. df['A'].fillna(df['A'].mean())
df['B'].fillna('missing')

D. df.fillna(df.mean())
df['B'] = df['B'].fillna('missing')

Solution

1. Step 1: Replace missing in 'A' with its mean

   Use df['A'].fillna(df['A'].mean()) and assign back to 'A' to update the column.

2. Step 2: Replace missing in 'B' with 'missing'

   Use df['B'].fillna('missing') and assign back to 'B' to update that column.

3. Step 3: Check other options for correctness

   df.fillna({'A': df['A'].mean(), 'B': 'missing'}) does not assign back, so changes are not saved; df['A'].fillna(df['A'].mean()) df['B'].fillna('missing') also misses assignment; df.fillna(df.mean()) df['B'] = df['B'].fillna('missing') tries to fill numeric mean for all columns which fails for 'B'.

4. Final Answer:

   df['A'] = df['A'].fillna(df['A'].mean()) df['B'] = df['B'].fillna('missing') -> Option B

5. Quick Check:

   Assign fillna results back to columns [OK]

Hint: Assign fillna results back to columns to save changes [OK]

Common Mistakes:

• Not assigning fillna result back to DataFrame
• Using fillna without specifying columns
• Trying to fill numeric mean into string column