Data validation checks in Pandas - Step-by-Step Execution

Concept Flow - Data validation checks

Load DataFrame

↓

Check for missing values?

Yes→Report or fill missing

|No↓

Check data types?

Mismatch→Convert or report

↓

Check value ranges?

Out of range→Report or fix

↓

Check unique constraints?

Duplicates→Report or remove

↓

Data is valid

↓

Proceed with analysis

This flow shows how to check a DataFrame step-by-step for missing values, data types, value ranges, and duplicates to ensure data quality before analysis.

Execution Sample

Pandas

import pandas as pd

df = pd.DataFrame({
    'age': [25, 30, None, 22],
    'score': [88, 92, 85, 90]
})

missing = df.isnull().sum()

This code creates a DataFrame and checks how many missing values each column has.

Execution Table

Step	Action	Evaluation	Result
1	Create DataFrame	DataFrame with 4 rows, 2 columns	DataFrame created
2	Check missing values with df.isnull().sum()	age: 1 missing, score: 0 missing	Missing values found in 'age'
3	Check data types with df.dtypes	age: float64, score: int64	Data types as expected
4	Check value ranges for 'age' (0-120)	Values: 25,30,None,22	None is missing, others in range
5	Check duplicates with df.duplicated()	No duplicates found	Data unique
6	Decide action on missing	Fill missing with mean age	Missing values filled
7	Final DataFrame ready	No missing, valid data	Data ready for analysis

💡 All validation checks passed or handled, data is clean for analysis

Variable Tracker

Variable	Start	After Step 2	After Step 6	Final
df	{'age':[25,30,None,22],'score':[88,92,85,90]}	Same	{'age':[25,30,25.6667,22],'score':[88,92,85,90]}	Cleaned DataFrame

Key Moments - 3 Insights

Why do we check for missing values first?

Missing values can cause errors or wrong results in later steps, so we detect and handle them early as shown in step 2 and 6 of the execution_table.

What if data types don't match expected types?

If types mismatch, we may need to convert them before analysis to avoid errors, as indicated in step 3 where data types are checked.

Why check for duplicates?

Duplicates can bias analysis results, so we identify and remove them if needed, as done in step 5.

Visual Quiz - 3 Questions

Look at the execution_table, how many missing values are found in the 'age' column at step 2?

A 0

B 2

C 1

D 3

💡 Hint

Check the 'Evaluation' column in row for step 2 in execution_table

At which step is the missing value in 'age' filled?

A Step 6

B Step 3

C Step 4

D Step 5

💡 Hint

Look for the step mentioning filling missing values in execution_table

If the 'score' column had duplicates, which step would detect it?

A Step 4

B Step 5

C Step 2

D Step 7

💡 Hint

Check the step about duplicates in execution_table

Concept Snapshot

Data validation checks in pandas:
- Check missing values with df.isnull().sum()
- Verify data types with df.dtypes
- Validate value ranges manually
- Detect duplicates with df.duplicated()
- Handle issues before analysis to ensure clean data

Full Transcript

This visual execution shows how to validate data in a pandas DataFrame step-by-step. First, we create a DataFrame with some missing values. We check for missing data using isnull().sum(), finding one missing in 'age'. Next, we verify data types to ensure they are as expected. Then, we check if values fall within acceptable ranges. We also look for duplicates to avoid bias. Finally, we fill missing values with the mean and confirm the data is clean and ready for analysis.

Practice

1. What does the pandas function isnull() check in a DataFrame?

easy

A. It checks for missing or null values in the data.

B. It checks for duplicate rows in the DataFrame.

C. It returns the data types of each column.

D. It sorts the DataFrame by a column.

Solution

1. Step 1: Understand the purpose of isnull()

   The isnull() function in pandas returns a DataFrame of the same shape with True where values are missing (NaN) and False otherwise.

2. Step 2: Compare with other functions

   Functions like duplicated() check duplicates, and dtypes shows data types, so they do not check for missing values.

3. Final Answer:

   It checks for missing or null values in the data. -> Option A

4. Quick Check:

   Missing data = isnull() [OK]

Hint: Missing values? Use isnull() to find them fast [OK]

Common Mistakes:

• Confusing isnull() with duplicated()
• Thinking isnull() returns data types
• Assuming isnull() sorts data

2. Which of the following is the correct syntax to check for duplicate rows in a pandas DataFrame named df?

easy

A. df.isnull()

B. df.check_duplicates()

C. df.duplicated()

D. df.find_duplicates()

Solution

1. Step 1: Recall pandas method for duplicates

   The correct pandas method to find duplicate rows is duplicated(), called as df.duplicated().

2. Step 2: Verify other options

   isnull() checks missing data, and check_duplicates() and find_duplicates() are not valid pandas methods.

3. Final Answer:

   df.duplicated() -> Option C

4. Quick Check:

   Duplicates = duplicated() [OK]

Hint: Duplicates? Use df.duplicated() to spot them [OK]

Common Mistakes:

• Using non-existent methods like check_duplicates()
• Mixing isnull() with duplicate checks
• Forgetting parentheses after duplicated

3. Given the DataFrame df below, what will df.isnull().sum() output?

import pandas as pd
data = {'A': [1, 2, None], 'B': [None, 2, 3], 'C': [1, 2, 3]}
df = pd.DataFrame(data)

medium

A. {'A': 1, 'B': 2, 'C': 1}

B. {'A': 0, 'B': 0, 'C': 0}

C. {'A': 2, 'B': 1, 'C': 0}

D. {'A': 1, 'B': 1, 'C': 0}

Solution

1. Step 1: Identify missing values in each column

   Column 'A' has one None (missing), 'B' has one None, and 'C' has no missing values.

2. Step 2: Understand what isnull().sum() does

   isnull() marks missing as True, and sum() counts True values per column.

3. Final Answer:

   {'A': 1, 'B': 1, 'C': 0} -> Option D

4. Quick Check:

   Count missing per column = isnull().sum() [OK]

Hint: Sum isnull() to count missing per column [OK]

Common Mistakes:

• Counting total missing instead of per column
• Confusing None with zero
• Misreading DataFrame values

4. The following code is intended to find duplicate rows in df. What is the error?

duplicates = df.duplicate()

medium

A. The method name should be duplicated(), not duplicate().

B. The DataFrame df is not defined.

C. The method duplicated() requires an argument.

D. The code should use df.isnull() to find duplicates.

Solution

1. Step 1: Check method name correctness

   The correct pandas method to find duplicates is duplicated(), not duplicate().

2. Step 2: Verify other options

   df is assumed defined, duplicated() does not require arguments, and isnull() checks missing data, not duplicates.

3. Final Answer:

   The method name should be duplicated(), not duplicate(). -> Option A

4. Quick Check:

   Correct method name = duplicated() [OK]

Hint: Remember: duplicated() has a 'd' at the end [OK]

Common Mistakes:

• Misspelling duplicated() as duplicate()
• Confusing duplicated() with isnull()
• Thinking duplicated() needs arguments

5. You have a DataFrame df with columns ['id', 'age', 'score']. You want to validate that age has no missing values and score has no duplicates. Which code correctly performs both checks?

hard

A. missing_age = df.isnull().sum()['age']
duplicate_score = df['score'].isnull().any()

B. missing_age = df['age'].isnull().sum()
duplicate_score = df['score'].duplicated().any()

C. missing_age = df['age'].duplicated().sum()
duplicate_score = df['score'].duplicated().sum()

D. missing_age = df['age'].duplicated().any()
duplicate_score = df['score'].isnull().sum()

Solution

1. Step 1: Check missing values in 'age'

   df['age'].isnull().sum() counts missing values correctly.

   missing_age = df['age'].isnull().sum()
   duplicate_score = df['score'].duplicated().any()

   uses this.

2. Step 2: Check duplicates in 'score'

   df['score'].duplicated().any() returns True if any duplicates exist, which is a good validation check.

3. Step 3: Review other options

   missing_age = df.isnull().sum()['age']
   duplicate_score = df['score'].isnull().any()

   wrongly checks duplicates with isnull(),

   missing_age = df['age'].duplicated().sum()
   duplicate_score = df['score'].duplicated().sum()

   uses duplicated() on 'age' which is wrong,

   missing_age = df['age'].duplicated().any()
   duplicate_score = df['score'].isnull().sum()

   wrongly uses duplicated() for missing and isnull() for duplicates.

4. Final Answer:

   missing_age = df['age'].isnull().sum(); duplicate_score = df['score'].duplicated().any() -> Option B

5. Quick Check:

   Missing count + duplicate check = correct validation [OK]

Hint: Sum isnull() for missing; use duplicated().any() for duplicates [OK]

Common Mistakes:

• Using duplicated() to count missing values
• Using isnull() to check duplicates
• Mixing sum() and any() incorrectly