Data validation checks in Pandas - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of data validation checks in pandas?

Data validation checks help ensure that the data is clean, accurate, and meets expected rules before analysis. They catch errors or inconsistencies early.

beginner

How can you check for missing values in a pandas DataFrame?

Use the isnull() or isna() methods to find missing values. For example, df.isnull().sum() shows the count of missing values per column.

intermediate

What pandas method helps to check if all values in a column meet a condition?

The all() method can be used after a condition. For example, (df['age'] > 0).all() checks if all ages are positive.

beginner

How do you find duplicate rows in a pandas DataFrame?

Use df.duplicated() to get a boolean Series marking duplicates. To see duplicates, use df[df.duplicated()].

intermediate

Explain how to validate data types of columns in pandas.

Use df.dtypes to see data types of each column. You can check if a column has the expected type, e.g., df['price'].dtype == 'float64'.

Which pandas method helps identify missing values in a DataFrame?

A isnull()

B drop_duplicates()

C astype()

D head()

The isnull() method returns a DataFrame of booleans indicating missing values.

How do you check if all values in a column 'age' are greater than zero?

A (df['age'] > 0).all()

B df['age'] > 0

C df['age'].sum() > 0

D df['age'].isnull()

Using (df['age'] > 0).all() returns True only if every value in 'age' is greater than zero.

What does df.duplicated() return?

A Rows with missing values

B Boolean Series marking duplicate rows

C Count of unique rows

D Data types of columns

df.duplicated() returns a boolean Series where True marks duplicate rows.

Which method shows the data types of each column in a DataFrame?

A df.info()

B df.describe()

C df.dtypes

D df.head()

df.dtypes returns a Series with the data type of each column.

To count missing values per column, which code is correct?

A df.duplicated()

B df.dropna()

C df.fillna(0)

D df.isnull().sum()

df.isnull().sum() counts missing values in each column.

Describe how you would perform basic data validation checks on a new dataset using pandas.

Think about common data issues like missing data, duplicates, and wrong types.

Explain why data validation checks are important before analyzing data.

Consider what happens if you analyze bad data.

Practice

1. What does the pandas function isnull() check in a DataFrame?

easy

A. It checks for missing or null values in the data.

B. It checks for duplicate rows in the DataFrame.

C. It returns the data types of each column.

D. It sorts the DataFrame by a column.

Solution

1. Step 1: Understand the purpose of isnull()

   The isnull() function in pandas returns a DataFrame of the same shape with True where values are missing (NaN) and False otherwise.

2. Step 2: Compare with other functions

   Functions like duplicated() check duplicates, and dtypes shows data types, so they do not check for missing values.

3. Final Answer:

   It checks for missing or null values in the data. -> Option A

4. Quick Check:

   Missing data = isnull() [OK]

Hint: Missing values? Use isnull() to find them fast [OK]

Common Mistakes:

• Confusing isnull() with duplicated()
• Thinking isnull() returns data types
• Assuming isnull() sorts data

2. Which of the following is the correct syntax to check for duplicate rows in a pandas DataFrame named df?

easy

A. df.isnull()

B. df.check_duplicates()

C. df.duplicated()

D. df.find_duplicates()

Solution

1. Step 1: Recall pandas method for duplicates

   The correct pandas method to find duplicate rows is duplicated(), called as df.duplicated().

2. Step 2: Verify other options

   isnull() checks missing data, and check_duplicates() and find_duplicates() are not valid pandas methods.

3. Final Answer:

   df.duplicated() -> Option C

4. Quick Check:

   Duplicates = duplicated() [OK]

Hint: Duplicates? Use df.duplicated() to spot them [OK]

Common Mistakes:

• Using non-existent methods like check_duplicates()
• Mixing isnull() with duplicate checks
• Forgetting parentheses after duplicated

3. Given the DataFrame df below, what will df.isnull().sum() output?

import pandas as pd
data = {'A': [1, 2, None], 'B': [None, 2, 3], 'C': [1, 2, 3]}
df = pd.DataFrame(data)

medium

A. {'A': 1, 'B': 2, 'C': 1}

B. {'A': 0, 'B': 0, 'C': 0}

C. {'A': 2, 'B': 1, 'C': 0}

D. {'A': 1, 'B': 1, 'C': 0}

Solution

1. Step 1: Identify missing values in each column

   Column 'A' has one None (missing), 'B' has one None, and 'C' has no missing values.

2. Step 2: Understand what isnull().sum() does

   isnull() marks missing as True, and sum() counts True values per column.

3. Final Answer:

   {'A': 1, 'B': 1, 'C': 0} -> Option D

4. Quick Check:

   Count missing per column = isnull().sum() [OK]

Hint: Sum isnull() to count missing per column [OK]

Common Mistakes:

• Counting total missing instead of per column
• Confusing None with zero
• Misreading DataFrame values

4. The following code is intended to find duplicate rows in df. What is the error?

duplicates = df.duplicate()

medium

A. The method name should be duplicated(), not duplicate().

B. The DataFrame df is not defined.

C. The method duplicated() requires an argument.

D. The code should use df.isnull() to find duplicates.

Solution

1. Step 1: Check method name correctness

   The correct pandas method to find duplicates is duplicated(), not duplicate().

2. Step 2: Verify other options

   df is assumed defined, duplicated() does not require arguments, and isnull() checks missing data, not duplicates.

3. Final Answer:

   The method name should be duplicated(), not duplicate(). -> Option A

4. Quick Check:

   Correct method name = duplicated() [OK]

Hint: Remember: duplicated() has a 'd' at the end [OK]

Common Mistakes:

• Misspelling duplicated() as duplicate()
• Confusing duplicated() with isnull()
• Thinking duplicated() needs arguments

5. You have a DataFrame df with columns ['id', 'age', 'score']. You want to validate that age has no missing values and score has no duplicates. Which code correctly performs both checks?

hard

A. missing_age = df.isnull().sum()['age']
duplicate_score = df['score'].isnull().any()

B. missing_age = df['age'].isnull().sum()
duplicate_score = df['score'].duplicated().any()

C. missing_age = df['age'].duplicated().sum()
duplicate_score = df['score'].duplicated().sum()

D. missing_age = df['age'].duplicated().any()
duplicate_score = df['score'].isnull().sum()

Solution

1. Step 1: Check missing values in 'age'

   df['age'].isnull().sum() counts missing values correctly.

   missing_age = df['age'].isnull().sum()
   duplicate_score = df['score'].duplicated().any()

   uses this.

2. Step 2: Check duplicates in 'score'

   df['score'].duplicated().any() returns True if any duplicates exist, which is a good validation check.

3. Step 3: Review other options

   missing_age = df.isnull().sum()['age']
   duplicate_score = df['score'].isnull().any()

   wrongly checks duplicates with isnull(),

   missing_age = df['age'].duplicated().sum()
   duplicate_score = df['score'].duplicated().sum()

   uses duplicated() on 'age' which is wrong,

   missing_age = df['age'].duplicated().any()
   duplicate_score = df['score'].isnull().sum()

   wrongly uses duplicated() for missing and isnull() for duplicates.

4. Final Answer:

   missing_age = df['age'].isnull().sum(); duplicate_score = df['score'].duplicated().any() -> Option B

5. Quick Check:

   Missing count + duplicate check = correct validation [OK]

Hint: Sum isnull() for missing; use duplicated().any() for duplicates [OK]

Common Mistakes:

• Using duplicated() to count missing values
• Using isnull() to check duplicates
• Mixing sum() and any() incorrectly