Data quality assertions in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a data quality assertion in Apache Spark?

A data quality assertion is a check or condition applied to data to ensure it meets expected standards, such as no missing values or valid ranges, helping to catch errors early.

beginner

How can you check for null values in a Spark DataFrame column using assertions?

You can use the filter or where method to find rows with nulls and then assert that the count is zero, ensuring no nulls exist in that column.

beginner

Why are data quality assertions important before running data analysis?

They help catch bad data early, preventing wrong conclusions and saving time by ensuring the data is clean and reliable before analysis.

intermediate

Example of a simple data quality assertion in Spark to check if all ages are positive.

Use assert(df.filter(df.age <= 0).count() == 0, 'Age must be positive') to ensure no age is zero or negative.

beginner

What happens if a data quality assertion fails in Spark?

The assertion throws an error and stops the program, signaling that the data does not meet the expected quality rules.

What does a data quality assertion typically check in a Spark DataFrame?

A If the data is sorted alphabetically

B If data meets expected conditions like no nulls or valid ranges

C If the DataFrame has more than 100 rows

D If the Spark cluster is running

Data quality assertions check that data meets expected conditions such as no null values or valid ranges.

How do you assert that a column 'salary' has no negative values in Spark?

A assert(df.filter(df.salary < 0).count() == 0, 'No negative salaries')

B assert(df.count() > 0)

C assert(df.salary == 0)

D assert(df.salary.isNull())

Filtering rows where salary is negative and asserting the count is zero ensures no negative salaries.

What is the result if a data quality assertion fails in Spark?

A The Spark session restarts

B The program ignores the failure and continues

C The data is automatically fixed

D The program throws an error and stops

When an assertion fails, Spark throws an error and stops execution to alert the user.

Which Spark method helps to find rows violating a data quality rule?

A filter or where

B groupBy

C join

D cache

The filter or where method selects rows that violate conditions for assertions.

Why should data quality assertions be run early in a data pipeline?

A To speed up Spark cluster startup

B To reduce data size

C To catch bad data before analysis

D To create visualizations

Running assertions early helps catch bad data and prevents errors downstream.

Explain what data quality assertions are and why they matter in Apache Spark data processing.

Think about how checking data before analysis helps avoid mistakes.

Describe how you would implement a data quality assertion to check for null values in a Spark DataFrame column.

Focus on filtering rows with nulls and asserting zero count.

Practice

1. What is the main purpose of data quality assertions in Apache Spark?

easy

A. To create new columns based on calculations

B. To speed up data processing by caching

C. To check if data meets expected rules and find errors

D. To visualize data trends over time

Solution

1. Step 1: Understand data quality assertions

   Data quality assertions are used to verify if data follows expected rules and to detect errors like missing or invalid values.

2. Step 2: Compare with other options

   Options B, C, and D describe other Spark tasks unrelated to data quality assertions.

3. Final Answer:

   To check if data meets expected rules and find errors -> Option C

4. Quick Check:

   Data quality assertions = check rules and errors [OK]

Hint: Assertions check data rules and errors, not processing speed [OK]

Common Mistakes:

• Confusing assertions with performance optimizations
• Thinking assertions create new data columns
• Mixing assertions with visualization tasks

2. Which of the following is the correct way to assert that a Spark DataFrame column age has no null values?

easy

A. df.filter(df.age.isNull()).count() == 0

B. df.filter(df.age.isNotNull()).count() == 0

C. df.filter(df.age == null).count() == 0

D. df.filter(df.age != null).count() == 0

Solution

1. Step 1: Understand null checking in Spark

   To check for nulls, use isNull(). Counting rows where age is null should be zero for no nulls.

2. Step 2: Analyze each option

   df.filter(df.age.isNull()).count() == 0 correctly checks if count of null age is zero. df.filter(df.age.isNotNull()).count() == 0 checks for non-null count equals zero, which is wrong. Options C and D use null with == or != which is invalid in Spark.

3. Final Answer:

   df.filter(df.age.isNull()).count() == 0 -> Option A

4. Quick Check:

   Use isNull() to find nulls, count should be zero [OK]

Hint: Use isNull() to find nulls, count zero means no nulls [OK]

Common Mistakes:

• Using == null instead of isNull()
• Checking isNotNull() count equals zero
• Confusing null checks with non-null checks

3. Given the following Spark code, what will be the output?

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
data = [(1, 'Alice', 25), (2, 'Bob', None), (3, 'Charlie', 30)]
df = spark.createDataFrame(data, ['id', 'name', 'age'])
null_count = df.filter(df.age.isNull()).count()
print(null_count)

medium

A. 0

B. 1

C. 2

D. 3

Solution

1. Step 1: Identify null values in the data

   The data has one row where age is None (null): the second row with Bob.

2. Step 2: Count rows with null age

   The filter df.age.isNull() selects rows with null age, so count will be 1.

3. Final Answer:

   1 -> Option B

4. Quick Check:

   One null age value = 1 [OK]

Hint: Count rows with isNull() to find null entries [OK]

Common Mistakes:

• Counting non-null rows instead of null rows
• Misreading None as a valid age
• Assuming no nulls without checking

4. You wrote this assertion to check if all values in column salary are positive:

assert df.filter(df.salary <= 0).count() == 0

But it passes even when there are null values in the salary column. What is the likely cause?

medium

A. The column salary contains null values, which are skipped by the comparison

B. The filter condition uses <= which is invalid in Spark

C. The count() method is not supported on DataFrames

D. The assert statement syntax is incorrect in Python

Solution

1. Step 1: Understand filter behavior with nulls

   In Spark, null <= 0 evaluates to null, which is falsy in filter(), so null rows are skipped and not counted, causing the assertion to pass incorrectly when nulls exist.

2. Step 2: Eliminate other options

   A: <= is valid in Spark filters. C: count() is supported on DataFrames. D: assert syntax is correct. B identifies the null-skipping issue.

3. Final Answer:

   The column salary contains null values, which are skipped by the comparison -> Option A

4. Quick Check:

   Nulls skipped in comparison filters [OK]

Hint: Nulls yield null in comparisons and are skipped by filter [OK]

Common Mistakes:

• Not realizing nulls are skipped in comparison filters
• Thinking comparisons catch nulls as invalid
• Blaming syntax instead of null handling

5. You want to assert that a DataFrame df has no duplicate user_id values. Which Spark code correctly performs this data quality assertion?

hard

A. assert df.dropDuplicates().count() == df.count()

B. assert df.select('user_id').count() == df.count()

C. assert df.filter(df.user_id.isNull()).count() == 0

D. assert df.groupBy('user_id').count().filter('count > 1').count() == 0

Solution

1. Step 1: Understand duplicate detection logic

   Duplicates mean some user_id appear more than once. Grouping by user_id and counting, then filtering counts > 1 finds duplicates.

2. Step 2: Analyze each option

   A: df.dropDuplicates().count() == df.count() checks for completely identical rows across all columns, missing duplicates in user_id with differing other columns. B: df.select('user_id').count() == df.count() always true. C: Checks only nulls. D: Correctly finds groups with count > 1.

3. Final Answer:

   assert df.groupBy('user_id').count().filter('count > 1').count() == 0 -> Option D

4. Quick Check:

   Group and filter counts > 1 to find duplicates [OK]

Hint: Group by column, filter count > 1 to detect duplicates [OK]

Common Mistakes:

• Using dropDuplicates() without column subset
• Using count() without distinct or groupBy
• Confusing null checks with duplicate checks