Data quality assertions in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a data quality assertion in Apache Spark?

A data quality assertion is a check or condition applied to data to ensure it meets expected standards, such as no missing values or valid ranges, helping to catch errors early.

beginner

How can you check for null values in a Spark DataFrame column using assertions?

You can use the filter or where method to find rows with nulls and then assert that the count is zero, ensuring no nulls exist in that column.

beginner

Why are data quality assertions important before running data analysis?

They help catch bad data early, preventing wrong conclusions and saving time by ensuring the data is clean and reliable before analysis.

intermediate

Example of a simple data quality assertion in Spark to check if all ages are positive.

Use assert(df.filter(df.age <= 0).count() == 0, 'Age must be positive') to ensure no age is zero or negative.

beginner

What happens if a data quality assertion fails in Spark?

The assertion throws an error and stops the program, signaling that the data does not meet the expected quality rules.

What does a data quality assertion typically check in a Spark DataFrame?

A If the data is sorted alphabetically

B If data meets expected conditions like no nulls or valid ranges

C If the DataFrame has more than 100 rows

D If the Spark cluster is running

Data quality assertions check that data meets expected conditions such as no null values or valid ranges.

How do you assert that a column 'salary' has no negative values in Spark?

A assert(df.filter(df.salary < 0).count() == 0, 'No negative salaries')

B assert(df.count() > 0)

C assert(df.salary == 0)

D assert(df.salary.isNull())

Filtering rows where salary is negative and asserting the count is zero ensures no negative salaries.

What is the result if a data quality assertion fails in Spark?

A The Spark session restarts

B The program ignores the failure and continues

C The data is automatically fixed

D The program throws an error and stops

When an assertion fails, Spark throws an error and stops execution to alert the user.

Which Spark method helps to find rows violating a data quality rule?

A filter or where

B groupBy

C join

D cache

The filter or where method selects rows that violate conditions for assertions.

Why should data quality assertions be run early in a data pipeline?

A To speed up Spark cluster startup

B To reduce data size

C To catch bad data before analysis

D To create visualizations

Running assertions early helps catch bad data and prevents errors downstream.

Explain what data quality assertions are and why they matter in Apache Spark data processing.

Think about how checking data before analysis helps avoid mistakes.

Describe how you would implement a data quality assertion to check for null values in a Spark DataFrame column.

Focus on filtering rows with nulls and asserting zero count.