Bird
Raised Fist0
Apache Sparkdata~30 mins

Data quality assertions in Apache Spark - Mini Project: Build & Apply

Choose your learning style10 modes available
LearnWhyDeepVisualTryChallengeProjectRecallTime
Data Quality Assertions with Apache Spark
📖 Scenario: You work as a data analyst at a retail company. You receive daily sales data and want to make sure the data is clean and reliable before using it for reports.
🎯 Goal: Build a simple Apache Spark program that loads sales data, sets up quality checks (assertions) on the data, and prints whether the data passes these checks.
📋 What You'll Learn
Create a Spark DataFrame with given sales data
Define a threshold for minimum sales amount
Write assertions to check data quality conditions
Print the results of the data quality checks
💡 Why This Matters
🌍 Real World
Data quality checks are essential in real-world data pipelines to ensure reports and decisions are based on accurate data.
💼 Career
Data analysts and data engineers often write data quality assertions to catch errors early and maintain trust in data systems.
Progress0 / 4 steps
1
Create the sales data DataFrame
Create a Spark DataFrame called sales_df with these exact rows: ("2024-06-01", "StoreA", 150), ("2024-06-01", "StoreB", 80), ("2024-06-02", "StoreA", 200), ("2024-06-02", "StoreB", 50). Use columns named date, store, and sales.
Apache Spark
Hint

Use spark.createDataFrame() with a list of tuples and a list of column names.

2
Set the minimum sales threshold
Create a variable called min_sales and set it to 100. This will be the minimum acceptable sales amount for quality checks.
Apache Spark
Hint

Just create a variable named min_sales and assign the value 100.

3
Write data quality assertions
Create a variable called low_sales_count that counts rows in sales_df where sales is less than min_sales. Use the filter method and count().
Apache Spark
Hint

Use sales_df.filter(sales_df.sales < min_sales).count() to count low sales rows.

4
Print the data quality check result
Print the message "Number of low sales records:" followed by the value of low_sales_count.
Apache Spark
Hint

Use print("Number of low sales records:", low_sales_count) to show the result.

Practice

(1/5)
1. What is the main purpose of data quality assertions in Apache Spark?
easy
A. To create new columns based on calculations
B. To speed up data processing by caching
C. To check if data meets expected rules and find errors
D. To visualize data trends over time

Solution

  1. Step 1: Understand data quality assertions

    Data quality assertions are used to verify if data follows expected rules and to detect errors like missing or invalid values.

  2. Step 2: Compare with other options

    Options B, C, and D describe other Spark tasks unrelated to data quality assertions.

  3. Final Answer:

    To check if data meets expected rules and find errors -> Option C

  4. Quick Check:

    Data quality assertions = check rules and errors [OK]
Hint: Assertions check data rules and errors, not processing speed [OK]
Common Mistakes:
  • Confusing assertions with performance optimizations
  • Thinking assertions create new data columns
  • Mixing assertions with visualization tasks
2. Which of the following is the correct way to assert that a Spark DataFrame column age has no null values?
easy
A. df.filter(df.age.isNull()).count() == 0
B. df.filter(df.age.isNotNull()).count() == 0
C. df.filter(df.age == null).count() == 0
D. df.filter(df.age != null).count() == 0

Solution

  1. Step 1: Understand null checking in Spark

    To check for nulls, use isNull(). Counting rows where age is null should be zero for no nulls.

  2. Step 2: Analyze each option

    df.filter(df.age.isNull()).count() == 0 correctly checks if count of null age is zero. df.filter(df.age.isNotNull()).count() == 0 checks for non-null count equals zero, which is wrong. Options C and D use null with == or != which is invalid in Spark.

  3. Final Answer:

    df.filter(df.age.isNull()).count() == 0 -> Option A

  4. Quick Check:

    Use isNull() to find nulls, count should be zero [OK]
Hint: Use isNull() to find nulls, count zero means no nulls [OK]
Common Mistakes:
  • Using == null instead of isNull()
  • Checking isNotNull() count equals zero
  • Confusing null checks with non-null checks
3. Given the following Spark code, what will be the output? 
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
data = [(1, 'Alice', 25), (2, 'Bob', None), (3, 'Charlie', 30)]
df = spark.createDataFrame(data, ['id', 'name', 'age'])
null_count = df.filter(df.age.isNull()).count()
print(null_count)
medium
A. 0
B. 1
C. 2
D. 3

Solution

  1. Step 1: Identify null values in the data

    The data has one row where age is None (null): the second row with Bob.

  2. Step 2: Count rows with null age

    The filter df.age.isNull() selects rows with null age, so count will be 1.

  3. Final Answer:

    1 -> Option B

  4. Quick Check:

    One null age value = 1 [OK]
Hint: Count rows with isNull() to find null entries [OK]
Common Mistakes:
  • Counting non-null rows instead of null rows
  • Misreading None as a valid age
  • Assuming no nulls without checking
4. You wrote this assertion to check if all values in column salary are positive: 
assert df.filter(df.salary <= 0).count() == 0
But it passes even when there are null values in the salary column. What is the likely cause?
medium
A. The column salary contains null values, which are skipped by the comparison
B. The filter condition uses <= which is invalid in Spark
C. The count() method is not supported on DataFrames
D. The assert statement syntax is incorrect in Python

Solution

  1. Step 1: Understand filter behavior with nulls

    In Spark, null <= 0 evaluates to null, which is falsy in filter(), so null rows are skipped and not counted, causing the assertion to pass incorrectly when nulls exist.

  2. Step 2: Eliminate other options

    A: <= is valid in Spark filters. C: count() is supported on DataFrames. D: assert syntax is correct. B identifies the null-skipping issue.

  3. Final Answer:

    The column salary contains null values, which are skipped by the comparison -> Option A

  4. Quick Check:

    Nulls skipped in comparison filters [OK]
Hint: Nulls yield null in comparisons and are skipped by filter [OK]
Common Mistakes:
  • Not realizing nulls are skipped in comparison filters
  • Thinking comparisons catch nulls as invalid
  • Blaming syntax instead of null handling
5. You want to assert that a DataFrame df has no duplicate user_id values. Which Spark code correctly performs this data quality assertion?
hard
A. assert df.dropDuplicates().count() == df.count()
B. assert df.select('user_id').count() == df.count()
C. assert df.filter(df.user_id.isNull()).count() == 0
D. assert df.groupBy('user_id').count().filter('count > 1').count() == 0

Solution

  1. Step 1: Understand duplicate detection logic

    Duplicates mean some user_id appear more than once. Grouping by user_id and counting, then filtering counts > 1 finds duplicates.

  2. Step 2: Analyze each option

    A: df.dropDuplicates().count() == df.count() checks for completely identical rows across all columns, missing duplicates in user_id with differing other columns. B: df.select('user_id').count() == df.count() always true. C: Checks only nulls. D: Correctly finds groups with count > 1.

  3. Final Answer:

    assert df.groupBy('user_id').count().filter('count > 1').count() == 0 -> Option D

  4. Quick Check:

    Group and filter counts > 1 to find duplicates [OK]
Hint: Group by column, filter count > 1 to detect duplicates [OK]
Common Mistakes:
  • Using dropDuplicates() without column subset
  • Using count() without distinct or groupBy
  • Confusing null checks with duplicate checks
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard