Data quality assertions in Apache Spark - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to check if the DataFrame has any null values.

Apache Spark

df.selectExpr('count(*) as total', 'count([1]) as non_null').show()

Drag options to blanks, or click blank then click option'

Anull

Bcolumn_name

C*

Dcount

Attempts:

3 left

💡 Hint

Common Mistakes

Using a specific column name without context causes errors.

Using 'null' as a string is invalid in selectExpr.

Using '*' counts all rows, so comparing total rows with non-null counts helps detect nulls.

2fill in blank

medium

Complete the code to assert that the column 'age' has no null values.

Apache Spark

assert df.filter(df.age.[1](None)).count() == 0, 'Null values found in age column'

Drag options to blanks, or click blank then click option'

AisNotNull

BisNull

Cisnan

DisEmpty

Attempts:

3 left

💡 Hint

Common Mistakes

Using isNotNull() filters non-null rows, which is opposite of the goal.

Using isnan() is for NaN values, not nulls.

Filtering rows where 'age' is null and asserting count is zero ensures no nulls.

3fill in blank

hard

Fix the error in the code to assert that all values in 'salary' are positive.

Apache Spark

assert df.filter(df.salary [1] 0).count() == 0, 'Negative or zero salary found'

Drag options to blanks, or click blank then click option'

A<=

B>=

C<

D>

Attempts:

3 left

💡 Hint

Common Mistakes

Using '>=' filters salaries greater or equal to zero, which is incorrect here.

Using '>' misses zero values.

Filtering salaries less than or equal to zero and asserting count zero ensures all are positive.

4fill in blank

hard

Fill both blanks to create a dictionary of counts for each unique value in 'department' where count is greater than 5.

Apache Spark

dept_counts = {row['[1]']: row['[2]'] for row in df.groupBy('department').count().collect() if row['count'] > 5}

Drag options to blanks, or click blank then click option'

Adepartment

Bcount

Cdept

Dvalue

Attempts:

3 left

💡 Hint

Common Mistakes

Using incorrect keys like 'dept' or 'value' which don't exist in rows.

Confusing 'count' with other column names.

Use 'department' as key and 'count' as value to build the dictionary.

5fill in blank

hard

Fill all three blanks to create a filtered DataFrame with no nulls in 'email' and 'phone' columns and only rows where 'age' is greater than 18.

Apache Spark

filtered_df = df.filter(df.email.[1]() & df.phone.[2]() & (df.age [3] 18))

Drag options to blanks, or click blank then click option'

AisNotNull

B>

CisNull

D<

Attempts:

3 left

💡 Hint

Common Mistakes

Using isNull() instead of isNotNull() includes nulls.

Using '<' instead of '>' filters wrong age range.

Use isNotNull() to filter non-null emails and phones, and '>' to filter age > 18.

Practice

1. What is the main purpose of data quality assertions in Apache Spark?

easy

A. To create new columns based on calculations

B. To speed up data processing by caching

C. To check if data meets expected rules and find errors

D. To visualize data trends over time

Solution

1. Step 1: Understand data quality assertions

   Data quality assertions are used to verify if data follows expected rules and to detect errors like missing or invalid values.

2. Step 2: Compare with other options

   Options B, C, and D describe other Spark tasks unrelated to data quality assertions.

3. Final Answer:

   To check if data meets expected rules and find errors -> Option C

4. Quick Check:

   Data quality assertions = check rules and errors [OK]

Hint: Assertions check data rules and errors, not processing speed [OK]

Common Mistakes:

• Confusing assertions with performance optimizations
• Thinking assertions create new data columns
• Mixing assertions with visualization tasks

2. Which of the following is the correct way to assert that a Spark DataFrame column age has no null values?

easy

A. df.filter(df.age.isNull()).count() == 0

B. df.filter(df.age.isNotNull()).count() == 0

C. df.filter(df.age == null).count() == 0

D. df.filter(df.age != null).count() == 0

Solution

1. Step 1: Understand null checking in Spark

   To check for nulls, use isNull(). Counting rows where age is null should be zero for no nulls.

2. Step 2: Analyze each option

   df.filter(df.age.isNull()).count() == 0 correctly checks if count of null age is zero. df.filter(df.age.isNotNull()).count() == 0 checks for non-null count equals zero, which is wrong. Options C and D use null with == or != which is invalid in Spark.

3. Final Answer:

   df.filter(df.age.isNull()).count() == 0 -> Option A

4. Quick Check:

   Use isNull() to find nulls, count should be zero [OK]

Hint: Use isNull() to find nulls, count zero means no nulls [OK]

Common Mistakes:

• Using == null instead of isNull()
• Checking isNotNull() count equals zero
• Confusing null checks with non-null checks

3. Given the following Spark code, what will be the output?

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
data = [(1, 'Alice', 25), (2, 'Bob', None), (3, 'Charlie', 30)]
df = spark.createDataFrame(data, ['id', 'name', 'age'])
null_count = df.filter(df.age.isNull()).count()
print(null_count)

medium

A. 0

B. 1

C. 2

D. 3

Solution

1. Step 1: Identify null values in the data

   The data has one row where age is None (null): the second row with Bob.

2. Step 2: Count rows with null age

   The filter df.age.isNull() selects rows with null age, so count will be 1.

3. Final Answer:

   1 -> Option B

4. Quick Check:

   One null age value = 1 [OK]

Hint: Count rows with isNull() to find null entries [OK]

Common Mistakes:

• Counting non-null rows instead of null rows
• Misreading None as a valid age
• Assuming no nulls without checking

4. You wrote this assertion to check if all values in column salary are positive:

assert df.filter(df.salary <= 0).count() == 0

But it passes even when there are null values in the salary column. What is the likely cause?

medium

A. The column salary contains null values, which are skipped by the comparison

B. The filter condition uses <= which is invalid in Spark

C. The count() method is not supported on DataFrames

D. The assert statement syntax is incorrect in Python

Solution

1. Step 1: Understand filter behavior with nulls

   In Spark, null <= 0 evaluates to null, which is falsy in filter(), so null rows are skipped and not counted, causing the assertion to pass incorrectly when nulls exist.

2. Step 2: Eliminate other options

   A: <= is valid in Spark filters. C: count() is supported on DataFrames. D: assert syntax is correct. B identifies the null-skipping issue.

3. Final Answer:

   The column salary contains null values, which are skipped by the comparison -> Option A

4. Quick Check:

   Nulls skipped in comparison filters [OK]

Hint: Nulls yield null in comparisons and are skipped by filter [OK]

Common Mistakes:

• Not realizing nulls are skipped in comparison filters
• Thinking comparisons catch nulls as invalid
• Blaming syntax instead of null handling

5. You want to assert that a DataFrame df has no duplicate user_id values. Which Spark code correctly performs this data quality assertion?

hard

A. assert df.dropDuplicates().count() == df.count()

B. assert df.select('user_id').count() == df.count()

C. assert df.filter(df.user_id.isNull()).count() == 0

D. assert df.groupBy('user_id').count().filter('count > 1').count() == 0

Solution

1. Step 1: Understand duplicate detection logic

   Duplicates mean some user_id appear more than once. Grouping by user_id and counting, then filtering counts > 1 finds duplicates.

2. Step 2: Analyze each option

   A: df.dropDuplicates().count() == df.count() checks for completely identical rows across all columns, missing duplicates in user_id with differing other columns. B: df.select('user_id').count() == df.count() always true. C: Checks only nulls. D: Correctly finds groups with count > 1.

3. Final Answer:

   assert df.groupBy('user_id').count().filter('count > 1').count() == 0 -> Option D

4. Quick Check:

   Group and filter counts > 1 to find duplicates [OK]

Hint: Group by column, filter count > 1 to detect duplicates [OK]

Common Mistakes:

• Using dropDuplicates() without column subset
• Using count() without distinct or groupBy
• Confusing null checks with duplicate checks