Why data quality prevents downstream failures in Apache Spark - Challenge Your Understanding

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of Spark DataFrame after filtering nulls

Given the Spark DataFrame code below, what will be the output after filtering out rows with null values in the 'age' column?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName('DataQuality').getOrCreate()
data = [(1, 'Alice', 25), (2, 'Bob', None), (3, 'Charlie', 30), (4, 'David', None)]
df = spark.createDataFrame(data, ['id', 'name', 'age'])
df_filtered = df.filter(col('age').isNotNull())
df_filtered.show()

A

+---+-------+----+
| id|   name| age|
+---+-------+----+
|  2|    Bob|null|
|  4|  David|null|
+---+-------+----+

B

Empty DataFrame
Columns: [id, name, age]
[]

C

+---+-------+---+
| id|   name|age|
+---+-------+---+
|  1|  Alice| 25|
|  2|    Bob|null|
|  3|Charlie| 30|
|  4|  David|null|
+---+-------+---+

D

+---+-------+---+
| id|   name|age|
+---+-------+---+
|  1|  Alice| 25|
|  3|Charlie| 30|
+---+-------+---+

💡 Hint

Filtering removes rows where 'age' is null.

The filter keeps only rows where 'age' is not null, so rows with Bob and David are removed.

❓ data_output

intermediate

Count of distinct values after cleaning data

After removing duplicate rows and rows with null 'email' values from a Spark DataFrame, what is the count of distinct emails?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName('DataQuality').getOrCreate()
data = [(1, '[e-mail]'), (2, '[e-mail]'), (3, None), (4, '[e-mail]'), (5, None)]
df = spark.createDataFrame(data, ['id', 'email'])
df_clean = df.dropDuplicates(['email']).filter(col('email').isNotNull())
count = df_clean.select('email').distinct().count()
print(count)

A 2

B 5

C 4

D 3

💡 Hint

Duplicates and nulls are removed before counting distinct emails.

Only '[e-mail]' and '[e-mail]' remain after cleaning, so count is 2.

🔧 Debug

advanced

Identify the error in Spark data validation code

What error will this Spark code raise when checking for negative values in the 'salary' column?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName('DataQuality').getOrCreate()
data = [(1, 5000), (2, -1000), (3, 7000)]
df = spark.createDataFrame(data, ['id', 'salary'])
invalid_rows = df.filter(col('salary') < 0).collect()
print(invalid_rows[0]['salary'])

A No error, prints -1000

B TypeError

C IndexError

D KeyError

💡 Hint

Check if filter returns rows and how to access them.

The filter returns rows with salary < 0, collect() returns a list, accessing first element and 'salary' key works fine.

❓ visualization

advanced

Interpreting Spark DataFrame summary statistics

Given the summary statistics output below from a Spark DataFrame's 'score' column, what is the median score?

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('DataQuality').getOrCreate()
data = [(1, 50), (2, 80), (3, 90), (4, 70), (5, 60)]
df = spark.createDataFrame(data, ['id', 'score'])
summary = df.describe('score')
summary.show()

A 75

B Cannot determine from describe() output

C 80

D 70

💡 Hint

Spark's describe() does not provide median.

The describe() method shows count, mean, stddev, min, max but not median.

🚀 Application

expert

Choosing the best approach to prevent downstream failures

You have a Spark pipeline that fails downstream due to unexpected nulls in a critical column. Which approach best prevents these failures?

A Replace nulls with zeros without checking if zeros are valid

B Ignore nulls and handle errors only when failures occur downstream

C Add a filter step early to remove rows with nulls in the critical column

D Skip data quality checks to improve pipeline speed

💡 Hint

Prevent problems early by cleaning data before processing.

Filtering out nulls early avoids errors later. Ignoring or skipping checks causes failures. Blind replacement may cause wrong data.