Why data quality prevents downstream failures in Apache Spark - Challenge Your Understanding

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of Spark DataFrame after filtering nulls

Given the Spark DataFrame code below, what will be the output after filtering out rows with null values in the 'age' column?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName('DataQuality').getOrCreate()
data = [(1, 'Alice', 25), (2, 'Bob', None), (3, 'Charlie', 30), (4, 'David', None)]
df = spark.createDataFrame(data, ['id', 'name', 'age'])
df_filtered = df.filter(col('age').isNotNull())
df_filtered.show()

A

+---+-------+----+
| id|   name| age|
+---+-------+----+
|  2|    Bob|null|
|  4|  David|null|
+---+-------+----+

B

Empty DataFrame
Columns: [id, name, age]
[]

C

+---+-------+---+
| id|   name|age|
+---+-------+---+
|  1|  Alice| 25|
|  2|    Bob|null|
|  3|Charlie| 30|
|  4|  David|null|
+---+-------+---+

D

+---+-------+---+
| id|   name|age|
+---+-------+---+
|  1|  Alice| 25|
|  3|Charlie| 30|
+---+-------+---+

💡 Hint

Filtering removes rows where 'age' is null.

The filter keeps only rows where 'age' is not null, so rows with Bob and David are removed.

❓ data_output

intermediate

Count of distinct values after cleaning data

After removing duplicate rows and rows with null 'email' values from a Spark DataFrame, what is the count of distinct emails?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName('DataQuality').getOrCreate()
data = [(1, '[e-mail]'), (2, '[e-mail]'), (3, None), (4, '[e-mail]'), (5, None)]
df = spark.createDataFrame(data, ['id', 'email'])
df_clean = df.dropDuplicates(['email']).filter(col('email').isNotNull())
count = df_clean.select('email').distinct().count()
print(count)

A 2

B 5

C 4

D 3

💡 Hint

Duplicates and nulls are removed before counting distinct emails.

Only '[e-mail]' and '[e-mail]' remain after cleaning, so count is 2.

🔧 Debug

advanced

Identify the error in Spark data validation code

What error will this Spark code raise when checking for negative values in the 'salary' column?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName('DataQuality').getOrCreate()
data = [(1, 5000), (2, -1000), (3, 7000)]
df = spark.createDataFrame(data, ['id', 'salary'])
invalid_rows = df.filter(col('salary') < 0).collect()
print(invalid_rows[0]['salary'])

A No error, prints -1000

B TypeError

C IndexError

D KeyError

💡 Hint

Check if filter returns rows and how to access them.

The filter returns rows with salary < 0, collect() returns a list, accessing first element and 'salary' key works fine.

❓ visualization

advanced

Interpreting Spark DataFrame summary statistics

Given the summary statistics output below from a Spark DataFrame's 'score' column, what is the median score?

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('DataQuality').getOrCreate()
data = [(1, 50), (2, 80), (3, 90), (4, 70), (5, 60)]
df = spark.createDataFrame(data, ['id', 'score'])
summary = df.describe('score')
summary.show()

A 75

B Cannot determine from describe() output

C 80

D 70

💡 Hint

Spark's describe() does not provide median.

The describe() method shows count, mean, stddev, min, max but not median.

🚀 Application

expert

Choosing the best approach to prevent downstream failures

You have a Spark pipeline that fails downstream due to unexpected nulls in a critical column. Which approach best prevents these failures?

A Replace nulls with zeros without checking if zeros are valid

B Ignore nulls and handle errors only when failures occur downstream

C Add a filter step early to remove rows with nulls in the critical column

D Skip data quality checks to improve pipeline speed

💡 Hint

Prevent problems early by cleaning data before processing.

Filtering out nulls early avoids errors later. Ignoring or skipping checks causes failures. Blind replacement may cause wrong data.

Practice

1. Why is maintaining good data quality important in Apache Spark pipelines?

easy

A. It prevents errors and failures in later processing steps

B. It makes the data files smaller in size

C. It speeds up the Spark installation process

D. It automatically creates visualizations

Solution

1. Step 1: Understand the role of data quality

   Good data quality means data is accurate, complete, and consistent, which helps avoid problems later.

2. Step 2: Connect data quality to pipeline stability

   If data has errors or missing values, Spark jobs can fail or produce wrong results downstream.

3. Final Answer:

   It prevents errors and failures in later processing steps -> Option A

4. Quick Check:

   Good data quality = fewer downstream errors [OK]

Hint: Good data quality stops errors later [OK]

Common Mistakes:

• Thinking data size reduction is main goal
• Confusing data quality with installation speed
• Assuming data quality creates visualizations

2. Which Spark DataFrame method is used to remove rows with missing values to improve data quality?

easy

A. fillna()

B. dropna()

C. select()

D. groupBy()

Solution

1. Step 1: Identify method to remove missing data

   dropna() removes rows with null or missing values from a DataFrame.

2. Step 2: Differentiate from other methods

   fillna() fills missing values but does not remove rows; select() chooses columns; groupBy() aggregates data.

3. Final Answer:

   dropna() -> Option B

4. Quick Check:

   dropna() removes missing rows [OK]

Hint: dropna() removes missing rows [OK]

Common Mistakes:

• Confusing fillna() with dropna()
• Using select() to remove rows
• Thinking groupBy() cleans missing data

3. Given this Spark code snippet:

df = spark.createDataFrame([(1, 'Alice'), (None, 'Bob'), (3, None)], ['id', 'name'])
df_clean = df.dropna()
df_clean.show()

What will be the output?

medium

A. +----+-----+
| id | name|
+----+-----+
|null| Bob |
+----+-----+

B. +----+-----+
| id | name|
+----+-----+
| 1 |Alice|
| 3 | null|
+----+-----+

C. +---+-----+
| id| name|
+---+-----+
| 1|Alice|
+---+-----+

D. Error: dropna() is not a valid method

Solution

1. Step 1: Understand dropna() effect on DataFrame

   dropna() removes any row with null in any column.

2. Step 2: Apply dropna() to given data

   Rows (None, 'Bob') and (3, None) have nulls, so only (1, 'Alice') remains.

3. Final Answer:

   +---+-----+ | id| name| +---+-----+ | 1|Alice| +---+-----+ -> Option C

4. Quick Check:

   dropna() removes rows with nulls [OK]

Hint: dropna() removes any row with null [OK]

Common Mistakes:

• Expecting rows with null to remain
• Confusing dropna() with fillna()
• Thinking dropna() causes error

4. You run this code but get an error:

df = spark.createDataFrame([(1, 'Alice'), (2, None)], ['id', 'name'])
df_clean = df.dropna(how='anyy')
df_clean.show()

What is the cause of the error?

medium

A. The 'how' parameter value 'anyy' is invalid

B. dropna() cannot be used on DataFrames

C. The DataFrame has no missing values

D. show() method is not available on DataFrames

Solution

1. Step 1: Check dropna() parameters

   dropna() accepts 'how' parameter with values 'any' or 'all' only.

2. Step 2: Identify invalid parameter usage

   Using 'anyy' causes a runtime error because it's not a valid option.

3. Final Answer:

   The 'how' parameter value 'anyy' is invalid -> Option A

4. Quick Check:

   Invalid parameter value causes error [OK]

Hint: Check parameter spelling carefully [OK]

Common Mistakes:

• Thinking dropna() is invalid on DataFrames
• Ignoring typo in 'how' parameter
• Assuming show() causes error

5. You have a Spark DataFrame with many missing and invalid values. Which approach best prevents downstream failures in your data pipeline?

hard

A. Convert all data to strings to avoid type errors

B. Ignore missing data and handle errors only when they occur downstream

C. Run transformations first, then clean data at the end

D. Remove or fix bad data early using dropna() and fillna() before transformations

Solution

1. Step 1: Understand impact of bad data early

   Bad data causes errors or wrong results if not handled before transformations.

2. Step 2: Choose best practice for data quality

   Removing or fixing missing/invalid data early with dropna() and fillna() prevents failures downstream.

3. Step 3: Evaluate other options

   Ignoring errors delays failure; cleaning last risks wasted work; converting all to strings loses data meaning.

4. Final Answer:

   Remove or fix bad data early using dropna() and fillna() before transformations -> Option D

5. Quick Check:

   Early cleaning prevents downstream failures [OK]

Hint: Clean data early to avoid later errors [OK]

Common Mistakes:

• Delaying cleaning until after transformations
• Ignoring missing data causes hidden errors
• Converting all data to strings loses type info