Null and duplicate detection in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of detecting null values in a dataset?

Detecting null values helps identify missing or incomplete data, which can affect analysis and model accuracy. Handling nulls properly ensures cleaner, more reliable results.

beginner

How do you check for null values in a Spark DataFrame?

You can use the filter or where method with isNull() on a column, for example: df.filter(df['column'].isNull()).show().

intermediate

What is the difference between dropDuplicates() and distinct() in Spark?

dropDuplicates() removes duplicate rows based on specified columns, while distinct() removes duplicate rows considering all columns.

beginner

Why is it important to detect duplicate rows in data?

Duplicate rows can bias analysis and models by over-representing some data points. Removing duplicates ensures fair and accurate insights.

intermediate

How can you count the number of nulls in each column of a Spark DataFrame?

Use aggregation with sum and when functions: df.select([F.sum(F.when(F.col(c).isNull(),1).otherwise(0)).alias(c) for c in df.columns]).show().

Which Spark function helps to find rows with null values in a column?

A isNull()

B isEmpty()

C isNaN()

D isDuplicate()

The isNull() function checks if a column value is null.

What does dropDuplicates(['col1', 'col2']) do in Spark?

A Removes duplicate rows based on col1 and col2 values

B Removes all duplicates in the DataFrame

C Removes rows with nulls in col1 or col2

D Removes rows where col1 equals col2

It removes rows that have the same values in both col1 and col2.

Which method removes duplicate rows considering all columns in a Spark DataFrame?

A dropDuplicates()

B distinct()

C dropNulls()

D filterDuplicates()

distinct() removes duplicate rows based on all columns.

Why should you handle null values before analysis?

A Nulls are always replaced with zeros automatically

B Nulls improve model accuracy

C Nulls increase dataset size

D Nulls can cause errors or bias in results

Null values can cause errors or bias if not handled properly.

How can you count nulls in each column of a Spark DataFrame?

A Using <code>count()</code>

B Using <code>dropDuplicates()</code>

C Using <code>sum(when(col.isNull(),1).otherwise(0))</code>

D Using <code>filter(col.isNotNull())</code>

This aggregation counts nulls by summing 1 for nulls and 0 otherwise.

Explain how to detect and handle null values in a Spark DataFrame.

Think about checking, counting, and then cleaning nulls.

Describe the difference between removing duplicates with dropDuplicates() and distinct() in Spark.

Focus on which columns each method considers.