Select, filter, and where operations in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does the select() operation do in Apache Spark DataFrame?

The select() operation chooses specific columns from a DataFrame, like picking certain ingredients from a recipe.

beginner

How do filter() and where() operations differ in Apache Spark?

They do the same thing: keep only rows that meet a condition. filter() and where() are just two names for the same operation.

beginner

Why use filter() or where() in data analysis?

To focus on rows that matter, like finding all customers from a certain city or sales above a number.

intermediate

Show a simple example of using select() and filter() together.

Example: df.select('name', 'age').filter(df.age > 30) picks only the 'name' and 'age' columns and keeps rows where age is over 30.

intermediate

Can you chain multiple filter() or where() conditions?

Yes! You can chain them or combine conditions with & (and), | (or) to filter rows with multiple rules.

What does df.select('col1', 'col2') do?

A Selects only columns 'col1' and 'col2' from the DataFrame

B Filters rows where 'col1' and 'col2' are true

C Deletes columns 'col1' and 'col2'

D Sorts the DataFrame by 'col1' and 'col2'

select() picks columns, not rows or sorting.

Which operation keeps rows where a condition is true?

A groupBy()

B select()

C filter()

D join()

filter() keeps rows matching a condition.

Are filter() and where() different in Spark?

A No, they are the same

B Yes, they do different things

C Only <code>filter()</code> works on DataFrames

D Only <code>where()</code> works on DataFrames

Both do the same filtering job.

How to filter rows where age is greater than 25?

A df.filter(df.age &lt; 25)

B df.filter(df.age &gt; 25)

C df.select(df.age &gt; 25)

D df.where(df.age == 25)

Use filter() or where() with the condition.

What happens if you chain select() and filter()?

A It sorts the DataFrame

B You filter rows first, then select columns

C It causes an error

D You select columns first, then keep rows matching filter

Operations run in order: select columns, then filter rows.

Explain how to use select(), filter(), and where() in Apache Spark DataFrames with simple examples.

Think about picking columns like choosing ingredients and filtering rows like picking fruits that are ripe.

Describe the difference and similarity between filter() and where() in Spark.

They are like two words for the same action.