Window functions in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a window function in Apache Spark?

A window function performs calculations across a set of rows related to the current row without collapsing the rows into a single output row. It lets you do things like running totals or ranking within groups.

beginner

What does the partitionBy clause do in a window specification?

It divides the data into groups (partitions) so the window function works independently within each group, like calculating ranks separately for each department in a company.

intermediate

Explain the difference between orderBy and partitionBy in window functions.

partitionBy groups rows into separate sets. orderBy sorts rows within each partition to define the order for calculations like running totals or ranks.

intermediate

What is the purpose of the rowsBetween method in window functions?

It defines the frame of rows around the current row to include in the calculation, for example, the previous 2 rows and the next 2 rows, helping to calculate moving averages.

beginner

Give an example of a common window function in Spark and its use.

The rank() function assigns a rank to each row within a partition ordered by a column. For example, ranking salespeople by sales amount within each region.

What does a window function NOT do?

A Calculate running totals

B Calculate moving averages

C Rank rows within groups

D Aggregate rows into a single output row

Window functions do calculations across rows but keep all rows in the output; they do not collapse rows into one.

Which clause in a window specification divides data into groups?

A orderBy

B partitionBy

C rowsBetween

D filter

partitionBy splits data into groups for independent window calculations.

What does orderBy do inside a window function?

A Filters rows

B Removes duplicates

C Sorts rows within partitions

D Groups rows

orderBy sorts rows within each partition to define calculation order.

Which method defines the frame of rows around the current row?

A rowsBetween

B orderBy

C partitionBy

D select

rowsBetween sets the range of rows to include in the window frame.

What would rank() do in a window function?

A Assign a rank to rows within partitions

B Calculate sum of values

C Filter rows by rank

D Calculate average

rank() assigns ranks to rows ordered within each partition.

Explain how window functions differ from regular aggregation functions in Spark.

Think about whether rows are combined or kept separate.

Describe how you would use partitionBy and orderBy together in a window function.

Consider grouping first, then sorting inside groups.

Practice

1. What is the main purpose of window functions in Apache Spark?

easy

A. To perform calculations across rows related to the current row without collapsing the data

B. To group data and reduce the number of rows

C. To filter rows based on a condition

D. To join two datasets together

Solution

1. Step 1: Understand window function behavior

   Window functions calculate values across related rows but keep all original rows intact.

2. Step 2: Compare with grouping

   Grouping reduces rows by aggregation, but window functions add columns without reducing rows.

3. Final Answer:

   To perform calculations across rows related to the current row without collapsing the data -> Option A

4. Quick Check:

   Window functions add calculations without grouping ≠ A [OK]

Hint: Window functions add columns, grouping reduces rows [OK]

Common Mistakes:

• Confusing window functions with groupBy aggregations
• Thinking window functions filter rows
• Assuming window functions join datasets

2. Which of the following is the correct way to define a window specification in PySpark to partition by 'department' and order by 'salary'?

easy

A. Window.groupBy('department').sortBy('salary')

B. Window.sortBy('salary').partitionBy('department')

C. Window.partitionBy('department').orderBy('salary')

D. Window.partition('department').order('salary')

Solution

1. Step 1: Recall PySpark window spec syntax

   The correct methods are partitionBy() and orderBy() chained on Window.

2. Step 2: Check method order and names

   partitionBy comes first, then orderBy; groupBy and sortBy are incorrect here.

3. Final Answer:

   Window.partitionBy('department').orderBy('salary') -> Option C

4. Quick Check:

   Use partitionBy then orderBy methods [OK]

Hint: Use Window.partitionBy().orderBy() exactly [OK]

Common Mistakes:

• Using groupBy instead of partitionBy
• Using sortBy instead of orderBy
• Using incorrect method names like partition or order

3. Given the following PySpark code, what will be the output of the 'rank' column?

from pyspark.sql import SparkSession
from pyspark.sql.window import Window
from pyspark.sql.functions import rank

spark = SparkSession.builder.getOrCreate()
data = [("Alice", "Sales", 5000), ("Bob", "Sales", 4000), ("Charlie", "Sales", 5000)]
df = spark.createDataFrame(data, ["name", "department", "salary"])
windowSpec = Window.partitionBy("department").orderBy("salary")
df = df.withColumn("rank", rank().over(windowSpec))
df.orderBy("salary").show()

medium

A. [ ("Bob", 4000, 1), ("Alice", 5000, 2), ("Charlie", 5000, 2) ]

B. [ ("Bob", 4000, 1), ("Alice", 5000, 3), ("Charlie", 5000, 3) ]

C. [ ("Bob", 4000, 1), ("Alice", 5000, 2), ("Charlie", 5000, 3) ]

D. [ ("Bob", 4000, 1), ("Alice", 5000, 1), ("Charlie", 5000, 1) ]

Solution

1. Step 1: Understand rank() behavior

   Rank assigns the same rank to ties and skips ranks after ties.

2. Step 2: Apply rank over salary ascending

   Bob has salary 4000 ranked 1; Alice and Charlie both have 5000, so both get rank 2.

3. Final Answer:

   [ ("Bob", 4000, 1), ("Alice", 5000, 2), ("Charlie", 5000, 2) ] -> Option A

4. Quick Check:

   rank() ties share rank, next rank skips [OK]

Hint: rank() gives same rank to ties, skips next ranks [OK]

Common Mistakes:

• Confusing rank() with dense_rank()
• Assigning different ranks to tied values
• Not ordering data before applying rank

4. Identify the error in this PySpark code snippet using window functions:

from pyspark.sql.window import Window
from pyspark.sql.functions import row_number

windowSpec = Window.partitionBy('department')
df = df.withColumn('row_num', row_number().over(windowSpec.orderBy('salary')))

medium

A. Missing import for Window functions causes error

B. windowSpec.orderBy('salary') is valid; no error in code

C. row_number() cannot be used with partitionBy

D. windowSpec.orderBy('salary') is invalid; orderBy should be called on Window, not on windowSpec

Solution

1. Step 1: Check windowSpec usage

   windowSpec is a WindowSpec object; calling orderBy on it is valid and returns a new WindowSpec.

2. Step 2: Confirm chaining

   Chaining orderBy on windowSpec is allowed to refine the window specification.

3. Final Answer:

   windowSpec.orderBy('salary') is valid; no error in code -> Option B

4. Quick Check:

   WindowSpec supports chaining orderBy to refine window [OK]

Hint: WindowSpec supports chaining orderBy to refine window [OK]

Common Mistakes:

• Calling orderBy on windowSpec instead of Window
• Assuming row_number can't be used with partitionBy
• Forgetting to import Window or functions

5. You have a sales dataset with columns: 'store', 'date', and 'sales'. You want to add a column showing the 3-day moving average of sales per store ordered by date. Which PySpark window function setup and code will achieve this?

hard

A. Use Window.partitionBy('store').orderBy('date').rowsBetween(0, 2) and avg('sales').over(windowSpec)

B. Use Window.partitionBy('store').orderBy('date').rangeBetween(-2, 0) and avg('sales').over(windowSpec)

C. Use Window.orderBy('date').rowsBetween(-3, 0) and avg('sales').over(windowSpec)

D. Use Window.partitionBy('store').orderBy('date').rowsBetween(-2, 0) and avg('sales').over(windowSpec)

Solution

1. Step 1: Define window for 3-day moving average per store

   Partition by 'store' to separate stores, order by 'date' for time sequence, use rowsBetween(-2, 0) for current and previous 2 rows.

2. Step 2: Apply avg() over this window

   Use avg('sales').over(windowSpec) to compute moving average over the defined window.

3. Final Answer:

   Use Window.partitionBy('store').orderBy('date').rowsBetween(-2, 0) and avg('sales').over(windowSpec) -> Option D

4. Quick Check:

   rowsBetween(-2,0) for 3-day window, partition by store [OK]

Hint: Use rowsBetween(-2,0) for 3-row moving average [OK]

Common Mistakes:

• Using rangeBetween instead of rowsBetween for row count
• Not partitioning by store
• Using wrong rowsBetween range (like 0 to 2)