Windowed aggregations in Apache Spark - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to import the window function module from PySpark.

Apache Spark

from pyspark.sql import [1]

Drag options to blanks, or click blank then click option'

Atypes

Bwindow

Cfunctions as F

Dsql

Attempts:

3 left

💡 Hint

Common Mistakes

Importing 'window' directly instead of 'functions as F'.

Forgetting to alias functions as F.

We import PySpark SQL functions as F to use window functions like row_number().

2fill in blank

medium

Complete the code to define a window partitioned by 'department' and ordered by 'salary' descending.

Apache Spark

from pyspark.sql.window import Window

window_spec = Window.partitionBy('department').[1](F.col('salary').desc())

Drag options to blanks, or click blank then click option'

AorderBy

BsortBy

Corder

Dsort

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'sortBy' which is not a valid method for window specs.

Using 'order' or 'sort' which do not exist.

The correct method to order rows in a window is orderBy.

3fill in blank

hard

Fix the error in the code to add a row number column using the window specification.

Apache Spark

df = df.withColumn('row_num', F.[1]().over(window_spec))

Drag options to blanks, or click blank then click option'

Arow_number

Brank

Cdense_rank

Dcount

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'rank' or 'dense_rank' which assign ranks but can have ties.

Using 'count' which is an aggregation, not a window function.

row_number() assigns a unique row number within the window partition.

4fill in blank

hard

Fill both blanks to create a window partitioned by 'team' and ordered by 'date' ascending.

Apache Spark

window_spec = Window.[1]('team').[2](F.col('date').asc())

Drag options to blanks, or click blank then click option'

ApartitionBy

BorderBy

CsortBy

DgroupBy

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'groupBy' instead of 'partitionBy'.

Using 'sortBy' which is invalid for window specs.

Window specs use partitionBy to group and orderBy to sort rows.

5fill in blank

hard

Fill all three blanks to create a dictionary comprehension that maps each word to its length if length is greater than 3.

Apache Spark

lengths = { [1]: [2] for [3] in words if len([3]) > 3 }

Drag options to blanks, or click blank then click option'

Aword

Blen(word)

Ditem

Attempts:

3 left

💡 Hint

Common Mistakes

Using different variable names for key and loop variable.

Using 'item' instead of 'word' as loop variable.

The comprehension maps each word to its length using len(word) for words longer than 3 characters.