Partition tuning (repartition vs coalesce) in Apache Spark - Interactive Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to increase the number of partitions of the DataFrame.

Apache Spark

df_repartitioned = df.[1](10)

Drag options to blanks, or click blank then click option'

Arepartition

Bcoalesce

CpartitionBy

DgroupBy

Attempts:

3 left

💡 Hint

Common Mistakes

Using coalesce to increase partitions causes no change or error.

Using partitionBy or groupBy does not change partitions directly.

Use repartition to increase or change the number of partitions.

2fill in blank

medium

Complete the code to reduce the number of partitions without a full shuffle.

Apache Spark

df_coalesced = df.[1](5)

Drag options to blanks, or click blank then click option'

Arepartition

Bcoalesce

CgroupBy

DpartitionBy

Attempts:

3 left

💡 Hint

Common Mistakes

Using repartition to reduce partitions causes unnecessary shuffle.

Using groupBy or partitionBy does not reduce partitions directly.

coalesce reduces partitions efficiently without a full shuffle.

3fill in blank

hard

Fix the error in the code to repartition the DataFrame by a column.

Apache Spark

df_partitioned = df.[1]('category')

Drag options to blanks, or click blank then click option'

Arepartition

Bcoalesce

CpartitionBy

DgroupBy

Attempts:

3 left

💡 Hint

Common Mistakes

Using coalesce with a column causes an error.

Using partitionBy is for writing files, not repartitioning DataFrames.

Use repartition with a column name to shuffle and repartition by that column.

4fill in blank

hard

Fill both blanks to create a DataFrame with fewer partitions using the efficient method.

Apache Spark

df_small = df.[1]([2])

Drag options to blanks, or click blank then click option'

Acoalesce

Brepartition

C10

D5

Attempts:

3 left

💡 Hint

Common Mistakes

Using repartition to reduce partitions causes shuffle.

Choosing a larger number of partitions does not reduce partitions.

coalesce(5) reduces partitions to 5 efficiently without shuffle.

5fill in blank

hard

Fill all three blanks to create a dictionary mapping words to their lengths for words longer than 3 letters.

Apache Spark

word_counts = {word: [1] for word in words if len(word) [2] 3 and word [3] stop_words}

Drag options to blanks, or click blank then click option'

Alen(word)

B>

Cnot in

Din

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'in' instead of 'not in' includes stop words.

Using '<' instead of '>' filters wrong words.

This dictionary comprehension maps words to their lengths for words longer than 3 letters and not in stop_words.