Why optimization prevents job failures in Apache Spark - Test Your Understanding

Practice - 5 Tasks

1fill in blank

easy

Complete the code to cache the DataFrame to optimize performance.

Apache Spark

df = spark.read.csv('data.csv')
df.[1]()

Drag options to blanks, or click blank then click option'

Ashow

Bcollect

Ccache

Dcount

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'collect' which brings data to the driver and can cause memory errors.

Using 'show' which only displays data but does not optimize.

Using 'count' which triggers computation but does not cache.

Caching the DataFrame stores it in memory, which speeds up repeated operations and helps prevent job failures due to repeated expensive computations.

2fill in blank

medium

Complete the code to repartition the DataFrame for better parallelism.

Apache Spark

df = df.[1](10)

Drag options to blanks, or click blank then click option'

Arepartition

Bpersist

Ccollect

Dcache

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'cache' which stores data but does not change partitions.

Using 'collect' which brings data to the driver and can cause failures.

Using 'persist' which is similar to cache but does not repartition.

Repartitioning the DataFrame increases the number of partitions, improving parallel processing and reducing job failures caused by data skew or resource bottlenecks.

3fill in blank

hard

Fix the error in the code to avoid job failure by using the correct action.

Apache Spark

result = df.filter(df.age > 30)
result.[1]()

Drag options to blanks, or click blank then click option'

Apersist

Bshow

Ccache

Dmap

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'cache' or 'persist' without an action, so job does not run.

Using 'map' which is not an action and causes errors.

'show()' is an action that triggers computation and displays results. Using 'cache()' or 'persist()' alone does not trigger execution, and 'map' is a transformation, not an action.

4fill in blank

hard

Fill both blanks to create a dictionary of word lengths for words longer than 3 characters.

Apache Spark

lengths = {word: [1] for word in words if len(word) [2] 3}

Drag options to blanks, or click blank then click option'

Alen(word)

B>

C<

Dword

Attempts:

3 left

💡 Hint

Common Mistakes

Using '<' instead of '>' causing wrong filtering.

Using 'word' as value instead of length.

The dictionary comprehension maps each word to its length using 'len(word)'. The condition filters words with length greater than 3 using '>'.

5fill in blank

hard

Fill all three blanks to create a filtered dictionary with uppercase keys for words longer than 4 characters.

Apache Spark

result = [1]([2]: [3] for [2] in words if len([2]) > 4)

Drag options to blanks, or click blank then click option'

Adict

Bword

Cword.upper()

Dlist

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'list' instead of 'dict' causing type errors.

Using 'word.upper()' as key instead of 'word'.

Using inconsistent variable names.

This code creates a dictionary ('dict') with keys as the original words ('word') and values as uppercase words ('word.upper()') filtered by length greater than 4.