Understanding partitions in Apache Spark - Interactive Quiz & Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to get the number of partitions in the DataFrame.

Apache Spark

num_partitions = df.rdd.[1]

Drag options to blanks, or click blank then click option'

Acount

BgetNumPartitions

Cpartitions

DnumPartitions

Attempts:

3 left

💡 Hint

Common Mistakes

Using a method name instead of the property.

Trying to call a method that doesn't exist.

The numPartitions property gives the number of partitions in the RDD underlying the DataFrame.

2fill in blank

medium

Complete the code to repartition the DataFrame into 4 partitions.

Apache Spark

df_repart = df.[1](4)

Drag options to blanks, or click blank then click option'

ApartitionBy

Bcoalesce

Crepartition

Dsplit

Attempts:

3 left

💡 Hint

Common Mistakes

Using coalesce which only reduces partitions without shuffle.

Using partitionBy which is for writing files.

The repartition() method reshuffles the data into the specified number of partitions.

3fill in blank

hard

Fix the error in the code to reduce partitions without full shuffle.

Apache Spark

df_less = df.[1](2)

Drag options to blanks, or click blank then click option'

Acoalesce

BpartitionBy

Crepartition

Dsplit

Attempts:

3 left

💡 Hint

Common Mistakes

Using repartition which causes a full shuffle.

Using partitionBy which is not for repartitioning in memory.

The coalesce() method reduces the number of partitions without a full shuffle, making it efficient.

4fill in blank

hard

Fill both blanks to create a dictionary with word lengths for words longer than 3 characters.

Apache Spark

lengths = {word: [1] for word in words if len(word) [2] 3}

Drag options to blanks, or click blank then click option'

Alen(word)

B>

C<

Dword

Attempts:

3 left

💡 Hint

Common Mistakes

Using the word itself as the value instead of its length.

Using the wrong comparison operator.

The dictionary comprehension maps each word to its length if the word length is greater than 3.

5fill in blank

hard

Fill all three blanks to create a dictionary with uppercase words as keys and their lengths as values for words longer than 4.

Apache Spark

result = { [1]: [2] for w in words if len(w) [3] 4 }

Drag options to blanks, or click blank then click option'

Aw.upper()

Blen(w)

C>

Dw.lower()

Attempts:

3 left

💡 Hint

Common Mistakes

Using lowercase instead of uppercase for keys.

Using the wrong comparison operator.

Swapping keys and values.

This dictionary comprehension creates keys as uppercase words and values as their lengths, filtering words longer than 4 characters.

Practice

1. What is the main purpose of partitions in Apache Spark?

easy

A. To compress data to save space

B. To split data into smaller chunks for parallel processing

C. To encrypt data for security

D. To store data permanently on disk

Solution

1. Step 1: Understand what partitions do in Spark

   Partitions divide data into smaller parts so Spark can process them at the same time.

2. Step 2: Compare options with this purpose

   Only To split data into smaller chunks for parallel processing matches this purpose; others describe unrelated tasks.

3. Final Answer:

   To split data into smaller chunks for parallel processing -> Option B

4. Quick Check:

   Partitions = split data for parallelism [OK]

Hint: Partitions split data to run tasks in parallel [OK]

Common Mistakes:

• Confusing partitions with data storage
• Thinking partitions encrypt data
• Assuming partitions compress data

2. Which of the following is the correct way to set the number of partitions when creating an RDD in Spark?

easy

A. sc.parallelize(data, 4)

B. sc.parallelize(data, numPartitions=4)

C. sc.parallelize(data).setPartitions(4)

D. sc.parallelize(data).partitions(4)

Solution

1. Step 1: Recall the syntax of sc.parallelize

   The method sc.parallelize takes data and an optional second argument for number of partitions as a positional argument.

2. Step 2: Check which option matches this syntax

   sc.parallelize(data, 4) uses sc.parallelize(data, 4) which is correct. sc.parallelize(data, numPartitions=4) uses a keyword argument which is invalid. Options C and D use non-existent methods.

3. Final Answer:

   sc.parallelize(data, 4) -> Option A

4. Quick Check:

   sc.parallelize(data, 4) sets partitions [OK]

Hint: Use second positional argument to set partitions [OK]

Common Mistakes:

• Using keyword argument numPartitions (not supported)
• Trying to set partitions after creation with wrong methods
• Confusing method names for partition setting

3. Given the following Spark code, what will be the number of partitions after execution?

rdd = sc.parallelize(range(10), 3)
rdd2 = rdd.repartition(5)
print(rdd2.getNumPartitions())

medium

A. 3

B. 1

C. 10

D. 5

Solution

1. Step 1: Understand initial partitioning

   The RDD is created with 3 partitions using sc.parallelize(range(10), 3).

2. Step 2: Analyze repartition call

   Calling rdd.repartition(5) changes the number of partitions to 5.

3. Final Answer:

   5 -> Option D

4. Quick Check:

   repartition(5) sets partitions to 5 [OK]

Hint: repartition changes partitions to given number [OK]

Common Mistakes:

• Assuming repartition keeps original partition count
• Confusing repartition with coalesce
• Thinking getNumPartitions returns data size

4. Identify the error in the following Spark code snippet:

rdd = sc.parallelize([1, 2, 3, 4])
rdd2 = rdd.repartition(-2)
print(rdd2.getNumPartitions())

medium

A. parallelize requires at least 2 partitions

B. repartition method does not exist

C. Negative number of partitions is invalid

D. getNumPartitions cannot be called on rdd2

Solution

1. Step 1: Check repartition argument

   repartition requires a positive integer for number of partitions. Negative values are invalid.

2. Step 2: Verify other method usages

   repartition exists, parallelize accepts any positive partition count, getNumPartitions works on RDDs.

3. Final Answer:

   Negative number of partitions is invalid -> Option C

4. Quick Check:

   Partitions must be positive [OK]

Hint: Partitions count must be positive integer [OK]

Common Mistakes:

• Using negative or zero partitions
• Thinking repartition method is missing
• Assuming getNumPartitions is invalid

5. You have a large dataset with 100 partitions, but your cluster has only 10 cores. Which Spark operation is best to optimize performance by reducing the number of partitions without causing a full shuffle?

hard

A. Use coalesce(10) to reduce partitions

B. Use repartition(100) to keep partitions same

C. Use repartition(10) to reduce partitions

D. Use mapPartitions to manually split partitions

Solution

1. Step 1: Understand cluster resources and partitions

   Having more partitions than cores can cause overhead. Reducing partitions to match cores improves efficiency.

2. Step 2: Choose operation that reduces partitions without full shuffle

   coalesce reduces partitions efficiently without full shuffle, unlike repartition which causes shuffle.

3. Final Answer:

   Use coalesce(10) to reduce partitions -> Option A

4. Quick Check:

   coalesce reduces partitions without shuffle [OK]

Hint: Use coalesce to reduce partitions without shuffle [OK]

Common Mistakes:

• Using repartition causes expensive shuffle
• Not matching partitions to cluster cores
• Trying manual partition splitting unnecessarily