Memory management and garbage collection in Apache Spark - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to set the Spark executor memory to 4 gigabytes.

Apache Spark

spark.conf.set("spark.executor.memory", "[1]")

Drag options to blanks, or click blank then click option'

A40g

B4000mb

C4mb

D4g

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'mb' instead of 'g' for gigabytes

Setting an unrealistically large memory size

The correct way to set executor memory is using '4g' for 4 gigabytes.

2fill in blank

medium

Complete the code to trigger garbage collection manually in Spark's JVM.

Apache Spark

System.[1]()

Drag options to blanks, or click blank then click option'

Acleanup

Bcollect

Cgc

Dclear

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'collect' which is not a JVM method

Using 'clear' or 'cleanup' which do not exist on System

Calling System.gc() requests JVM to perform garbage collection.

3fill in blank

hard

Fix the error in the Spark configuration to enable dynamic allocation of executors.

Apache Spark

spark.conf.set("spark.dynamicAllocation.[1]", "true")

Drag options to blanks, or click blank then click option'

Aenabled

Benable

Cactivator

Dactive

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'enable' instead of 'enabled'

Using unrelated words like 'activator' or 'active'

The correct property name is 'spark.dynamicAllocation.enabled' to enable dynamic allocation.

4fill in blank

hard

Fill both blanks to configure Spark to use a specific garbage collector and set its pause time.

Apache Spark

spark.conf.set("spark.executor.extraJavaOptions", "-XX:+Use[1]GC -XX:MaxGCPauseMillis=[2]")

Drag options to blanks, or click blank then click option'

AG1

B100

CParallel

D200

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'Parallel' with a pause time that is too low

Mixing garbage collector names and pause time values incorrectly

Use '-XX:+UseG1GC' to select the G1 garbage collector and set max pause time to 100 ms.

5fill in blank

hard

Fill all three blanks to create a Spark configuration dictionary that sets executor memory, enables dynamic allocation, and sets shuffle partitions.

Apache Spark

{"spark.executor.memory": "[1]", "spark.dynamicAllocation.enabled": "[2]", "spark.sql.shuffle.partitions": "[3]"}

Drag options to blanks, or click blank then click option'

A8g

Btrue

C200

Dfalse

Attempts:

3 left

💡 Hint

Common Mistakes

Setting dynamic allocation to 'false' when it should be 'true'

Using too low or too high shuffle partition numbers

This dictionary sets executor memory to 8 GB, enables dynamic allocation, and sets shuffle partitions to 200.

Practice

1. What is the main purpose of garbage collection in Apache Spark?

easy

A. To increase the size of the memory available to Spark

B. To automatically free up memory by removing unused objects

C. To manually allocate memory for Spark tasks

D. To store data permanently on disk

Solution

1. Step 1: Understand garbage collection role

   Garbage collection automatically removes objects that are no longer needed to free memory.

2. Step 2: Identify purpose in Spark context

   This helps Spark avoid running out of memory and keeps it running smoothly.

3. Final Answer:

   To automatically free up memory by removing unused objects -> Option B

4. Quick Check:

   Garbage collection = frees unused memory [OK]

Hint: Garbage collection cleans unused data automatically [OK]

Common Mistakes:

• Thinking garbage collection increases memory size
• Confusing manual memory allocation with garbage collection
• Believing garbage collection stores data permanently

2. Which of the following is the correct way to set the executor memory in Spark configuration?

easy

A. spark.executor.memory: 4g

B. spark.executor.memory = 4g

C. set spark.executor.memory to 4g

D. spark.executor.memory 4g

Solution

1. Step 1: Recall Spark config syntax

   Spark configuration in spark-defaults.conf uses "key value" format with space separator, no equals sign.

2. Step 2: Check each option

   A: "spark.executor.memory 4g" matches exactly. B has = incorrect for conf file. C and D invalid syntax.

3. Final Answer:

   spark.executor.memory 4g -> Option D

4. Quick Check:

   Config syntax = key value [OK]

Hint: Spark configs use key value format with space [OK]

Common Mistakes:

• Using '=' sign instead of space separator
• Writing config as a sentence
• Using colon instead of space

3. Given the Spark code snippet below, what will happen if the executor memory is too small?

val spark = SparkSession.builder().appName("Test").getOrCreate()
spark.conf.set("spark.executor.memory", "512m")
val data = spark.range(100000000).cache()
data.count()

medium

A. The job may fail with an OutOfMemoryError during execution

B. The job will store data only on disk without caching

C. The job will ignore the memory setting and use default

D. The job will run faster because less memory is used

Solution

1. Step 1: Understand executor memory effect

   Setting executor memory too low limits how much data Spark can hold in memory.

2. Step 2: Analyze caching and large data

   Caching large data with small memory causes Spark to run out of memory and throw errors.

3. Final Answer:

   The job may fail with an OutOfMemoryError during execution -> Option A

4. Quick Check:

   Small memory + caching large data = failure [OK]

Hint: Small executor memory + caching large data causes failure [OK]

Common Mistakes:

• Assuming job runs faster with less memory
• Thinking Spark ignores memory settings
• Believing caching always stores data on disk

4. You see frequent garbage collection pauses slowing down your Spark job. Which change can help reduce these pauses?

medium

A. Disable caching of data

B. Decrease the number of Spark partitions

C. Increase executor memory size

D. Set spark.executor.memory to zero

Solution

1. Step 1: Identify cause of GC pauses

   Frequent garbage collection happens when memory is too small to hold data efficiently.

2. Step 2: Choose solution to reduce GC pauses

   Increasing executor memory gives more space, reducing how often GC runs.

3. Final Answer:

   Increase executor memory size -> Option C

4. Quick Check:

   More memory = fewer GC pauses [OK]

Hint: More memory reduces garbage collection frequency [OK]

Common Mistakes:

• Setting memory to zero causes job failure
• Decreasing partitions may not reduce GC pauses
• Disabling caching may not fix GC issues

5. You want to optimize a Spark job that processes large datasets and experiences frequent garbage collection. Which combination of settings is best to improve performance?

hard

A. Increase executor memory and enable Tungsten memory management

B. Decrease executor memory and disable caching

C. Set executor memory to default and increase shuffle partitions

D. Disable garbage collection and increase driver memory

Solution

1. Step 1: Understand memory and GC optimization

   Increasing executor memory reduces GC pressure. Tungsten optimizes memory use in Spark.

2. Step 2: Evaluate options for large data processing

   Increase executor memory and enable Tungsten memory management combines more memory with efficient memory management, improving performance.

3. Final Answer:

   Increase executor memory and enable Tungsten memory management -> Option A

4. Quick Check:

   More memory + Tungsten = better performance [OK]

Hint: Combine more memory with Tungsten for best results [OK]

Common Mistakes:

• Decreasing memory worsens GC issues
• Disabling GC causes crashes
• Only increasing shuffle partitions doesn't fix GC