Caching and persistence in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output count after caching?

Consider the following Apache Spark code snippet:

data = spark.range(1000)
cached_data = data.cache()
count1 = cached_data.count()
count2 = cached_data.count()

What is the value of count2?

Apache Spark

data = spark.range(1000)
cached_data = data.cache()
count1 = cached_data.count()
count2 = cached_data.count()
print(count2)

A Raises an error

B 0

C None

D 1000

💡 Hint

Caching stores the data in memory after the first action.

The first count() triggers computation and caches the data. The second count() uses the cached data, so it returns 1000 immediately.

❓ data_output

intermediate

What is the storage level after persisting with MEMORY_AND_DISK?

Given this Spark code:

df = spark.range(10)
df.persist(StorageLevel.MEMORY_AND_DISK)
print(df.storageLevel)

What will be printed?

Apache Spark

from pyspark import StorageLevel
df = spark.range(10)
df.persist(StorageLevel.MEMORY_AND_DISK)
print(df.storageLevel)

A StorageLevel(memory=True, disk=True, offHeap=False, deserialized=True, replication=1)

B StorageLevel(memory=False, disk=True, offHeap=False, deserialized=True, replication=1)

C StorageLevel(memory=True, disk=False, offHeap=False, deserialized=True, replication=1)

D StorageLevel(memory=False, disk=False, offHeap=False, deserialized=False, replication=1)

💡 Hint

MEMORY_AND_DISK means data is stored in memory and spilled to disk if needed.

The persist(StorageLevel.MEMORY_AND_DISK) sets the storage level to keep data in memory and disk with deserialized format.

🔧 Debug

advanced

Why does unpersisting not free memory immediately?

In Spark, after calling df.unpersist(), the memory is not freed immediately. Why?

A Because unpersist() is asynchronous and frees memory lazily

B Because unpersist() deletes the original data source

C Because unpersist() caches data again automatically

D Because unpersist() triggers a job that blocks memory release

💡 Hint

Think about how Spark manages memory and tasks asynchronously.

Calling unpersist() marks the data for removal but frees memory lazily after current tasks finish.

🚀 Application

advanced

Choosing the right persistence level for iterative algorithms

You have a large DataFrame used in multiple iterations of a machine learning algorithm. Which persistence level is best to optimize performance and resource usage?

A StorageLevel.DISK_ONLY

B StorageLevel.MEMORY_AND_DISK_SER

C StorageLevel.MEMORY_ONLY

D StorageLevel.OFF_HEAP

💡 Hint

Consider serialization and fallback when memory is insufficient.

MEMORY_AND_DISK_SER stores serialized data in memory and spills to disk if needed, saving memory and improving performance for iterative tasks.

🧠 Conceptual

expert

What happens if you cache a DataFrame but the cluster runs out of memory?

When you cache a DataFrame in Spark and the cluster memory is full, what is the expected behavior?

A Spark will automatically increase cluster memory to fit the cache

B Spark will crash immediately due to out-of-memory error

C Spark will evict cached partitions using LRU policy and recompute them when needed

D Spark will convert the DataFrame to disk-only storage without eviction

💡 Hint

Think about Spark's memory management and eviction policies.

Spark uses a Least Recently Used (LRU) policy to evict cached partitions when memory is full, allowing recomputation later.