Caching and persistence in Apache Spark - Step-by-Step Execution

Concept Flow - Caching and persistence

Create RDD/DataFrame

↓

Perform Transformations

↓

Cache or Persist?

No→Compute on action

Yes↓

Store in Memory/Disk

↓

Trigger Action (e.g., count)

↓

Reuse Cached Data

↓

Faster Subsequent Actions

↓

Optionally Unpersist to Free Memory

This flow shows how Spark caches or persists data after transformations to speed up repeated actions by storing data in memory or disk.

Execution Sample

Apache Spark

rdd = sc.parallelize([1, 2, 3, 4])
rdd2 = rdd.map(lambda x: x * 2)
rdd2.cache()
print(rdd2.count())
print(rdd2.collect())

This code creates an RDD, doubles each element, caches the result, then counts and collects the cached data.

Execution Table

Step	Action	Evaluation	Result
1	Create RDD with [1,2,3,4]	RDD created	[1,2,3,4]
2	Apply map to double each element	Transformation defined	RDD with [2,4,6,8] (not computed yet)
3	Call cache()	Mark RDD for caching	RDD marked cached (lazy)
4	Call count() action	Triggers computation and caching	Count = 4; RDD cached in memory
5	Call collect() action	Uses cached data	[2,4,6,8] returned quickly

💡 All actions complete; cached data reused for collect after count

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5
rdd	[1,2,3,4]	[1,2,3,4]	[1,2,3,4]	[1,2,3,4]	[1,2,3,4]
rdd2	undefined	RDD with [2,4,6,8] (lazy)	Marked cached	Cached in memory	Cached in memory

Key Moments - 2 Insights

Why doesn't calling cache() immediately compute the RDD?

cache() only marks the RDD to be cached; actual computation happens when an action like count() triggers it, as shown in step 4 of the execution_table.

How does caching speed up the collect() action after count()?

Because the RDD was cached in memory during count(), collect() reuses this cached data instead of recomputing, making it faster (step 5).

Visual Quiz - 3 Questions

Look at the execution_table, what is the result of the count() action at step 4?

A 4

B 8

C [2,4,6,8]

D RDD marked cached

💡 Hint

Check the 'Result' column at step 4 in the execution_table.

At which step is the RDD actually cached in memory?

A Step 3

B Step 4

C Step 2

D Step 5

💡 Hint

Look for when caching is triggered by an action in the execution_table.

If we remove the cache() call, what would happen at step 5 when collect() is called?

A Collect would use cached data

B Count would fail

C Collect would recompute the RDD

D RDD would be cached automatically

💡 Hint

Refer to the role of cache() in the key_moments and execution_table.

Concept Snapshot

Caching and persistence in Spark:
- cache() marks data to store in memory
- persist() can store in memory/disk with levels
- Actual caching happens on action (e.g., count)
- Speeds up repeated actions by reusing data
- Use unpersist() to free memory when done

Full Transcript

In Apache Spark, caching and persistence help speed up repeated computations. When you create an RDD or DataFrame and apply transformations, the data is not computed immediately. Calling cache() or persist() marks the data to be stored in memory or disk after an action triggers computation. For example, calling count() computes and caches the data. Subsequent actions like collect() reuse this cached data, making them faster. This process avoids recomputing the same data multiple times. You can free memory by calling unpersist() when caching is no longer needed.

Practice

1. What is the main purpose of using cache() in Apache Spark?

easy

A. To delete data from memory

B. To permanently save data to disk

C. To store data in memory for faster repeated access

D. To convert data into a different format

Solution

1. Step 1: Understand what caching means in Spark

   Caching stores data in memory to speed up repeated operations on the same data.

2. Step 2: Identify the role of cache()

   The cache() method specifically stores the DataFrame or RDD in memory for quick access.

3. Final Answer:

   To store data in memory for faster repeated access -> Option C

4. Quick Check:

   Caching = store in memory [OK]

Hint: Cache means keep data in memory for speed [OK]

Common Mistakes:

• Thinking cache saves data permanently on disk
• Confusing cache with deleting data
• Assuming cache changes data format

2. Which of the following is the correct syntax to persist a DataFrame df with disk storage level in Spark?

easy

A. df.persist(StorageLevel.DISK_ONLY)

B. df.cache(StorageLevel.DISK_ONLY)

C. df.persist('disk_only')

D. df.cache('disk_only')

Solution

1. Step 1: Recall the correct method to specify storage level

   To persist with a specific storage level, use persist() with a StorageLevel object.

2. Step 2: Check the syntax for disk-only persistence

   The correct syntax is df.persist(StorageLevel.DISK_ONLY). The cache() method does not accept parameters.

3. Final Answer:

   df.persist(StorageLevel.DISK_ONLY) -> Option A

4. Quick Check:

   persist + StorageLevel object = correct syntax [OK]

Hint: Use persist() with StorageLevel, cache() has no params [OK]

Common Mistakes:

• Using cache() with parameters
• Passing storage level as string instead of StorageLevel object
• Using persist() without StorageLevel

3. Consider the following Spark code snippet:

val df = spark.read.csv("data.csv")
df.cache()
df.count()
df.count()

What is the effect of calling df.count() twice after caching?

medium

A. Both counts reload data from disk without caching

B. The first count triggers data loading and caching; the second count uses cached data for faster result

C. The second count triggers caching; the first count uses cached data

D. Both counts use cached data without loading from disk

Solution

1. Step 1: Understand caching behavior on first action

   The first action (count()) triggers loading data and caches it in memory.

2. Step 2: Effect on second action

   The second count() uses the cached data, so it runs faster without reloading from disk.

3. Final Answer:

   The first count triggers data loading and caching; the second count uses cached data for faster result -> Option B

4. Quick Check:

   First action caches, second uses cache [OK]

Hint: First action caches, second action reads cache [OK]

Common Mistakes:

• Assuming cache happens before first action
• Thinking both counts reload from disk
• Believing second count triggers caching

4. Given this Spark code:

val df = spark.read.json("data.json")
df.persist(StorageLevel.MEMORY_AND_DISK)
df.unpersist()

What is the main issue with this code?

medium

A. Calling unpersist() immediately removes cached data, so persistence is ineffective

B. The persist() method requires a string, not StorageLevel object

C. You must call cache() before persist()

D. The code will cause a syntax error due to missing parentheses

Solution

1. Step 1: Understand the effect of persist()

   The persist() call stores the DataFrame in memory and disk as specified.

2. Step 2: Effect of immediate unpersist()

   Calling unpersist() right after removes the cached data, negating the persistence benefit.

3. Final Answer:

   Calling unpersist() immediately removes cached data, so persistence is ineffective -> Option A

4. Quick Check:

   Immediate unpersist clears cache [OK]

Hint: Unpersist right after persist clears cache [OK]

Common Mistakes:

• Thinking persist needs string argument
• Believing cache must be called before persist
• Assuming syntax error due to parentheses

5. You have a large DataFrame df used multiple times in your Spark job. You want to cache it but ensure it uses memory first and spills to disk if memory is full. Which persistence level should you use and why?

hard

A. Use df.persist(StorageLevel.DISK_ONLY) to avoid memory usage

B. Use df.cache() because it automatically spills to disk

C. Use df.unpersist() to keep data only in memory

D. Use df.persist(StorageLevel.MEMORY_AND_DISK) to store in memory and spill to disk if needed

Solution

1. Step 1: Identify persistence levels for memory and disk

   StorageLevel.MEMORY_AND_DISK stores data in memory and spills to disk if memory is insufficient.

2. Step 2: Compare with other options

   cache() only stores in memory and may fail if memory is full; DISK_ONLY avoids memory; unpersist() removes cached data.

3. Final Answer:

   Use df.persist(StorageLevel.MEMORY_AND_DISK) to store in memory and spill to disk if needed -> Option D

4. Quick Check:

   MEMORY_AND_DISK = memory first, then disk spill [OK]

Hint: MEMORY_AND_DISK spills to disk if memory full [OK]

Common Mistakes:

• Assuming cache spills to disk automatically
• Choosing DISK_ONLY to save memory but slower
• Using unpersist to keep data cached