Caching and persistence in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is caching in Apache Spark?

Caching in Apache Spark means storing a dataset in memory to speed up repeated access. It helps avoid recomputing the dataset every time it is used.

beginner

What does persistence mean in Spark?

Persistence means saving a dataset to memory and/or disk with different storage levels. It allows Spark to reuse the dataset efficiently across multiple operations.

intermediate

Name two common storage levels used in Spark persistence.

Two common storage levels are MEMORY_ONLY (store data only in memory) and MEMORY_AND_DISK (store data in memory, spill to disk if not enough memory).

intermediate

How does caching improve performance in iterative algorithms?

Caching keeps the data in memory so iterative algorithms can reuse it without recomputing or reading from disk each time, making the process faster.

intermediate

What is the difference between cache() and persist() in Spark?

cache() is shorthand for persist() with the default storage level MEMORY_AND_DISK. persist() lets you choose other storage levels like DISK_ONLY or MEMORY_ONLY_SER.

What happens when you call cache() on a DataFrame in Spark?

A The DataFrame is stored in memory for faster access.

B The DataFrame is deleted from memory.

C The DataFrame is saved permanently to disk.

D The DataFrame is converted to a different format.

cache() stores the DataFrame in memory to speed up future operations.

Which storage level stores data only on disk in Spark persistence?

A MEMORY_ONLY

B OFF_HEAP

C MEMORY_AND_DISK

D DISK_ONLY

DISK_ONLY stores the dataset only on disk, not in memory.

Why would you use persist() instead of cache()?

A To convert the dataset to JSON format.

B To delete the dataset from memory.

C To choose a specific storage level other than the default.

D To speed up the first computation only.

persist() allows selecting different storage levels like MEMORY_ONLY or DISK_ONLY.

What is a benefit of caching in iterative machine learning algorithms?

A It increases disk usage.

B It reduces repeated data loading and computation.

C It slows down the algorithm.

D It deletes intermediate results.

Caching avoids recomputing data in each iteration, speeding up the process.

If memory is limited, which storage level helps avoid out-of-memory errors?

A MEMORY_AND_DISK

B MEMORY_ONLY

C DISK_ONLY

D NONE

MEMORY_AND_DISK stores data in memory and spills to disk if memory is full, preventing errors.

Explain caching and persistence in Apache Spark and how they help improve performance.

Think about how storing data in memory or disk helps avoid recomputation.

Describe the difference between cache() and persist() methods in Spark with examples of when to use each.

Consider how you might want to store data differently depending on memory availability.