Hadoop vs Spark comparison - Practice Questions

Challenge - 5 Problems

🧠 Conceptual

intermediate

Key difference in data processing models

Which option correctly describes the main difference between Hadoop MapReduce and Apache Spark in terms of data processing?

A Both Hadoop MapReduce and Spark process data only in real-time streaming mode.

B Spark uses disk storage for all intermediate data, whereas Hadoop MapReduce keeps data in memory throughout the process.

C Hadoop MapReduce processes data in batches using disk storage between steps, while Spark processes data in-memory for faster computation.

D Hadoop MapReduce is designed for in-memory processing, while Spark is optimized for batch processing on disk.

💡 Hint

Think about how each system handles intermediate data during processing.

Hadoop MapReduce writes intermediate data to disk between map and reduce steps, which slows down processing. Spark keeps data in memory, making it faster especially for iterative tasks.

🧠 Conceptual

intermediate

Fault tolerance mechanisms

Which statement best explains how fault tolerance is handled differently in Hadoop and Spark?

A Hadoop uses data replication on HDFS for fault tolerance, while Spark uses lineage information to recompute lost data.

B Spark replicates data blocks across nodes like Hadoop to ensure fault tolerance.

C Both Hadoop and Spark rely solely on checkpointing to handle faults.

D Hadoop does not support fault tolerance, but Spark does through data replication.

💡 Hint

Consider how each system recovers lost data after a failure.

Hadoop relies on replicating data blocks on HDFS to recover from failures. Spark tracks transformations (lineage) and can recompute lost data without replicating it.

❓ data_output

advanced

Comparing execution time for iterative tasks

Given the following scenario: A machine learning algorithm runs 10 iterations on a large dataset. Which framework will likely complete the task faster and why?

A Both will have similar speeds because they use the same storage system.

B Spark, because it caches data in memory across iterations, reducing disk I/O.

C Hadoop MapReduce, because it writes intermediate results to disk for each iteration.

D Hadoop MapReduce, because it uses in-memory caching by default.

💡 Hint

Think about how iterative algorithms benefit from in-memory data storage.

Spark caches data in memory, so iterative algorithms run faster by avoiding repeated disk reads/writes. Hadoop MapReduce writes intermediate data to disk each iteration, slowing down the process.

🧠 Conceptual

advanced

Resource management differences

Which option correctly describes how Hadoop and Spark manage cluster resources?

A Spark manages resources internally without any external cluster manager.

B Spark only runs on Hadoop YARN and cannot use other resource managers.

C Hadoop and Spark both use the same resource manager exclusively called Mesos.

D Hadoop uses YARN for resource management, while Spark can run on YARN, Mesos, or standalone cluster managers.

💡 Hint

Consider the flexibility of Spark in cluster environments.

Hadoop uses YARN as its default resource manager. Spark is more flexible and can run on YARN, Mesos, or its own standalone cluster manager.

🚀 Application

expert

Choosing the right framework for a real-time analytics project

You need to build a system that processes streaming data from sensors in near real-time and performs complex analytics. Which framework is best suited and why?

A Spark, because it supports real-time streaming with low latency and in-memory processing.

B Hadoop MapReduce, because it is optimized for real-time streaming data processing.

C Neither Spark nor Hadoop can handle streaming data; a traditional database is better.

D Hadoop MapReduce, because it processes data faster by using disk-based batch jobs.

💡 Hint

Think about which framework supports streaming and fast analytics.

Spark supports real-time streaming with Spark Streaming and Structured Streaming APIs, enabling low-latency analytics. Hadoop MapReduce is batch-oriented and not designed for real-time streaming.

Practice

1. Which of the following best describes the main difference between Hadoop and Spark?

easy

A. Spark requires more disk space than Hadoop.

B. Hadoop is only for small data; Spark is only for big data.

C. Hadoop uses disk storage for processing; Spark uses in-memory computing.

D. Hadoop is faster than Spark for all tasks.

Solution

1. Step 1: Understand Hadoop's processing method

   Hadoop processes data using MapReduce which reads and writes data to disk between steps.

2. Step 2: Understand Spark's processing method

   Spark processes data in memory, which makes it faster for many tasks.

3. Final Answer:

   Hadoop uses disk storage for processing; Spark uses in-memory computing. -> Option C

4. Quick Check:

   Disk vs memory processing = A [OK]

Hint: Remember: Hadoop = disk, Spark = memory speed [OK]

Common Mistakes:

• Thinking Spark always uses more disk space
• Believing Hadoop is faster for all tasks
• Confusing data size limits for each tool

2. Which syntax correctly describes how Spark handles data compared to Hadoop?

easy

A. Spark.read().cache() stores data in memory; Hadoop writes to disk.

B. Hadoop.cache() stores data in memory; Spark writes to disk.

C. Spark.write().disk() stores data on disk; Hadoop caches in memory.

D. Hadoop.read().cache() stores data in memory; Spark writes to disk.

Solution

1. Step 1: Identify Spark's caching method

   Spark uses cache() to keep data in memory for faster access.

2. Step 2: Identify Hadoop's data handling

   Hadoop writes intermediate data to disk during MapReduce jobs.

3. Final Answer:

   Spark.read().cache() stores data in memory; Hadoop writes to disk. -> Option A

4. Quick Check:

   Spark cache = memory, Hadoop = disk [OK]

Hint: Spark.cache() means in-memory storage [OK]

Common Mistakes:

• Confusing which tool uses cache()
• Assuming Hadoop has cache() method
• Mixing read and write operations

3. Given the following code snippets, which one will likely run faster for iterative machine learning tasks?

medium

A. Spark job caching data in memory and reusing it across iterations.

B. Hadoop MapReduce job reading and writing data to disk each iteration.

C. Hadoop job using in-memory caching for iterations.

D. Spark job writing intermediate results to disk every iteration.

Solution

1. Step 1: Analyze Hadoop's iterative processing

   Hadoop MapReduce writes data to disk after each iteration, slowing down iterative tasks.

2. Step 2: Analyze Spark's iterative processing

   Spark caches data in memory, avoiding repeated disk I/O, speeding up iterations.

3. Final Answer:

   Spark job caching data in memory and reusing it across iterations. -> Option A

4. Quick Check:

   In-memory caching speeds iterations = B [OK]

Hint: Iterative tasks run faster with Spark's memory caching [OK]

Common Mistakes:

• Assuming Hadoop can cache in memory like Spark
• Thinking disk writes are faster than memory
• Confusing Spark's disk writes with Hadoop's

4. Identify the error in this Spark code snippet meant to cache data for faster processing:

val data = spark.read.text("file.txt")
data.cache
val count = data.count()

medium

A. Using read.text instead of read.csv.

B. Missing parentheses after cache method call.

C. Calling count before caching data.

D. No error; code is correct.

Solution

1. Step 1: Check method call syntax

   The cache method requires parentheses: cache().

2. Step 2: Verify method order

   Calling count() after caching is correct to trigger caching.

3. Final Answer:

   Missing parentheses after cache method call. -> Option B

4. Quick Check:

   cache() needs () to execute [OK]

Hint: Remember: methods need () to run in Scala [OK]

Common Mistakes:

• Ignoring missing parentheses on cache
• Thinking read.text is wrong for text files
• Believing count must come before cache

5. You have a large batch job that processes petabytes of data once a day and requires fault tolerance. Which tool is better and why?

hard

A. Hadoop, because it cannot handle large data but is faster.

B. Spark, because in-memory processing is always better for large batch jobs.

C. Spark, because it does not support fault tolerance but is faster.

D. Hadoop, because it writes intermediate data to disk ensuring fault tolerance.

Solution

1. Step 1: Understand batch job needs

   Large batch jobs need fault tolerance to recover from failures without losing data.

2. Step 2: Compare Hadoop and Spark fault tolerance

   Hadoop writes intermediate data to disk, which helps recover from failures. Spark relies on memory and lineage but may lose data if memory fails.

3. Final Answer:

   Hadoop, because it writes intermediate data to disk ensuring fault tolerance. -> Option D

4. Quick Check:

   Disk storage = fault tolerance = Hadoop [OK]

Hint: For big, stable batch jobs, choose Hadoop [OK]

Common Mistakes:

• Assuming Spark is always better for big data
• Ignoring fault tolerance importance
• Confusing speed with reliability