Hadoop vs Spark: Key Differences and When to Use Each

Use Hadoop when you need reliable, large-scale batch processing with disk-based storage and fault tolerance. Choose Spark for faster, in-memory data processing, real-time analytics, and iterative machine learning tasks.

Quick Comparison

This table summarizes the main differences between Hadoop and Spark across key factors.

Factor	Hadoop	Spark
Processing Model	Batch processing with MapReduce	In-memory processing with DAG engine
Speed	Slower due to disk I/O	Faster due to in-memory computation
Fault Tolerance	High, uses data replication	High, uses lineage and RDDs
Use Cases	Large batch jobs, ETL	Real-time analytics, iterative ML
Ease of Use	Complex MapReduce code	Rich APIs in Java, Scala, Python
Resource Usage	Disk-heavy, slower	Memory-heavy, faster

Key Differences

Hadoop uses the MapReduce programming model which writes intermediate data to disk, making it reliable but slower. It is designed for batch processing of very large datasets where latency is not critical. Hadoop's storage layer, HDFS, replicates data across nodes to ensure fault tolerance.

Spark improves speed by keeping data in memory during processing, which is ideal for iterative algorithms like machine learning and real-time data analysis. It uses Resilient Distributed Datasets (RDDs) to recover lost data without heavy disk I/O. Spark also supports batch and streaming data, making it more versatile.

While Hadoop requires writing complex MapReduce jobs, Spark provides easy-to-use APIs in multiple languages, speeding up development. However, Spark needs more memory resources, whereas Hadoop can work efficiently with disk storage.

Code Comparison

java

import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

  public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
      String[] tokens = value.toString().split("\\s+");
      for (String token : tokens) {
        word.set(token);
        context.write(word, one);
      }
    }
  }

  public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
      int sum = 0;
      for (IntWritable val : values) {
        sum += val.get();
      }
      result.set(sum);
      context.write(key, result);
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

Output

word1	3
word2	5
word3	2
...

Spark Equivalent

python

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('WordCount').getOrCreate()

# Read text file
text_file = spark.read.text('input.txt')

# Split lines into words and count
counts = text_file.rdd.flatMap(lambda line: line[0].split()) \
    .map(lambda word: (word, 1)) \
    .reduceByKey(lambda a, b: a + b)

# Collect and print results
for word, count in counts.collect():
    print(f'{word}\t{count}')

spark.stop()

Output

word1	3
word2	5
word3	2
...

When to Use Which

Choose Hadoop when: you have very large batch jobs that can tolerate slower processing, need strong fault tolerance with disk-based storage, and want to use a mature ecosystem for ETL workflows.

Choose Spark when: you need fast, in-memory processing for real-time analytics, iterative machine learning, or interactive data exploration, and have enough memory resources to support it.

In summary, use Hadoop for heavy batch workloads and Spark for speed and flexibility in data processing.

Key Takeaways

Hadoop is best for large, reliable batch processing with disk storage.

Spark excels at fast, in-memory processing for real-time and iterative tasks.

Spark offers simpler APIs and supports multiple languages.

Hadoop uses MapReduce which is slower but highly fault tolerant.

Choose based on your workload size, speed needs, and resource availability.