Hadoop vs Spark

Five questions, the same data

Apache Spark, and how it relates to Hadoop

Words you'll meet in this lesson

From our lab (Spark 4.0.4 on our YARN cluster, 3 executors of 512–768 MB, 1 core each)

Spark vs MapReduce at three levels

MapReduce and Spark side by side

Architecture at a glance: one cluster, two engines

Left: a MapReduce job (Lesson 8.6). Right: a Spark application on the same YARN and HDFS. The yellow boxes are what each engine writes to HDFS between steps.

Animated example: a fraud model that needs ten passes

The same job three ways, on our cluster

We unpacked spark-4.0.4-bin-hadoop3 in the namenode container and pointed it at our Hadoop configuration. No Spark daemons were started: YARN starts everything per application.

1. Submit to YARN

   export SPARK_HOME=/tmp/opt/spark-4.0.4-bin-hadoop3 HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
   $SPARK_HOME/bin/spark-submit --master yarn --num-executors 3 --executor-memory 512m \
     --executor-cores 1 --driver-memory 512m item_revenue_spark.py /user/hadoop/data/canteen_orders.csv

   spark-submit wall time 46 s (includes starting the YARN application and 3 executors)
   --master yarn makes the client ask our ResourceManager for an application, exactly like hadoop jar. The driver runs here (client mode); the AM and 3 executors run in containers on the workers.

2. The RDD way: MapReduce thinking

   .map(lambda line: line.split(","))
   .filter(lambda f: len(f) == 3 and f[0] != "order_id")
   .map(lambda f: (f[1], (1, int(f[2]))))                  # map: (item, (1, amount))
   .reduceByKey(lambda a, b: (a[0] + b[0], a[1] + b[1]))   # combine + shuffle + reduce

   RDD chai 50170 752550 RDD dosa 49948 2247660 … RDD job took 7.4 s, partitions: 2
   The same mapper and reducer as ItemRevenue.java, in four lines, with the same answers. Two partitions = two input splits of the 4.7 MB file.

3. The DataFrame way: SQL thinking

   df = spark.read.csv(path, header=True, inferSchema=True)
   df.groupBy("item").agg(F.count("*").alias("orders"), F.sum("amount").alias("revenue")).orderBy("item")

   | chai| 50170| 752550| | dosa| 49948|2247660| DataFrame job took 11.3 s
   You say what, Catalyst decides how. It took longer here because inferSchema reads the file once just to guess types; with a declared schema it reads once.

4. Read the plan: find MapReduce inside

   Sort [item ASC]
   +- Exchange rangepartitioning(item, 200)
      +- HashAggregate(keys=[item], functions=[count(1), sum(amount)])          ← reduce
         +- Exchange hashpartitioning(item, 200)                               ← shuffle
            +- HashAggregate(keys=[item], functions=[partial_count(1), partial_sum(amount)])   ← combiner
               +- FileScan csv [item, amount] ReadSchema: struct<item:string,amount:int>   ← map + column pruning

   Bottom to top: read only two columns, partially aggregate inside each map task (the combiner of Lesson 9.1), shuffle by hash of item (Lesson 9.2), finish the aggregate, then a second shuffle to sort globally.

5. Five passes: no cache vs cache

   no cache: 19.8, 15.6, 13.5, 13.6, 12.8 s  = 75.3 s
   cache   : 21.4,  4.2,  3.7,  3.3,  2.7 s  = 35.3 s

   Without cache Spark re-reads HDFS every pass, but executors stay alive, so it is already far faster than starting five jobs. With persist(MEMORY_AND_DISK) pass 1 costs a bit more (it fills the cache); later passes skip HDFS completely.

6. The MapReduce way: five jobs

   for k in 1 2 3 4 5; do hadoop jar mr.jar ItemRevenue orders_x30.csv /out/iter$k; done

   MapReduce pass 1: 39 s … pass 5: 41 s MapReduce: 5 passes took 199 s
   Each job starts an ApplicationMaster and new map and reduce JVMs, reads 140 MB from HDFS and writes its result with replication 3. Nothing is reused between jobs.

Spark on our YARN cluster

Six steps of our real runs: submit, the RDD job, the plan, five uncached passes, five cached passes, and five MapReduce jobs.

Spark on Hadoop at work

Versions as of October 2026: Spark 4.2 (July 2026), 4.1 and 4.0 are current lines and 3.5 is the long-term line with security fixes until November 2027. Managed platforms often lag: Amazon EMR 7.14 ships Spark 3.5.8, Google Dataproc 2.3.37 ships Spark 3.5.3 and Cloudera Base 7.3.2 ships Spark 3.5 (sources below).

Simplified; organisations differ in tools and policies. Sources: endoflife.date/apache-spark, Amazon EMR 7.x versions, Dataproc 2.3 versions, Cloudera Base 7.3.2.

A model of k passes, fitted to our runs

The per-pass times below are our measurements. The model predicts any number of passes; check that k = 5 gives back the lab totals (199 s, 75.3 s, 35.3 s). It leaves out Spark's one-time application start-up (about 46 s in our lab), which MapReduce pays inside every pass.

# Per-pass times measured on our lab (5 passes over 9 M orders, 140 MB)
MR_PASS = 199 / 5                     # every pass is a new MapReduce job: start + read + write
NOCACHE_FIRST, NOCACHE_NEXT = 19.8, (15.6 + 13.5 + 13.6 + 12.8) / 4
CACHE_FIRST, CACHE_NEXT = 21.4, (4.2 + 3.7 + 3.3 + 2.7) / 4

def mapreduce(k):
    return k * MR_PASS

def spark(k, cache):
    if cache:
        return CACHE_FIRST + (k - 1) * CACHE_NEXT
    return NOCACHE_FIRST + (k - 1) * NOCACHE_NEXT

for k in [1, 5, 10, 20]:
    mr, nc, c = mapreduce(k), spark(k, False), spark(k, True)
    print(f"{k:>2} passes: MapReduce {mr:6.1f} s | Spark {nc:6.1f} s | Spark+cache {c:5.1f} s | {mr / c:4.1f}x")

• MR_PASS = 199 / 5 # every pass is a new MapReduce job: start + read + writeFive ItemRevenue jobs took 39, 39, 39, 41 and 41 s: about 40 s each, every time.
• CACHE_FIRST, CACHE_NEXT = 21.4, (4.2 + 3.7 + 3.3 + 2.7) / 4Pass 1 pays for reading and filling the cache; the next passes cost about 3.5 s.
• return CACHE_FIRST + (k - 1) * CACHE_NEXTThe cached line grows slowly, so the advantage grows with every extra pass.

Run it on your PC: Spark on the Module 8 cluster

Spark needs no daemons of its own here: unpack it on the namenode container, point HADOOP_CONF_DIR at the Hadoop configuration, and spark-submit --master yarn asks our ResourceManager for an ApplicationMaster and executors, just like hadoop jar. Executors of 512–768 MB fit our 2 GB NodeManagers. The outputs below are real; Spark's INFO/WARN log lines and the YARN status report are filtered out.

# item_revenue_spark.py: the same job as ItemRevenue.java (Lesson 8.1), in PySpark, twice:
# with the RDD API (map / reduceByKey, MapReduce thinking) and with DataFrames (SQL thinking).
import sys, time
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("item revenue (spark)").getOrCreate()
path = sys.argv[1]

t = time.time()
rdd = (spark.sparkContext.textFile(path)
       .map(lambda line: line.split(","))
       .filter(lambda f: len(f) == 3 and f[0] != "order_id")
       .map(lambda f: (f[1], (1, int(f[2]))))                       # map: (item, (1, amount))
       .reduceByKey(lambda a, b: (a[0] + b[0], a[1] + b[1])))       # shuffle + reduce (with map-side combine)
for item, (n, total) in sorted(rdd.collect()):
    print(f"RDD  {item:9} {n:>8} {total:>10}")
print(f"RDD job took {time.time() - t:.1f} s, partitions: {rdd.getNumPartitions()}")

t = time.time()
df = spark.read.csv(path, header=True, inferSchema=True)
out = df.groupBy("item").agg(F.count("*").alias("orders"), F.sum("amount").alias("revenue")).orderBy("item")
out.show()
print(f"DataFrame job took {time.time() - t:.1f} s")
out.explain()
spark.stop()

# One-time setup on the namenode container of the Module 8 lab (no Spark daemons needed):
#   curl -O https://dlcdn.apache.org/spark/spark-4.0.4/spark-4.0.4-bin-hadoop3.tgz
#   mkdir -p /tmp/opt && tar xzf spark-4.0.4-bin-hadoop3.tgz -C /tmp/opt
#   export SPARK_HOME=/tmp/opt/spark-4.0.4-bin-hadoop3 HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
#   $SPARK_HOME/bin/spark-submit --master yarn --num-executors 3 --executor-memory 512m \
#       --executor-cores 1 --driver-memory 512m item_revenue_spark.py /user/hadoop/data/canteen_orders.csv

# iterative_spark.py: five passes over the same 140 MB (9 million orders).
# Pass k counts orders per item with amount >= threshold k. With cache(), only pass 1 reads HDFS.
import sys, time
from pyspark import StorageLevel
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("five passes (spark)").getOrCreate()
cache = sys.argv[2] == "cache"
orders = (spark.sparkContext.textFile(sys.argv[1])
          .map(lambda l: l.split(","))
          .filter(lambda f: f[0] != "order_id")
          .map(lambda f: (f[1], int(f[2]))))
if cache:
    orders = orders.persist(StorageLevel.MEMORY_AND_DISK)
total = time.time()
for k, threshold in enumerate([0, 18, 20, 25, 30], 1):
    t = time.time()
    counts = orders.filter(lambda p, th=threshold: p[1] >= th).map(lambda p: (p[0], 1)).reduceByKey(lambda a, b: a + b).collect()
    print(f"pass {k}: amount >= {threshold:2}: {sum(c for _, c in counts):>9,} orders, {time.time() - t:5.1f} s")
print(f"cache={cache}: 5 passes took {time.time() - total:.1f} s")
spark.stop()

# One-time setup on the namenode container of the Module 8 lab (no Spark daemons needed):
#   curl -O https://dlcdn.apache.org/spark/spark-4.0.4/spark-4.0.4-bin-hadoop3.tgz
#   mkdir -p /tmp/opt && tar xzf spark-4.0.4-bin-hadoop3.tgz -C /tmp/opt
#   export SPARK_HOME=/tmp/opt/spark-4.0.4-bin-hadoop3 HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
#   for c in nocache cache; do
#     $SPARK_HOME/bin/spark-submit --master yarn --num-executors 3 --executor-memory 768m \
#       --executor-cores 1 --driver-memory 512m iterative_spark.py /user/hadoop/data/orders_x30.csv $c
#   done

# iterative_mr.sh: the MapReduce way - every pass is a new job that reads HDFS again
cd /tmp/mr; total=$(date +%s)
for k in 1 2 3 4 5; do
  s=$(date +%s); hdfs dfs -rm -r -f -skipTrash /out/iter$k > /dev/null 2>&1
  hadoop jar mr.jar ItemRevenue /user/hadoop/data/orders_x30.csv /out/iter$k > /dev/null 2>&1
  echo "MapReduce pass $k: $(( $(date +%s) - s )) s"
done
echo "MapReduce: 5 passes took $(( $(date +%s) - total )) s"

# Run inside the namenode container:  bash iterative_mr.sh

Lazy transformations, one action

Spark records transformations and runs them only when an action asks for a result. Watch the plan grow without any data moving, then the action run it all at once.

Worked examples: real problems, solved and traced in memory

Four decisions engineers face when choosing between MapReduce and Spark, worked with our measured numbers.

MR 39.8 s/pass · cache 21.4 s + 3.475 s/pass · start-up 46 s

MapReduce 398 s, Spark+cache 52.7 s, with start-up 98.7 s (4.0x)

2 TB CSV · 100 × 16 GB executors · ratio 4.4/19.4

memory 942 GB, data cached ~464 GB, fits: True

3 workloads

a: one-pass 50 TB copy       -> MapReduce/DistCp is fine
b: dashboard every minute    -> Spark Structured Streaming
c: 30-pass model             -> Spark + cache

400 jobs · 2 days × ₹8,000 · 6,000 container-hours/night · −55% · ₹4/hour

rewrite Rs 6,400,000, saves Rs 13,200 a night, payback 485 nights

Check yourself

Quick revision

Interview and viva questions

Practice problems

Exercises using the real MapReduce and Spark numbers from our cluster.

199 s · 75.3 s · 35.3 s

vs MapReduce 5.6x, vs uncached Spark 2.1x

print(f"vs MapReduce {199 / 35.3:.1f}x, vs uncached Spark {75.3 / 35.3:.1f}x")

21.4 s of 35.3 s

61%

print(f"{21.4 / 35.3:.0%}")

50170 orders, 752550

True

print(752550 == 50170 * 15)

21.4 vs 19.8 · 3.475 vs 13.875

1 passes: cache 21.4 no cache 19.8
2 passes: cache 24.9 no cache 33.7
3 passes: cache 28.3 no cache 47.5

for k in range(1, 4):
    print(k, "passes: cache", round(21.4 + (k - 1) * 3.475, 1), "no cache", round(19.8 + (k - 1) * 13.875, 1))

3 lines of feedback

[('chai', 2), ('cold', 1), ('dosa', 1), ('hot', 1), ('late', 1), ('was', 3)]

lines = ["chai was hot", "dosa was cold", "chai was late"]
words = [w for line in lines for w in line.split()]       # flatMap
pairs = [(w, 1) for w in words]                           # map
counts = {}
for w, n in pairs:                                        # reduceByKey
    counts[w] = counts.get(w, 0) + n
print(sorted(counts.items()))

140,759,640 and 4,692,009 bytes

140759640 -> 2 partitions
4692009 -> 2 partitions

import math
for size in [140_759_640, 4_692_009]:
    print(size, "->", max(2, math.ceil(size / (128 * 1024 * 1024))), "partitions")

120 jobs × 10 passes

10.0 hours of job time saved per day

mr = 10 * 39.8
spark = 21.4 + 9 * 3.475 + 46
print(f"{120 * (mr - spark) / 3600:.1f} hours of job time saved per day")

3 partitions, lineage = filter ≥ 18 then double

recomputed 1 partition; total same: True

source = [[15, 18, 20], [25, 30, 45], [18, 80, 15]]
lineage = lambda part: [x * 2 for x in part if x >= 18]
cache = [lineage(p) for p in source]
before = sum(map(sum, cache))
cache[1] = None                              # executor lost
recomputed = 0
for i, p in enumerate(cache):
    if p is None:
        cache[i] = lineage(source[i]); recomputed += 1
print("recomputed", recomputed, "partition; total same:", sum(map(sum, cache)) == before)

Real-world scenario questions (asked in company interviews)

Hadoop vs Spark is asked in almost every big data interview, often as a trap: the best answers start by saying Spark usually runs on Hadoop.