Hadoop 1, 2 and 3: what changed

Three generations in one office

Hadoop 1, 2 and 3

Words you'll meet in this lesson

Release facts

Dates from the Apache Hadoop release pages. Check each release's notes for its supported Java versions before installing.

The versions at three levels

Hadoop 1 vs 2 vs 3 at a glance

Scale limits are the commonly quoted design figures; real clusters vary with hardware and workload.

Port numbers that changed in Hadoop 3

Why Hadoop 1's JobTracker hit a wall

1. One process, every decision

   In Hadoop 1, every TaskTracker heartbeats to the JobTracker, and the JobTracker decides every task placement for every running job.

   It also keeps the progress, counters and history of all jobs in its own memory. The more nodes and jobs, the more work for this single process.

2. Count the load

   nodes, hb_interval_s = 4000, 3
   tasks_running = 40000
   print(nodes // hb_interval_s, "heartbeats per second, tracking", tasks_running, "tasks")

   1333 heartbeats per second, tracking 40000 tasks
   Around 4,000 nodes and 40,000 concurrent tasks is where Hadoop 1 clusters typically topped out. Beyond that the JobTracker became slow or unstable.

3. Slots waste resources

   Each node had fixed map slots and reduce slots. During a map-heavy phase, reduce slots sat idle even though maps were waiting.

   Utilisation suffered, and an admin had to guess the right map/reduce split for every node.

4. Hadoop 2 splits the JobTracker

   The ResourceManager only tracks free resources and hands out containers. Each job's ApplicationMaster (running in an ordinary container on a worker) handles its own tasks.

   The heavy, per-job work is now spread across the cluster. The ResourceManager stays small and fast, so clusters can grow much larger.

5. Any engine can join

   Spark, Tez and others ship their own ApplicationMaster. YARN doesn't care what runs inside a container.

   This is why “Spark on YARN” is so common: one cluster, one storage layer, many engines, shared fairly through queues.

The same two jobs on Hadoop 1 and on Hadoop 2

A map-heavy job and a Spark job arrive at a 3-node cluster. Left: Hadoop 1 with a JobTracker and slots. Right: Hadoop 2 with YARN and containers.

Try it yourself: Hadoop 2's two big fixes

This cluster runs Hadoop 2+ features: jobs run through the ResourceManager, and the NameNode has a standby. Try the failure that would have stopped a Hadoop 1 cluster.

Version changes in real projects

Examples are simplified descriptions of common practice.

Slots vs containers: measure the waste

In Hadoop 1, each node has fixed map slots and reduce slots. This program fills them with a map-heavy workload and measures how much of the cluster is idle, then does the same with YARN-style containers that can run either kind of task.

nodes = 10
map_slots, reduce_slots = 8, 4          # Hadoop 1: fixed per node
containers = map_slots + reduce_slots    # Hadoop 2: same capacity, any task type

maps_waiting, reduces_waiting = 200, 10

h1_used = min(maps_waiting, nodes * map_slots) + min(reduces_waiting, nodes * reduce_slots)
h2_used = min(maps_waiting + reduces_waiting, nodes * containers)

total = nodes * containers
print(f"Hadoop 1: {h1_used} of {total} busy ({h1_used / total:.0%})")
print(f"Hadoop 2: {h2_used} of {total} busy ({h2_used / total:.0%})")

• map_slots, reduce_slots = 8, 4In Hadoop 1, an admin fixed these numbers per node. A reduce slot could never run a map task.
• h1_used = min(...) + min(...)Map tasks can only use map slots and reduce tasks only reduce slots, so each is capped separately.
• h2_used = min(maps_waiting + reduces_waiting, nodes * containers)With containers, any waiting task can use any free capacity.

Placing tasks in slots vs containers, one line at a time

Watch the same five tasks being placed on one node, first with Hadoop 1 slots (2 map + 2 reduce) and then with 4 YARN containers. Notice which tasks are left waiting.

Worked examples: real problems, solved and traced in memory

Three problems from real version decisions: an upgrade checklist, the storage savings of Hadoop 3, and how many NameNodes a cluster needs. Each runs live and is traced in memory.

java 7 · monitored ports [50070, 8088, 50075] · jobs [("daily_sales", "mapred"), ("fraud", "mapreduce")]

Upgrade Java: Hadoop 3 needs Java 8 or later
Monitoring: change port 50070 to 9870
Monitoring: change port 50075 to 9864
Review job 'daily_sales': uses the old mapred API

1.2 PB · 70% cold · 3× → 1.5× · 144 TB per node

Before: 3600 TB raw, after: 2340 TB raw
Freed: 1260 TB = about 8 nodes

targets: survive 1 failure (Hadoop 2 style) and 2 failures (Hadoop 3 style)

Survive 1: 2 NameNodes + 3 JournalNodes (Hadoop 2 or 3)
Survive 2: 3 NameNodes + 5 JournalNodes (Hadoop 3 only)

Check yourself

Quick revision

Interview and viva questions

Practice problems

Questions that test version knowledge with small programs: slots, ports, storage schemes and HA sizing.

50 nodes · 6 map + 3 reduce

map: 300 | reduce: 150

print("map:", 50 * 6, "| reduce:", 50 * 3)

RS-6-3 · RS-3-2

RS-6-3: 1.50x
RS-3-2: 1.67x

for d, p in [(6, 3), (3, 2)]:
    print(f"RS-{d}-{p}: {(d + p) / d:.2f}x")

{"namenode-ui": 9870, "rm-ui": 8088, "datanode-ui": 9864}

Open http://localhost:9870

ports = {"namenode-ui": 9870, "rm-ui": 8088, "datanode-ui": 9864}
print("Open http://localhost:" + str(ports["namenode-ui"]))

400 map + 200 reduce slots · 1000 maps waiting

Hadoop 1 idle: 33%
Hadoop 2 idle: 0%

total = 400 + 200
used_h1 = min(1000, 400)
used_h2 = min(1000, total)
print(f"Hadoop 1 idle: {(total - used_h1) / total:.0%}")
print(f"Hadoop 2 idle: {(total - used_h2) / total:.0%}")

[3, 4, 5, 7]

3 JournalNodes survive 1 failure(s)
4 JournalNodes survive 1 failure(s)
5 JournalNodes survive 2 failure(s)
7 JournalNodes survive 3 failure(s)

for j in [3, 4, 5, 7]:
    print(j, "JournalNodes survive", (j - 1) // 2, "failure(s)")

["YARN", "erasure coding", "NameNode HA", "GPU scheduling", "Federation"]

['erasure coding', 'GPU scheduling']

hadoop3_only = {"erasure coding", "GPU scheduling", "multiple standby NameNodes"}
features = ["YARN", "erasure coding", "NameNode HA", "GPU scheduling", "Federation"]
print([f for f in features if f in hadoop3_only])

2,048 TB raw · 60% cold · node 144 TB = Rs 18 lakh

Freed 614 TB raw = 4 nodes = Rs 72 lakh avoided

import math
raw = 2048
data = raw / 3
cold = data * 0.60
freed = cold * 3 - cold * 1.5
nodes = math.floor(freed / 144)
print(f"Freed {freed:.0f} TB raw = {nodes} nodes = Rs {nodes * 18} lakh avoided")

peaks 30 and 20 at different times · quiet share 6 · +20%

Nodes needed: 44 instead of 70

import math
need = max(30, 20) + 6
print("Nodes needed:", math.ceil(need * 1.2), "instead of", 40 + 30)

Real-world scenario questions (asked in company interviews)

Version questions show up in almost every Hadoop interview. These scenarios turn the comparison table into practical answers.