Two research papers and a toy
Arjun Sir puts two old printouts on the table. One says The Google File System, 2003. The other says MapReduce: Simplified Data Processing on Large Clusters, 2004. “These twenty-odd pages changed the whole data industry,” he says. “Google didn't share its code. It shared its ideas. And two people building a free search engine read them very carefully.”
Sir, why would Google give away its secrets?
Research papers are how engineers share ideas and get credit. The papers explained the design, but not Google's actual code. Anyone who wanted it had to build it themselves.
And who did?
Doug Cutting and Mike Cafarella. They were building Nutch, an open-source web search engine, and had hit exactly the scaling wall Google described. They wrote their own versions: a distributed file system and a MapReduce engine. In 2006 that part was taken out of Nutch and given its own name: Hadoop, after a toy elephant that belonged to Cutting's young son.
A toy! And now banks run on it.
Yahoo! hired Cutting and put a large team on it, because they needed it for their own search. Within two years it was sorting a terabyte faster than anything else in the world. Today you'll follow that story year by year, and see which ideas from the papers became which parts of Hadoop.
Where Hadoop came from
Apache Hadoop is an open-source implementation of the distributed storage and processing designs published by Google in “The Google File System” (Ghemawat, Gobioff and Leung, SOSP, October 2003) and “MapReduce: Simplified Data Processing on Large Clusters” (Dean and Ghemawat, OSDI, December 2004). It was created by Doug Cutting and Mike Cafarella within the Apache Nutch web-search project, separated from Nutch as its own project in 2006, developed at scale largely at Yahoo!, and became a top-level Apache Software Foundation project in January 2008. HDFS corresponds to GFS and Hadoop MapReduce to Google's MapReduce.
Google explained how it stored and processed the web in two famous papers, but kept its code. Two open-source developers rebuilt those ideas for their own search engine. That code became Hadoop, Yahoo! helped grow it, and the Apache Software Foundation made it free for everyone.
Words you'll meet in this lesson
The key dates
| 2002 | Doug Cutting and Mike Cafarella start Apache Nutch, an open-source web search engine |
|---|---|
| Oct 2003 | Google publishes “The Google File System” paper |
| 2004 | Nutch developers start NDFS, an open-source distributed file system based on the GFS design |
| Dec 2004 | Google publishes the MapReduce paper; Nutch adds a MapReduce implementation in 2005 |
| 2006 | Doug Cutting joins Yahoo!; the storage and processing code leaves Nutch and becomes Hadoop, a sub-project of Apache Lucene |
| Jan 2008 | Hadoop becomes a top-level Apache project |
| 2008 | A 910-node Yahoo! Hadoop cluster sorts 1 TB in 209 seconds, winning the terabyte sort benchmark |
| 27 Dec 2011 | Hadoop 1.0.0 released |
| 15 Oct 2013 | Hadoop 2.2.0: first general-availability release of 2.x, with YARN |
| Dec 2017 | Hadoop 3.0.0 general availability (erasure coding and more) |
| 2 Apr 2026 | Hadoop 3.5.0, the first stable release of the 3.5 line |
Dates are from the Apache Hadoop release pages and the published papers. Some early milestones are given only to the year because sources differ by a few months.
The history at three levels
Beginner The recipe and the restaurant
- A famous restaurant (Google) published the recipe for its best dish, but not its kitchen.
- A small team (Cutting and Cafarella) cooked the recipe in their own kitchen and shared their kitchen with everyone for free.
- A big company (Yahoo!) paid many cooks to improve that free kitchen, because it needed to feed huge crowds.
- Soon thousands of companies were using the free kitchen, and new dishes (Hive, Pig, HBase) were added on top.
Intermediate Paper idea → Hadoop part
- GFS master → HDFS NameNode; GFS chunkservers → DataNodes; 64 MB chunks → HDFS blocks (64 MB in early Hadoop, 128 MB since Hadoop 2).
- GFS kept 3 replicas by default, placed across racks → HDFS does the same.
- MapReduce master → Hadoop 1's JobTracker; workers → TaskTrackers (replaced by YARN in Hadoop 2).
- The paper's map, shuffle, reduce, combiner, partitioner, re-execution of failed tasks, backup tasks for stragglers all appear in Hadoop MapReduce.
- Bigtable (2006 paper) → HBase.
Advanced Why the open-source route won
- Shared cost: Yahoo!, Facebook, LinkedIn, Twitter and others all improved the same code instead of each building its own.
- An ecosystem grew: Hive (from Facebook) added SQL; Pig (from Yahoo!) added a scripting language; HBase, ZooKeeper, Oozie, Sqoop, Flume and later Spark joined.
- A business grew: companies such as Cloudera (2008), MapR (2009) and Hortonworks (2011, spun out of Yahoo!'s Hadoop team) sold supported distributions. Cloudera and Hortonworks merged in 2019.
- Apache governance: no single company owns the project, so competitors could collaborate safely.
- The pattern repeated: Google's later papers and systems inspired more open-source projects, and Hadoop's own limits inspired Spark (from UC Berkeley).
Exam tip: For “Write a short note on the history of Hadoop”, give: Nutch (2002) → GFS paper (2003) and MapReduce paper (2004) → NDFS and MapReduce in Nutch → Hadoop (2006) with Yahoo! → top-level Apache project (2008) → 1.0 (2011), 2.x with YARN (2013), 3.0 (2017). Name Doug Cutting and Mike Cafarella, and say HDFS ≈ GFS, Hadoop MapReduce ≈ Google MapReduce.
Google's systems and their open-source cousins
| Google system (paper) | What it does | Open-source counterpart | Started at |
|---|---|---|---|
| GFS (2003) | Distributed file system for huge files | HDFS | Nutch / Yahoo! |
| MapReduce (2004) | Parallel batch processing | Hadoop MapReduce | Nutch / Yahoo! |
| Bigtable (2006) | Distributed, sorted key–value tables | HBase | Powerset, then the Hadoop community |
| Chubby (2006) | Lock and coordination service | ZooKeeper (similar purpose) | Yahoo! |
Projects that grew around Hadoop
Hive
Lets analysts query HDFS data with SQL (HiveQL). Created at Facebook, where many analysts knew SQL but not Java.
SQL on HadoopPig
A scripting language (Pig Latin) for data flows, created at Yahoo!. Much less used today.
ScriptsHBase
Bigtable-style database on top of HDFS for fast reads and writes of individual rows.
NoSQLZooKeeper
Coordination service used for leader election and configuration, for example NameNode failover.
CoordinationSpark
A faster, in-memory processing engine from UC Berkeley that often runs on YARN and reads HDFS. Now the most common engine for new work.
Successor engineThe name “Hadoop” has no meaning; it was a child's name for a toy. Doug Cutting liked that it was short, easy to say and not already used. Many ecosystem projects have similarly playful names.
How an idea in a paper became software on thousands of clusters
2002–03: Nutch hits the wall
Nutch could crawl and index millions of pages on a few machines, but the whole web needed far more, and running it across machines was painful.
Exactly the BharatSearch problem from Lesson 2.1: data too big, manual sharding, failures.2003–04: read the papers, build NDFS
The GFS paper described a design for exactly this. The Nutch team implemented it as NDFS.
Key ideas copied: one master for metadata, many chunk servers, big chunks, replication across racks, the master never in the data path.2005: add MapReduce
After the MapReduce paper, Nutch rewrote its main algorithms (crawling, link analysis, indexing) as map and reduce jobs running on NDFS.
Now storage and processing were general-purpose, not search-specific. That is why they could leave Nutch.2006: Hadoop is born, Yahoo! commits
The code moves into its own project called Hadoop. Doug Cutting joins Yahoo!, which assigns engineers to make it run on thousands of machines.
Yahoo! needed it for search; the open-source licence meant every improvement was public.2008: proven at scale
Hadoop becomes a top-level Apache project, and a 910-node cluster wins the terabyte sort benchmark in 209 seconds.
The record convinced many companies that free software on commodity hardware could beat expensive systems.2011 onwards: releases and an ecosystem
Hadoop 1.0 (2011), Hadoop 2 with YARN (2013), Hadoop 3 (2017), and a large ecosystem of tools and companies.
Lesson 2.4 covers what changed in each version and why.
From the GFS design to HDFS: what the paper drew, and what Hadoop built
Step through the main ideas of the GFS and MapReduce papers. Left: the idea as the paper describes it. Right: the matching part of Hadoop.
- GFS: a single master keeps the file-system metadata: names and where each chunk lives.
- Files are cut into large chunks (64 MB) stored on many chunkservers.
- Each chunk is replicated (3 copies by default), with copies on different racks.
- MapReduce: programmers write a map and a reduce function; the system runs them on thousands of machines.
- The master detects failed workers and re-executes their tasks, and runs backup tasks for slow ones.
- Locality: the master tries to schedule map tasks where a replica of their input already is.
- Hadoop's NameNode plays the GFS master's role: it holds the namespace and the location of every block.
- DataNodes store the blocks. Early Hadoop used 64 MB like GFS; Hadoop 2 raised the default to 128 MB.
- HDFS keeps 3 replicas by default and spreads them across racks, as in GFS.
- Hadoop MapReduce gives Java
MapperandReducerclasses to extend: the same two-function model. - Failure handling from the paper is all there: heartbeats, task re-execution and speculative execution.
- And the locality idea, which you saw in action in Lesson 1.5.
Try it yourself: the papers' ideas, running
Every idea in the animation is visible in this cluster. Find each one.
How the early adopters used Hadoop
1. Yahoo!: search and the web graph
Yahoo! ran Hadoop on tens of thousands of servers for search indexing, the web link graph, ad analytics and spam detection, and contributed much of the core code.
2. Facebook: SQL for analysts
Facebook stored huge volumes of user-activity data in Hadoop and created Hive so analysts could use SQL instead of writing Java MapReduce jobs.
3. Social networks: “people you may know”
Professional and social networks computed recommendations such as “people you may know” with large batch jobs over their connection graphs, a classic early Hadoop workload.
4. Benchmarks that changed minds
In 2008 a 910-node Hadoop cluster sorted 1 TB in 209 seconds and won the terabyte sort benchmark. Free software on commodity servers was now competitive with expensive specialised systems.
5. Distributions for everyone else
Banks, telecoms and retailers adopted Hadoop through supported distributions from companies like Cloudera, Hortonworks and MapR, which bundled HDFS, YARN, Hive, HBase and management tools.
Company examples describe publicly reported uses in general terms; details and scale changed over the years.
Hadoop's history as data
A timeline is just a list of (year, event) pairs. This program sorts the milestones, prints them, and works out how long each step took. Then try the challenges, including the famous 2008 sort record.
events = [
(2006, "Hadoop leaves Nutch; Doug Cutting joins Yahoo!"),
(2003, "Google publishes the GFS paper"),
(2008, "Hadoop becomes a top-level Apache project"),
(2004, "Google publishes the MapReduce paper"),
(2002, "Apache Nutch starts"),
(2011, "Hadoop 1.0.0 released"),
]
events.sort()
prev = None
for year, what in events:
gap = f"(+{year - prev} y)" if prev else ""
print(year, what, gap)
prev = yearevents.sort()Tuples sort by their first item, the year, so the list comes out in time order.for year, what in events:Unpacking: each tuple's two parts go intoyearandwhat.gap = f"(+{year - prev} y)" if prev else ""A conditional expression: show the gap from the previous event, except for the first one.
Building a timeline, one line at a time
Watch the list of milestones get sorted, then the loop calculate the gap between each event and the one before it.
Worked examples: real problems, solved and traced in memory
Three small problems built on real numbers from Hadoop's history: a release table, the sort record, and how block size changed. Each runs live and is traced in memory.
Years between major releases
Problem statement
Given the release years of Hadoop 1.0 (2011), 2.2 (2013), 3.0 (2017) and 3.5 (2026), print how many years each major line came after the previous one.
1.0 2011 · 2.2 2013 · 3.0 2017 · 3.5 2026
1.0 -> 2.2: 2 years 2.2 -> 3.0: 4 years 3.0 -> 3.5: 9 years
Approach
- Store versions and years as a list of pairs.
- Loop from the second item, comparing with the one before it.
- Print the gap.
Solution: run it live
What happens in memory, step by step
Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.
What did the 2008 sort record mean per disk?
Problem statement
The record cluster had 910 nodes with 4 disks each and sorted 1 TB in 209 s. A sort reads the data and writes it back at least once. Estimate the MB/s each disk handled for reading plus writing.
910 nodes × 4 disks · 1 TB · 209 s · read + write
Disks: 3640 Each disk: about 2.8 MB/s
Approach
- Total bytes moved through disks ≈ 2 × 1 TB (read once, write once).
- Divide by time and by the number of disks.
- Compare with a disk of that era (roughly 50–80 MB/s).
Solution: run it live
What happens in memory, step by step
Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.
How did the jump from 64 MB to 128 MB blocks help the NameNode?
Problem statement
The NameNode keeps an object for every block in memory (roughly 150 bytes each, a common rule of thumb). For 2 PB of data with 3 replicas, compare the number of block objects and the memory needed with 64 MB and 128 MB blocks.
2 PB of files · 64 vs 128 MB blocks · ~150 bytes per block object
64 MB: 33.6 million blocks, ~4.7 GB of NameNode RAM 128 MB: 16.8 million blocks, ~2.3 GB of NameNode RAM
Approach
- Blocks = data size ÷ block size (replicas are tracked inside each block object, so count blocks once).
- Memory ≈ blocks × 150 bytes.
- Print both sizes side by side.
Solution: run it live
What happens in memory, step by step
Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.
Check yourself
Quick revision
Lesson 2.2 in one screen
- 2002 Nutch (Cutting, Cafarella) → 2003 GFS paper → 2004 MapReduce paper; NDFS in Nutch.
- 2006 Hadoop splits from Nutch (Lucene sub-project); Cutting joins Yahoo!.
- Jan 2008 top-level Apache project; 2008 910 nodes sort 1 TB in 209 s.
- Releases: 1.0 Dec 2011 · 2.2 (YARN GA) Oct 2013 · 3.0 Dec 2017 · 3.5.0 Apr 2026.
- GFS master → NameNode; chunkservers → DataNodes; 64 MB chunks → blocks (128 MB since Hadoop 2).
- MapReduce paper → Hadoop MapReduce: map, shuffle, reduce, re-execution, backup (speculative) tasks, locality.
- Bigtable → HBase; Chubby → similar role as ZooKeeper.
- Ecosystem: Hive (Facebook), Pig (Yahoo!), HBase, ZooKeeper, Spark. Distributions: Cloudera, Hortonworks, MapR.
- Name: a toy elephant, not an acronym.
Interview and viva questions
Briefly explain the history of Hadoop.
Doug Cutting and Mike Cafarella started Apache Nutch, an open-source web search engine, in 2002. Google published the GFS paper in 2003 and the MapReduce paper in 2004; the Nutch team implemented both (NDFS and MapReduce). In 2006 that code was separated into a new project, Hadoop, and Cutting joined Yahoo!, which invested heavily in it. Hadoop became a top-level Apache project in January 2008, won the terabyte sort benchmark that year, and reached version 1.0 in December 2011, 2.x with YARN in 2013 and 3.0 in 2017.
How is HDFS related to GFS?
HDFS is an open-source design based on GFS. Both have a single metadata master (GFS master / NameNode) and many storage servers (chunkservers / DataNodes), split files into large chunks or blocks, replicate each chunk (3 by default) across racks, and keep the master out of the data path so clients read data directly from storage servers.
What was Yahoo!'s role in Hadoop?
Yahoo! hired Doug Cutting in 2006 and funded a large engineering team to make Hadoop run reliably on thousands of machines for its search and advertising data. Much of Hadoop's core code and scalability work came from Yahoo!, and Hortonworks was later spun out of its Hadoop team.
What is the Hadoop ecosystem?
The set of projects that grew around HDFS and MapReduce/YARN: Hive (SQL), Pig (data-flow scripts), HBase (NoSQL database), ZooKeeper (coordination), Oozie (workflows), Sqoop and Flume (data ingestion), and processing engines such as Tez and Spark. Module 13 covers which are still widely used.
Why did Hadoop become popular so quickly?
It was free and open source, ran on cheap commodity hardware, was proven at very large scale by Yahoo! and others, solved a problem many companies suddenly had (rapidly growing log and web data), and had a growing ecosystem and commercial support from distribution vendors.
Practice problems
Small programs around Hadoop's real history and numbers. The business problems ask the kind of questions a team asked when deciding to adopt it.
Years from paper to 1.0
The GFS paper came out in 2003 and Hadoop 1.0 in 2011. How many years apart?
2003 · 2011
8 years
Show solution and explanation
print(2011 - 2003, "years")How it works: Eight years from published idea to a 1.0 release, while it was already in heavy production use.
Records in the sort benchmark
The terabyte sort used 100-byte records. How many records are in 1 TB (1012 bytes, as the benchmark defines it)?
10^12 bytes · 100 bytes per record
10,000,000,000 records
Show solution and explanation
print(f"{10**12 // 100:,} records")How it works: 10 billion records, sorted in 209 seconds: almost 5 crore records per second across the cluster.
Look up the counterpart
Using a dictionary, print the open-source counterpart of “GFS” and of “Bigtable”.
{"GFS": "HDFS", "MapReduce": "Hadoop MapReduce", "Bigtable": "HBase"}HDFS | HBase
Show solution and explanation
m = {"GFS": "HDFS", "MapReduce": "Hadoop MapReduce", "Bigtable": "HBase"}
print(m["GFS"], "|", m["Bigtable"])How it works: Knowing these three pairs answers a very common viva question.
Releases per year
Given a list of release years, count how many releases happened in each year.
[2024, 2024, 2025, 2026, 2024, 2025]
{2024: 3, 2025: 2, 2026: 1}Show solution and explanation
years = [2024, 2024, 2025, 2026, 2024, 2025]
count = {}
for y in years:
count[y] = count.get(y, 0) + 1
print(dict(sorted(count.items())))How it works: A frequency count with a dictionary, the same pattern a word-count job uses (Module 8).
Sort rate per node
Print the number of 100-byte records each of the 910 nodes sorted per second in the 2008 record (10 billion records in 209 s).
10,000,000,000 records · 209 s · 910 nodes
52,579 records per node per second
Show solution and explanation
records, seconds, nodes = 10_000_000_000, 209, 910
print(f"{records / seconds / nodes:,.0f} records per node per second")How it works: About 52,580 records per node per second, on machines with 8 GB RAM and four SATA disks.
Which era is a version from?
Write a function that returns “Hadoop 1 (JobTracker)”, “Hadoop 2 (YARN)” or “Hadoop 3” for a version string like "2.7.3".
["1.2.1", "2.7.3", "3.3.6", "3.5.0"]
1.2.1 -> Hadoop 1 (JobTracker) 2.7.3 -> Hadoop 2 (YARN) 3.3.6 -> Hadoop 3 3.5.0 -> Hadoop 3
Show solution and explanation
def era(v):
major = int(v.split(".")[0])
return {1: "Hadoop 1 (JobTracker)", 2: "Hadoop 2 (YARN)"}.get(major, "Hadoop 3")
for v in ["1.2.1", "2.7.3", "3.3.6", "3.5.0"]:
print(v, "->", era(v))How it works: Useful when you meet an old cluster: the major version tells you which architecture you are dealing with.
Licence cost avoided
A company compares a proprietary system at Rs 4 lakh per node per year in licences with open-source Hadoop (no licence) plus Rs 60,000 per node per year for a support subscription. For 50 nodes over 3 years, what is saved?
50 nodes · 3 years · Rs 4 lakh vs Rs 60,000 per node-year
Saved: Rs 5.1 crore
Show solution and explanation
nodes, years = 50, 3
proprietary = nodes * years * 4_00_000
open_source = nodes * years * 60_000
print(f"Saved: Rs {(proprietary - open_source) / 1e7:.1f} crore")How it works: Rs 5.1 crore over three years (illustrative prices). Open source does not mean zero cost, but the economics drove adoption.
Was the record cluster network-bound?
The 2008 record cluster had 40 nodes per rack, each with 1 Gbps, and 8 Gbps of uplinks per rack. Print the oversubscription ratio and the uplink share per node when every node sends out of the rack at once.
40 × 1 Gbps · 8 Gbps uplinks
Oversubscription: 5 : 1 Each node gets 200 Mbps when all send at once
Show solution and explanation
down, up = 40 * 1, 8
print(f"Oversubscription: {down / up:.0f} : 1")
print(f"Each node gets {up / 40 * 1000:.0f} Mbps when all send at once")How it works: 5 : 1. During the shuffle of a sort, almost all data crosses racks, so the uplinks, not the disks, set the pace. The lesson from Module 1 was already true in 2008.
Real-world scenario questions (asked in company interviews)
History questions in interviews are usually about understanding, not memorising dates. These show how to turn the story into a strong answer.
“Tell me how Hadoop started.”
An interviewer opens with this to see how you explain things. You have one minute.
The interviewer is checking: can you tell a clear, correct story and connect it to the design?
Show a strong answer
“Around 2003–04 Google published two papers describing how it stored and processed the web: GFS and MapReduce. Doug Cutting and Mike Cafarella were building Nutch, an open-source search engine that had the same scaling problem, so they implemented those designs. In 2006 that code became its own project, Hadoop, and Yahoo! invested heavily to run it on thousands of machines. That's why HDFS looks like GFS (a NameNode master, DataNodes, big blocks, three replicas) and why Hadoop MapReduce follows the paper's map–shuffle–reduce model with automatic re-execution.”
Avoid: saying “Google created Hadoop.” Google published ideas; Hadoop was written by the open-source community.
You inherit a Hadoop 2.7 cluster
On your first day you find the company runs Hadoop 2.7, released around 2015–16, on Java 7. Your manager asks what you think.
The interviewer is checking: do you know the version history well enough to see the risks of very old releases?
Show a strong answer
- The 2.7 line is long out of active maintenance, so it no longer receives security fixes or bug fixes.
- Java 7 is far out of support, and Hadoop 3 needs at least Java 8; newer 3.x lines support newer Java versions.
- Missing features: erasure coding, multiple standby NameNodes, improved cloud connectors and many performance fixes (Lesson 2.4).
- Propose an upgrade plan: inventory jobs and dependencies, test on a staging cluster, upgrade Java, then HDFS and YARN with a rollback plan, or consider migrating to a supported distribution or cloud service.
Avoid: “It works, leave it.” Unsupported software is a security risk.
Why did an open-source copy beat the original?
The interviewer asks: “Google's internal systems were years ahead. Why did Hadoop, not Google's code, become the industry standard?”
The interviewer is checking: do you understand how open-source ecosystems grow?
Show a strong answer
- Google's code was internal and tied to its own infrastructure; nobody else could use it.
- Hadoop was free (Apache licence), ran on ordinary servers, and many companies improved the same code base, so it got better fast.
- An ecosystem (Hive, HBase, Pig, later Spark) and commercial support (Cloudera, Hortonworks, MapR) made it practical for enterprises.
- Neutral governance at the Apache Software Foundation let competitors collaborate.
Avoid: claiming Hadoop was technically superior to Google's systems; the advantage was openness and ecosystem.
“Does HDFS still use 64 MB blocks?”
You mention blocks while explaining HDFS, and the interviewer asks what the default block size is, and why it changed.
The interviewer is checking: are you up to date, and do you know why block size matters?
Show a strong answer
- Early Hadoop used 64 MB, like the 64 MB chunks of GFS. Since Hadoop 2 the default is 128 MB (
dfs.blocksize); many clusters use 256 MB for big files. - Bigger blocks mean fewer block objects in NameNode memory and fewer, longer map tasks (less scheduling overhead). Disks also got faster, so a 128 MB read still takes only about a second.
- Too big a block reduces parallelism for medium-sized files.
Avoid: quoting 64 MB as today's default.