A short history of big data: from one server to the lakehouse

Why is it called Hadoop?

How did big data begin?

Words you’ll meet in this lesson

Milestones to remember

The history at three levels

Five eras of data technology

The famous papers and what they became

Twenty-five years in six steps

1. 2000: one server and a nightly report

   A bank keeps every transaction in one large Oracle database. A data warehouse is loaded each night, and managers read reports the next morning.

   It works while data is gigabytes. When it grows, the only answer is a bigger, far costlier server. That is scaling up, and it has a ceiling.

2. 2003–2004: Google writes down its secret

   Google’s index of the web is far too big for any server. Its engineers store files as 64 MB blocks with 3 copies across thousands of cheap machines (GFS), and process them with two simple functions, map and reduce.

   The key ideas: expect failure, move code to data, and let a framework handle splitting, scheduling and retries. Google published both papers.

3. 2006–2008: Hadoop makes it free

   Doug Cutting and Mike Cafarella build an open-source version for their web-search project. Yahoo! hires Cutting and runs it at huge scale. It becomes Apache Hadoop: HDFS + MapReduce.

   Now any company can run a cluster on ordinary servers. Facebook adds Hive so analysts can write SQL instead of Java MapReduce code.

4. 2009–2014: faster and more varied

   MapReduce writes to disk after every step, so jobs with many steps are slow. Spark keeps data in memory. Kafka carries events in real time. NoSQL databases serve apps that need millisecond look-ups.

   The single “Hadoop stack” becomes an ecosystem of specialised tools, each solving one weakness of the original.

5. 2012 onwards: rent, don’t buy

   Instead of running their own clusters, companies store data in cloud object storage and query it with warehouses billed by usage: BigQuery, Redshift, Snowflake.

   Storage and compute are separated: keep petabytes cheaply, and rent 100 machines for 10 minutes when a query needs them.

6. Today: one copy for everything, and AI on top

   Table formats (Delta Lake, Iceberg, Hudi) add transactions and updates to files in the lake. The same tables feed dashboards, streaming jobs and ML models, and AI assistants turn questions into SQL.

   This is the lakehouse. Under the hood, it still uses the 2003 ideas: blocks, copies, parallel tasks and re-running failed work.

What users saw, and what engineers built

Step through the six frames. Left: what ordinary people saw on their screens in each era. Right: what engineers were building behind it.

Real-time scenarios: the history in action

Three live scenarios: the same business question answered in five eras, the five forces that made big data necessary, and an advanced look at why in-memory processing took over.

The same question in five eras

TiffinGo-sized data: a 10 TB orders table. The manager asks the simplest question: how many orders per city yesterday? Watch how the work, the code and the waiting time change across the eras. The bar shows how long the wait feels (longer bar = faster answer).

Five forces that created big data

Big data didn’t appear because someone invented Hadoop. Five changes in the world made it necessary, and each one still shapes how platforms are built.

Why Spark replaced MapReduce for most jobs

Machine learning and graph algorithms repeat the same calculation many times over the same data. Watch the two engines run a 5-step job, and see how Spark still recovers from a failure without saving every step to disk.

History you can see in today’s job descriptions

A timeline you can query

The milestones of big data are stored as a list of tuples: (year, event, era). The program prints the timeline and counts the events in each era. Change it, then try the challenges: find gaps between a paper and its open-source copy, sort events, and follow the “family tree” of tools.

timeline = [
    (2003, "GFS paper", "Google papers"),
    (2004, "MapReduce paper", "Google papers"),
    (2006, "Hadoop starts", "Hadoop era"),
    (2008, "Hive open-sourced", "Hadoop era"),
    (2011, "Kafka open-sourced", "Specialised tools"),
    (2014, "Spark 1.0", "Specialised tools"),
    (2012, "Redshift announced", "Cloud"),
    (2019, "Delta Lake open-sourced", "Lakehouse"),
]

for year, event, era in sorted(timeline):
    print(year, "-", event)

per_era = {}
for year, event, era in timeline:
    per_era[era] = per_era.get(era, 0) + 1
print("Events per era:", per_era)

• (2003, "GFS paper", "Google papers"),Each milestone is a tuple: (year, event, era). Tuples keep related values together.
• for year, event, era in sorted(timeline):sorted orders tuples by their first item, the year, so the 2012 entry moves into place.
• per_era[era] = per_era.get(era, 0) + 1The counting pattern: start at 0 for a new era, add 1 each time.

Run it on your PC: your life on the big data timeline

This program prints the milestones grouped by era, with a small text bar, and tells you how old you were at each one. Save it as history_timeline.py, change BORN to your birth year, and run it.

"""Print the big data timeline as a text chart, grouped by era.
Change BORN to your birth year to see how old you were at each milestone."""
BORN = 2006
NOW = 2026

milestones = [
    (1997, "Term 'big data' used by NASA researchers", "Before Hadoop"),
    (2001, "Doug Laney describes the 3 Vs", "Before Hadoop"),
    (2003, "Google File System paper", "Google papers"),
    (2004, "MapReduce paper", "Google papers"),
    (2006, "Bigtable paper; Hadoop starts; Amazon S3", "Google papers"),
    (2008, "Hadoop top-level Apache project; Hive, Cassandra", "Ecosystem"),
    (2011, "Kafka open-sourced by LinkedIn", "Ecosystem"),
    (2014, "Spark 1.0, Apache top-level project", "Ecosystem"),
    (2016, "UPI launched; cheap 4G in India", "Cloud and mobile"),
    (2019, "Delta Lake open-sourced", "Lakehouse"),
    (2023, "India's DPDP Act passed", "Lakehouse"),
]

era = None
for year, event, group in milestones:
    if group != era:
        era = group
        print(f"\n== {era} ==")
    age = year - BORN
    when = f"you were {age}" if age >= 0 else f"{-age} years before you were born"
    bar = "#" * ((year - 1995) // 2)
    print(f"{year} {bar:<16} {event}  ({when})")

span = milestones[-1][0] - milestones[0][0]
print(f"\n{len(milestones)} milestones over {span} years; {NOW - 2003} years since the GFS paper.")

# In a terminal:  python history_timeline.py

Paper to open source, one line at a time

The program works out how many years passed between each Google paper and its open-source copy, and keeps the longest gap. Watch longest change only when a bigger gap appears.

Worked examples: real problems, solved and traced in memory

Four small programs about the history, from grouping events by decade to walking a family tree of tools. Each runs live and is traced.

1997, 2001, 2003, 2004, 2006, 2008, 2011, 2014, 2016, 2019, 2023

1990s: 1 milestones
2000s: 5 milestones
2010s: 4 milestones
2020s: 1 milestones

GFS 2003→HDFS 2006 · MapReduce 2004→Hadoop 2006 · Bigtable 2006→HBase 2008 · Dynamo 2007→Cassandra 2008

Gaps: [3, 2, 2, 1]
Average: 2.0 years
Fastest copy: Dynamo -> Cassandra

store 500 TB of files, live order events, fast ML training loops, SQL for analysts

2006: store 500 TB of files    -> HDFS / object storage
2008: SQL for analysts         -> Hive, later cloud warehouses
2011: live order events        -> Kafka
2014: fast ML training loops   -> Spark

HBase ← Bigtable · Cassandra ← Bigtable, Dynamo · ScyllaDB ← Cassandra · DynamoDB ← Dynamo

Descended from Bigtable : ['HBase', 'Cassandra', 'ScyllaDB']

Check yourself

Quick revision

Interview and viva questions

Practice problems

Small programs built around the timeline. The last two are business tasks about technology decisions.

GFS 2003, Hadoop 2006, Kafka 2011, Spark 2014

GFS 23 years ago
Hadoop 20 years ago
Kafka 15 years ago
Spark 12 years ago

milestones = {"GFS": 2003, "Hadoop": 2006, "Kafka": 2011, "Spark": 2014}
for name, year in milestones.items():
    print(name, 2026 - year, "years ago")

[2011, 2003, 2019, 2006, 2014]

Earliest: 2003
Latest: 2019

years = [2011, 2003, 2019, 2006, 2014]
print("Earliest:", min(years))
print("Latest:", max(years))

Hive, Kafka, Spark, Iceberg

Hive was created at Facebook
Kafka was created at LinkedIn
Spark was created at UC Berkeley
Iceberg was created at Netflix

created_at = {"Hive": "Facebook", "Kafka": "LinkedIn", "Spark": "UC Berkeley", "Iceberg": "Netflix"}
for tool in ["Hive", "Kafka", "Spark", "Iceberg"]:
    print(f"{tool} was created at {created_at[tool]}")

6 events

2006 Hadoop
2008 Hive
2011 Kafka
2014 Spark 1.0

events = [(2019, "Delta Lake"), (2006, "Hadoop"), (2003, "GFS"),
          (2011, "Kafka"), (2014, "Spark 1.0"), (2008, "Hive")]
for year, name in sorted(events):
    if 2005 <= year <= 2015:
        print(year, name)

Hadoop 2006, Hive 2008, Kafka 2011, Spark 2014, Snowflake 2015, Delta 2019, Iceberg 2018

{'Hadoop era': 2, 'Specialised + cloud': 4, 'Lakehouse': 1}

tools = {"Hadoop": 2006, "Hive": 2008, "Kafka": 2011, "Spark": 2014,
         "Snowflake": 2015, "Delta": 2019, "Iceberg": 2018}
eras = {"Hadoop era": 0, "Specialised + cloud": 0, "Lakehouse": 0}
for name, y in tools.items():
    if y < 2009:
        eras["Hadoop era"] += 1
    elif y < 2019:
        eras["Specialised + cloud"] += 1
    else:
        eras["Lakehouse"] += 1
print(eras)

Delta Lake ← Hive tables ← Hadoop ← GFS

Delta Lake <- Hive tables <- Hadoop <- GFS

inspired_by = {"Delta Lake": "Hive tables", "Hive tables": "Hadoop", "Hadoop": "GFS"}
tool = "Delta Lake"
chain = [tool]
while tool in inspired_by:
    tool = inspired_by[tool]
    chain.append(tool)
print(" <- ".join(chain))

8 steps, 100 GB, 3 copies, 2 GB/s

Seconds per gap: 200.0
Minutes saved: 23.3

steps, gb, copies, speed = 8, 100, 3, 2
per_gap = (gb * copies + gb) / speed
total = (steps - 1) * per_gap
print("Seconds per gap:", per_gap)
print("Minutes saved:", round(total / 60, 1))

Rs 90 lakh · Rs 1,200/hour × 4 h/day × 365 × 5

Server, 5 years: Rs 9000000
Cloud, 5 years : Rs 8760000
Cheaper: cloud

server = 90_00_000
cloud = 1200 * 4 * 365 * 5
print("Server, 5 years: Rs", server)
print("Cloud, 5 years : Rs", cloud)
print("Cheaper:", "cloud" if cloud < server else "server")

120 jobs, 8 per week

Weeks needed: 15
Done after 6 weeks: 40 %

import math
jobs, per_week = 120, 8
print("Weeks needed:", math.ceil(jobs / per_week))
print("Done after 6 weeks:", round(6 * per_week / jobs * 100), "%")

Real-world scenario questions (asked in company interviews)

History questions that interviewers use to test real understanding. Decide your answer first, then open the model answer.