The problem Hadoop solved (and is Hadoop still relevant?)

BharatSearch: a search engine on four laptops

The problem Hadoop solved

Words you'll meet in this lesson

The four requirements, and Hadoop's answer

The problem at three levels

The options in 2003, compared

Is Hadoop still relevant in 2026? An honest answer

Scaling BharatSearch by hand, and where each step hurts

1. Too big for one disk: split by a rule

   import zlib
   def machine_for(url, machines):
       return zlib.crc32(url.encode()) % machines     # a stable hash of the URL
   print(machine_for("https://iitb.ac.in", 4), machine_for("https://coep.org.in", 4))

   2 1
   A hash spreads URLs evenly over 4 machines. Every program that reads or writes a page must know this rule. That is manual sharding.

2. Add a fifth machine

   Now % 5 instead of % 4. Most URLs suddenly belong to a different machine.

   With modulo hashing, about 80% of the pages must be moved when going from 4 to 5 machines (you'll measure it in the code below). Someone has to write and run that migration.

3. A disk dies at 2 am

   Machine 3's disk fails. A quarter of the pages are gone, and the indexing job that was reading them crashes.

   No copies, no automatic detection, no retry. The team re-crawls those pages and re-runs the whole job by hand.

4. Build the index in parallel

   Kabir writes a script that starts the indexer on each machine, waits, collects results and merges them.

   He is now writing a scheduler, a progress tracker and a merge step: the same distributed plumbing every team rebuilds. Hadoop's MapReduce provides all of it.

5. The realisation

   The interesting code (parse a page, count words) is 50 lines. The plumbing (splitting, copying, retrying, merging) is thousands.

   Hadoop's promise: you write the 50 lines. Storage, splitting, placement, parallelism and recovery come from the framework, tested on clusters of thousands of nodes.

Growing from 1 machine to 40: what the team sees, and what is really happening

Follow BharatSearch as the data grows. Left: the team's chat and screens. Right: the machines and the data.

Try it yourself: the four requirements in one cluster

Store a file, process it, and break a node while the job runs. Each step checks one of the four requirements Hadoop was built for.

The same problem, in many industries

Examples are simplified and describe common patterns, not any one company's setup.

Manual sharding, and why it hurts

Before Hadoop, teams split data across machines with their own rules. This program spreads 10,000 URLs over 4 machines with a hash, then counts how many URLs would have to move if a fifth machine is added. Then try the challenges.

import zlib

def machine_for(url, machines):
    return zlib.crc32(url.encode()) % machines

urls = [f"https://site{i}.in/page" for i in range(10000)]
before = {u: machine_for(u, 4) for u in urls}
after = {u: machine_for(u, 5) for u in urls}

moved = sum(1 for u in urls if before[u] != after[u])
print("URLs per machine (4):", [list(before.values()).count(m) for m in range(4)])
print(f"Adding a 5th machine moves {moved} of {len(urls)} URLs ({moved / len(urls):.0%})")

• zlib.crc32(url.encode()) % machinesA hash turns the URL into a number; % machines picks a machine. The same URL always lands on the same machine.
• before = {u: machine_for(u, 4) for u in urls}A dictionary comprehension: for each URL, which of 4 machines owns it.
• moved = sum(1 for u in urls if before[u] != after[u])Count URLs whose owner changes when there are 5 machines instead of 4.

Hash sharding, one line at a time

Watch three URLs being assigned to machines with a hash, first with 4 machines and then with 5. See which ones change owner.

Worked examples: real problems, solved and traced in memory

Three calculations that explain why Hadoop had to exist: how big the data was, how likely a long job was to fail, and how much a bigger cluster would cost without one. Each runs live and is traced in memory.

80 crore pages × 60 KB · ×3 · 4 × 250 GB per machine

Data: 45,776 GB, with copies: 137,329 GB
Machines: 138

1,000 machines · 8 hours · 4% per year

One machine fails during the job: 0.00365%
Job completes with no failures  : 96.4%

500 TB · A: Rs 40,000/TB · B: Rs 2,500/TB raw, ×3 ÷ 0.75 + Rs 20 lakh

A (storage array) : Rs 200 lakh
B (Hadoop cluster): Rs 70 lakh for 2,000 TB raw
A costs 2.9 times more

Check yourself

Quick revision

Interview and viva questions

Practice problems

Small calculations and programs about scale, sharding and failure: the reasons Hadoop exists. The business problems are the arguments teams really had before adopting it.

400 TB · 200 MB/s

24.3 days

print(round(400 * 1024 * 1024 / 200 / 86400, 1), "days")

400 TB · 1000 × 200 MB/s

35.0 minutes

print(round(400 * 1024 * 1024 / (1000 * 200) / 60, 1), "minutes")

["pune", "delhi", "goa"] · 3 machines

pune -> 2
delhi -> 2
goa -> 1

import zlib
for k in ["pune", "delhi", "goa"]:
    print(k, "->", zlib.crc32(k.encode()) % 3)

keys k0…k19999 · 10 → 11 machines

91% of keys move

import zlib
keys = [f"k{i}" for i in range(20000)]
moved = sum(1 for k in keys if zlib.crc32(k.encode()) % 10 != zlib.crc32(k.encode()) % 11)
print(f"{moved / len(keys):.0%} of keys move")

3000 machines · 24 h · 3%/year

21.9%

p = 0.03 * 24 / (365 * 24)
print(f"{1 - (1 - p) ** 3000:.1%}")

["Pune,240", "Delhi,120", "bad line", "Pune,60"]

{'Pune': 300, 'Delhi': 120}

raw = ["Pune,240", "Delhi,120", "bad line", "Pune,60"]
totals = {}
for line in raw:
    parts = line.split(",")
    if len(parts) != 2 or not parts[1].isdigit():
        continue
    totals[parts[0]] = totals.get(parts[0], 0) + int(parts[1])
print(totals)

50 TB · ×2 per year · Rs 40,000/TB · budget Rs 5 crore

Year 5 : 1600 TB costs Rs 6.4 crore

tb, year = 50, 0
while tb * 40_000 <= 5_00_00_000:
    tb *= 2
    year += 1
print("Year", year, ":", tb, "TB costs Rs", tb * 40_000 / 10**7, "crore")

2000 lines/job · 100 lines/day · Rs 6,000/day · 15 jobs

Saved: 300 developer-days = Rs 18 lakh a year

days_per_job = 2000 / 100
saving = days_per_job * 6000 * 15
print(f"Saved: {days_per_job * 15:.0f} developer-days = Rs {saving / 1e5:.0f} lakh a year")

Real-world scenario questions (asked in company interviews)

Questions interviewers ask to check that you understand why Hadoop exists, and where it fits today.