When not to use Hadoop

“We need Hadoop!” — “Do you, though?”

When Hadoop is the wrong tool

Words you'll meet in this lesson

Rough thresholds (2026, approximate)

Thresholds move every year as single machines get faster. Measure your own workload before deciding.

The decision at three levels

Better tools for the jobs Hadoop is bad at

When Hadoop is still a strong choice

A five-question checklist, applied to the pickle startup

1. How big is the data, and how fast is it growing?

   gb, growth, years = 40, 0.5, 3
   print(round(gb * (1 + growth) ** years), "GB in 3 years")

   135 GB in 3 years
   Even growing 50% a year, it stays small enough for one server. Distributed storage adds nothing.

2. How fast must answers come?

   The website needs order status in milliseconds; the founders want a sales dashboard updated daily.

   The website needs an OLTP database (they already have one). The daily dashboard is light analytics, not a batch job over terabytes.

3. Do records change?

   Orders change status (placed → packed → shipped → delivered).

   Frequent updates fit a database, not write-once files.

4. Who will run it?

   Three people, none of them infrastructure engineers.

   A self-managed cluster would need monitoring, upgrades, security and disk swaps. That is a job they cannot staff.

5. Decision

   Keep PostgreSQL for the app; copy data nightly into DuckDB or a small cloud warehouse for dashboards. Revisit if data reaches many TB.

   Cost: close to zero extra. Time to first dashboard: days, not months. Kabir has just given his first expert “no”.

Three projects, three answers: the request, and the reasoning behind it

Arjun Sir reviews three project requests that all say “use Hadoop”. Left: the request. Right: the checklist and the recommendation.

Real decisions about Hadoop

Examples describe common patterns, not any single company. Thresholds are approximate and change as hardware improves.

A tool-recommendation function

This function applies the five-question checklist from Part 4 to a workload and recommends a tool. Run it on the three projects from the animation, then try the challenges.

def recommend(size_tb, latency_s, updates, ops_team, must_be_onprem):
    if latency_s < 1:
        return "OLTP database or key-value store"
    if latency_s < 60:
        return "Stream processing (Kafka + Flink / Spark Streaming)"
    if size_tb < 1:
        return "One machine: PostgreSQL / DuckDB / pandas"
    if not ops_team and not must_be_onprem:
        return "Managed cloud warehouse or lakehouse"
    if updates:
        return "Lakehouse tables (Iceberg/Delta/Hudi) on Spark"
    return "Hadoop + Spark (batch, distributed)"

projects = {
    "pickle startup": (0.04, 86400, True, False, False),
    "fraud alerts":   (50, 2, False, True, False),
    "telecom CDRs":   (4000, 86400, False, True, True),
}
for name, p in projects.items():
    print(f"{name:15} -> {recommend(*p)}")

• def recommend(size_tb, latency_s, updates, ops_team, must_be_onprem):The five questions as parameters: size, how fast answers are needed, whether records change, whether a team can run a cluster, and where the data must live.
• if latency_s < 1:The order of the checks matters: latency rules out batch tools first, whatever the data size.
• print(f"{name:15} -> {recommend(*p)}")*p unpacks the tuple into the five arguments.

Run it on your PC: how much can one computer do?

Before reaching for a cluster, measure a single machine. This program writes 20 lakh fake orders to a CSV file and totals them per item with plain Python on one core. Save it as one_machine.py and run it.

# one_machine.py: how much data can ONE ordinary computer aggregate quickly?
import random, time, csv, os, tempfile

random.seed(1)
items = ["chai", "vada pav", "samosa", "poha", "idli", "dosa", "thali", "lassi"]
rows = 2_000_000
path = os.path.join(tempfile.gettempdir(), "orders_2m.csv")

t0 = time.time()
with open(path, "w", newline="") as f:
    w = csv.writer(f)
    for i in range(rows):
        w.writerow([i, random.choice(items), random.randint(10, 300)])
t1 = time.time()

totals = {}
with open(path, newline="") as f:
    for _, item, amount in csv.reader(f):
        totals[item] = totals.get(item, 0) + int(amount)
t2 = time.time()

size_mb = os.path.getsize(path) / 1024 / 1024
print(f"Rows: {rows:,}  File: {size_mb:.0f} MB")
print(f"Write: {t1 - t0:.1f} s   Group-by in plain Python: {t2 - t1:.1f} s")
print("Top item:", max(totals, key=totals.get))
print(f"Rough speed: {rows / (t2 - t1) / 1e6:.1f} million rows per second on one core")
os.remove(path)

# In a terminal:  python one_machine.py

The checklist, one line at a time

Follow one workload (2 TB, answers needed within 2 seconds) through the decision function. Watch which condition decides.

Worked examples: real problems, solved and traced in memory

Three decisions a data team really makes: whether a data set needs a cluster at all, what each option costs over three years, and how to score options fairly. Each runs live and is traced in memory.

sales 80 GB +30% · clicks 3 TB +80% · HR 2 GB +5% · sensors 600 GB +300%

sales       :   0.08 TB, +30%/yr -> one machine
clickstream :    3.0 TB, +80%/yr -> consider distributed
hr          :  0.002 TB, +5%/yr -> one machine
sensors     :    0.6 TB, +300%/yr -> consider distributed

A: 8 × 12 L + 30 L/yr · B: 20 TB × Rs 2,000/month + Rs 25,000/day · 3 years

A on-prem cluster : Rs 186 lakh
B cloud warehouse : Rs 288 lakh (storage 14 + compute 274)
Cheaper: A

weights 0.3/0.3/0.2/0.2 · three options with four scores each

Hadoop+Spark on-prem  : 3.8
Cloud warehouse       : 3.4
Cloud lakehouse       : 3.5
Choose: Hadoop+Spark on-prem

Check yourself

Quick revision

Interview and viva questions

Practice problems

Decision and cost exercises. Each one asks you to justify “Hadoop or not?” with numbers.

1 TB SSD · 120 GB · 50% free

Fits: True

print("Fits:", 120 <= 1024 * 0.5)

300 GB · +40% · 4 years

1152 GB

print(round(300 * 1.4 ** 4), "GB")

[0.05, 5, 3600]

0.05 s -> OLTP
5 s -> streaming
3600 s -> batch

for s in [0.05, 5, 3600]:
    print(s, "s ->", "OLTP" if s < 1 else "streaming" if s < 60 else "batch")

20 s · 50 MB/s · [100 MB, 10 GB, 1 TB]

     100 MB: start-up is 90.9% of the time
   10240 MB: start-up is 8.9% of the time
 1048576 MB: start-up is 0.1% of the time

for mb in [100, 10 * 1024, 1024 * 1024]:
    work = mb / 50
    print(f"{mb:>8} MB: start-up is {20 / (20 + work):.1%} of the time")

50,00,000 × 20 KB · 256 MB

5000000 -> 382 files

import math
files = 50_00_000
total_mb = files * 20 / 1024
print(files, "->", math.ceil(total_mb / 256), "files")

(50, 7200, True) · (5, 7200, True) · (50, 7200, False)

(50, 7200, True) -> Hadoop
(5, 7200, True) -> something else
(50, 7200, False) -> something else

def fits(size_tb, latency_s, ops):
    return "Hadoop" if size_tb >= 10 and latency_s >= 3600 and ops else "something else"
for c in [(50, 7200, True), (5, 7200, True), (50, 7200, False)]:
    print(c, "->", fits(*c))

Rs 1.2 crore / 5 years · 2 × Rs 25 L · Rs 8 L

Yearly: Rs 82 lakh; people = 61%

hardware = 120 / 5
people = 2 * 25
other = 8
total = hardware + people + other
print(f"Yearly: Rs {total:.0f} lakh; people = {people / total:.0%}")

Rs 1.5 L per TB-year · Rs 90 L per year fixed

On-prem cheaper above 61 TB

import math
print("On-prem cheaper above", math.floor(90 / 1.5) + 1, "TB")

Real-world scenario questions (asked in company interviews)

Interviewers often test judgement with a “should we use Hadoop?” question. Practise answering with questions, numbers and alternatives.