Why one computer is not enough for big data

Riya's laptop meets five years of data

Why isn't one computer enough?

Words you'll meet in this lesson

Typical numbers to remember (2026, approximate)

These are ballpark figures to reason with. Real speeds depend on the exact hardware and the workload.

One computer's limits at three levels

The five walls of one computer

Scale up vs scale out

Push 2 TB through one laptop and watch each wall appear

1. Storage: does it even fit?

   data_gb, disk_gb = 2048, 512
   print("Fits on disk?", data_gb <= disk_gb)

   Fits on disk? False
   The first wall is the simplest: 2 TB is four times the laptop's disk. An external drive helps, but then every read goes through a USB cable.

2. Memory: load it all into a list

   rows = open("orders_5yrs.csv").readlines()   # tries to hold everything in RAM

   MemoryError
   In Python, one small record object takes a few hundred bytes of RAM (you'll measure it below), far more than its size in the file. Billions of records need hundreds of GB of RAM. The laptop has 8.

3. Stream it instead: one line at a time

   total = 0
   for line in open("orders_5yrs.csv"):          # reads one line, forgets it
       total += int(line.split(",")[2])

   Streaming fixes the memory wall: only one line is in RAM at a time. But now the I/O wall appears: the whole 2 TB still has to be read from the disk.

4. I/O: how long does reading take?

   mb = 2 * 1024 * 1024          # 2 TB in MB
   print(round(mb / 200 / 3600, 1), "hours at 200 MB/s (external hard disk)")

   2.9 hours at 200 MB/s (external hard disk)
   Nearly three hours just to read the data once, before any calculation. Python parsing each line is usually slower still, and the plain loop keeps only one of the four cores busy.

5. Scale up: buy a bigger server

   A server with 64 cores, 512 GB RAM and fast NVMe disks finishes the job in minutes. For this year.

   Next year the data doubles, and the year after it doubles again. Each bigger server costs disproportionately more, and there is always a largest machine. If it fails, the report doesn't run.

6. Scale out: split the work

   machines = 20
   print(round(2 * 1024 * 1024 / machines / 200 / 60), "minutes, each machine reading its own 100 GB")

   9 minutes, each machine reading its own 100 GB
   Twenty ordinary machines, each with its own disk, each reading its own part at the same time. Need it faster next year? Add machines. This is the idea behind Hadoop and Spark, and it is exactly what Lesson 1.3 builds on.

One laptop vs 2 TB: what Riya sees, and what the hardware is doing

Step through the six frames. Left: Riya's screen. Right: the disk, memory and CPU inside.

Where the time goes: a rough budget for 2 TB

Reading is only the start: parsing, calculating and writing results add more time. The ratios are what matter.

Real-time scenarios: watching one machine run out of road

Three live scenarios: the race that convinced web companies to build Hadoop, a monthly report where every fix moves the bottleneck somewhere else, and an advanced one on the cost of scaling up versus scaling out.

Reading 100 TB with 1 disk vs 1,000 disks

This is the calculation that convinced web companies to build systems like Hadoop. Watch the race, and see what happens when one of the thousand disks fails.

Kabir's monthly report: every fix moves the bottleneck

TiffinGo's monthly orders file is 50 GB. Kabir tries to build the report on his 16 GB, 8-core laptop. Watch the four resource meters as he fixes one problem after another.

The cost of growing: scale up vs scale out

As data grows, compare the cost of buying ever-bigger single servers with adding ordinary machines to a cluster. Then see what each does when hardware fails, and how the cloud changes the maths.

Where one machine stopped being enough

Examples are simplified to show the scaling problem. Real setups vary by company.

Calculate when one machine is too slow

The core big data calculation: how long does it take just to read the data, with 1 machine or many? Each machine has one hard disk reading about 200 MB per second, and reads only its own share. Then try the challenges: will the data fit, how often do disks fail, and how to split work.

def read_minutes(data_tb, machines, mb_per_sec=200):
    data_mb = data_tb * 1024 * 1024
    per_machine = data_mb / machines        # each machine reads only its share
    return per_machine / mb_per_sec / 60

data_tb = 2
for machines in [1, 5, 20, 100]:
    m = read_minutes(data_tb, machines)
    print(f"{machines:>3} machine(s): {m:6.1f} minutes")

• def read_minutes(data_tb, machines, mb_per_sec=200):mb_per_sec=200 is a default value: a typical hard disk. Pass another value to model an SSD.
• per_machine = data_mb / machinesThe heart of scaling out: with the data spread evenly, each machine reads a smaller part, and all read at the same time.
• f"{machines:>3} machine(s): {m:6.1f} minutes"Format codes: >3 right-aligns in 3 characters, 6.1f shows one decimal place in 6 characters, so the columns line up.

Run it on your PC: measure your machine's memory wall

How much RAM do records really take inside Python? This program builds 10 lakh small payment records in a list and measures the memory used with Python's built-in tracemalloc. Save it as ram_test.py and run it on your computer.

import os, shutil, tracemalloc

print("CPU cores :", os.cpu_count())
total, used, free = shutil.disk_usage(".")
print("Disk      :", round(total / 1024**3), "GB total,", round(free / 1024**3), "GB free")

# How much memory do 10 lakh small payment records take inside Python?
tracemalloc.start()
records = [{"id": i, "item": "chai", "amount": 20} for i in range(1_000_000)]
used_mb = tracemalloc.get_traced_memory()[0] / 1024**2
tracemalloc.stop()
print("10 lakh records in a list use", round(used_mb), "MB of RAM")

per_record = used_mb * 1024**2 / len(records)
print("That is about", round(per_record), "bytes per record")
for n, label in [(100_000_000, "10 crore"), (10_000_000_000, "1,000 crore")]:
    gb = n * per_record / 1024**3
    print(f"{label:>11} records would need about {gb:,.0f} GB of RAM")

# In a terminal:  python ram_test.py

Scaling out, one line at a time

Watch how the same function is called three times, once for each cluster size. Each call gets its own frame with its own machines and per_machine; the frame disappears when the function returns.

Worked examples: real problems, solved and traced in memory

Four decisions, from beginner to advanced. Three a data engineer actually makes: does the data fit, which kind of scaling is cheaper, and how many failures to plan for. Each program runs live and is traced in memory. The last one finds the bottleneck of a whole job.

laptop: disk 512, ram 8 · job: disk 2048, ram 1760 (GB)

disk wall: 4 times too small
ram wall: 220 times too small

needed 256 cores · servers {64: 12 lakh, 128: 35 lakh, 256: 1.1 crore} · node 16 cores at 2.5 lakh

Scale up : one server, Rs 110.0 lakh
Scale out: 16 nodes,   Rs 40.0 lakh

1000 disks, 1.5% fail per year

Expected failures per year: 15.0
About one failure every 24 days

600 GB read · 48 core-hours · 120 GB send · 1000 MB/s · 16 cores · 125 MB/s

disk    :   10.2 minutes
cpu     :  180.0 minutes
network :   16.4 minutes
Bottleneck: cpu

Check yourself

Quick revision

Interview and viva questions

Practice problems

Capacity and time calculations that come up in real data platform planning. The last three are business tasks.

500 GB, 250 MB/s

34.1 minutes

mb = 500 * 1024
print(round(mb / 250 / 60, 1), "minutes")

90 TB, 4 TB per disk

23 disks

import math
print(math.ceil(90 / 4), "disks")

6 GB file, ×5 overhead, 16 GB RAM

Needs 30 GB; fits? False

needed = 6 * 5
print("Needs", needed, "GB; fits?", needed <= 16)

tasks T1..T7, 3 workers

{1: ['T1', 'T4', 'T7'], 2: ['T2', 'T5'], 3: ['T3', 'T6']}

tasks = [f"T{i}" for i in range(1, 8)]
workers = {1: [], 2: [], 3: []}
for i, t in enumerate(tasks):
    workers[i % 3 + 1].append(t)
print(workers)

120 min on 1, 10 min on 16

Speed-up: 12.0
Efficiency: 75.0 %

speedup = 120 / 10
efficiency = speedup / 16 * 100
print("Speed-up:", speedup)
print("Efficiency:", efficiency, "%")

files: 1.5, 3, 9 GB · RAM 16 GB · reserve 4 GB

1.5 GB file needs 9.0 GB: fits
3 GB file needs 18 GB: use 2 chunks
9 GB file needs 54 GB: use 5 chunks

import math
ram, reserve, blowup = 16, 4, 6
usable = ram - reserve
for size in [1.5, 3, 9]:
    need = size * blowup
    if need <= usable:
        print(f"{size} GB file needs {need} GB: fits")
    else:
        chunks = math.ceil(need / usable)
        print(f"{size} GB file needs {need} GB: use {chunks} chunks")

[8000, 21000, 64000, 12000]

hour 0:   8000 orders/s -> 2 nodes
hour 1:  21000 orders/s -> 5 nodes
hour 2:  64000 orders/s -> 13 nodes
hour 3:  12000 orders/s -> 3 nodes
Total node-hours: 23

import math
per_node = 5000
load = [8000, 21000, 64000, 12000]
node_hours = 0
for hour, rate in enumerate(load):
    nodes = max(2, math.ceil(rate / per_node))
    node_hours = node_hours + nodes
    print(f"hour {hour}: {rate:>6} orders/s -> {nodes} nodes")
print("Total node-hours:", node_hours)

30 TB, 2 hours, 200 MB/s per node

Minimum nodes: 22

import math
total_mb = 30 * 1024 * 1024
one_node_secs = total_mb / 200
nodes = math.ceil(one_node_secs / (2 * 3600))
print("Minimum nodes:", nodes)

Rs 40/node-hour, 25 node-hours

1 node(s) for 25 h -> Rs 1000
50 node(s) for 0.5 h -> Rs 1000

rate = 40
for nodes, hours in [(1, 25), (50, 0.5)]:
    print(nodes, "node(s) for", hours, "h -> Rs", round(nodes * hours * rate))

2 TB, +60% per year, limit 20 TB

Year 5 : about 21.0 TB

data, year = 2.0, 0
while data <= 20:
    data = data * 1.6
    year = year + 1
print("Year", year, ": about", round(data, 1), "TB")

Real-world scenario questions (asked in company interviews)

Questions about scaling that interviewers use to see whether you can reason with numbers. Decide your answer first, then open the model answer.