Scaling up vs scaling out (and back-of-envelope estimation)

“How many servers for Diwali?”

Scalability and back-of-envelope estimation

Words you’ll meet in this lesson

Numbers for quick estimates

Scaling and estimation at three levels

Four ways to scale

Side by side

A complete estimate for 10 crore orders a day

1. Write your assumptions

   10 crore orders on Diwali day · peak hour is 5× the average · each order record ≈ 1 KB · one API server handles ≈ 400 orders/s (from a load test) · keep 3 copies of data.

   Assumptions first, out loud. If one is wrong, the interviewer (or your manager) can correct it and the rest of the calculation still stands.

2. Orders per second, average and peak

   orders_per_day = 10_00_00_000          # 10 crore
   avg_per_s = orders_per_day / 1_00_000    # ~1 lakh seconds in a day
   peak_per_s = avg_per_s * 5               # Diwali dinner peak
   print("average:", avg_per_s, "orders/s | peak:", peak_per_s, "orders/s")

   average: 1000.0 orders/s | peak: 5000.0 orders/s
   Dividing by about 1 lakh seconds turns “per day” into “per second”. Systems must handle the peak, not the average.

3. Storage per day and per year

   orders_per_day = 10_00_00_000
   kb_per_order = 1
   gb_per_day = orders_per_day * kb_per_order / 1024 / 1024
   tb_per_year = gb_per_day * 365 / 1024
   print(f"{gb_per_day:.0f} GB/day, {tb_per_year:.0f} TB/year, {tb_per_year * 3:.0f} TB with 3 copies")

   95 GB/day, 34 TB/year, 102 TB with 3 copies
   About 95 GB a day is manageable; the yearly total with copies, around 100 TB, is what drives cost. Add indexes, logs and events and the real figure is several times larger.

4. Servers at the peak

   peak_per_s = 5000
   per_node = 400            # orders/s one API server can handle (measured)
   target_load = 0.6         # keep 40% headroom
   import math
   print("API servers at peak:", math.ceil(peak_per_s / (per_node * target_load)))

   API servers at peak: 21
   Divide the peak by what one server handles at the target load. The API tier is stateless, so these servers scale out behind a load balancer.

5. Network bandwidth

   peak_per_s, kb = 5000, 1
   mb_per_s = peak_per_s * kb / 1024
   print(f"incoming: {mb_per_s:.1f} MB/s = {mb_per_s * 8:.0f} Mbps")

   incoming: 4.9 MB/s = 39 Mbps
   Order data alone is small for the network. Images, app events and replication traffic usually dominate bandwidth, so estimate them separately.

6. Concurrency with Little’s law

   arrival = 5000          # requests per second
   time_inside = 0.2       # seconds each request spends in the system
   print("requests in flight:", int(arrival * time_inside))

   requests in flight: 1000
   At the peak about 1,000 requests are in progress at once. That sizes database connection pools and thread counts: too few and requests queue, too many and the database is overwhelmed.

Diwali night: what Riya sees, and what the autoscaler does

Step through the six frames. Left: the app as Riya experiences it. Right: orders per second and the number of servers behind it.

Real-time scenarios: scaling in practice

Three live scenarios: a two-minute capacity estimate, a cart that vanishes when servers are added, and an advanced look at why more machines can mean less speed.

Estimating Diwali day in two minutes

Watch Kabir answer the interview question line by line. Every step is simple arithmetic with round numbers; the skill is the order of the steps and writing down the assumptions.

Why Riya’s cart vanished when we added servers

Scaling out is easy only for stateless services. Watch what happens to a cart stored in a server’s memory when a second and third server are added.

When 64 machines are slower than 32

Adding machines is not free: they wait for shared resources and must talk to each other to stay in agreement. The Universal Scalability Law shows the effect. Watch throughput as a job grows from 8 to 64 nodes.

Scaling decisions at real companies

A back-of-envelope calculator

Change the assumptions at the top and run. The program turns “orders per day” into peak orders per second, storage per year and the number of servers. Then try the challenges: compare scaling up and out, simulate autoscaling, use Little's law, see the Universal Scalability Law and the tail-latency effect.

import math

orders_per_day = 10_00_00_000   # 10 crore
peak_factor = 5
kb_per_order = 1
copies = 3
per_server = 400                # orders/s one API server handles
target_load = 0.6               # keep 40% headroom

avg = orders_per_day / 86_400
peak = avg * peak_factor
gb_day = orders_per_day * kb_per_order / 1024 / 1024
tb_year = gb_day * 365 / 1024 * copies
servers = math.ceil(peak / (per_server * target_load))

print(f"average   : {avg:,.0f} orders/s")
print(f"peak      : {peak:,.0f} orders/s")
print(f"storage   : {gb_day:,.0f} GB/day, {tb_year:,.0f} TB/year with {copies} copies")
print(f"servers   : {servers} at {target_load:.0%} target load")

• avg = orders_per_day / 86_400Per day ÷ 86,400 seconds = per second. In your head, divide by 1 lakh.
• peak = avg * peak_factorSize for the busiest hour, never the average.
• servers = math.ceil(peak / (per_server * target_load))Round up, and plan for each server to run at only 60% so spikes and failures are absorbed.

Run it on your PC: an estimate calibrated by your own CPU

Good estimates start from a measurement. This program first measures how many order events one core of your computer can parse and validate in a second, then uses that number to estimate the cores, nodes and storage needed for 10 crore orders a day. Only the standard library is used. Save it as capacity_estimator.py and run it.

"""Back-of-envelope capacity plan, calibrated with a real measurement on your PC.
Step 1 measures how many order events one CPU core can parse and validate per second.
Step 2 uses that number to estimate the machines needed for a festival peak."""
import json, math, time

# ---- assumptions (change these)
ORDERS_PER_DAY = 10_00_00_000      # 10 crore orders a day
PEAK_FACTOR = 5                     # peak hour is 5x the average rate
EVENTS_PER_ORDER = 20               # clicks, status updates, pings per order
BYTES_PER_EVENT = 500
CORES_PER_NODE = 16
HEADROOM = 0.6                      # run nodes at 60% so spikes and failures are absorbed

def inr(x):
    """Indian digit grouping: 115741 -> 1,15,741"""
    s = str(round(x))
    head, tail = s[:-3], s[-3:]
    while len(head) > 2:
        tail = head[-2:] + "," + tail
        head = head[:-2]
    return (head + "," + tail) if head else tail


# ---- step 1: measure one core
event = json.dumps({"order_id": "O1043", "user": "U9", "city": "Pune", "amount": 420,
                    "items": [{"dish": "thali", "qty": 1}, {"dish": "lassi", "qty": 2}], "ts": 1792411000})
n, start = 0, time.perf_counter()
while time.perf_counter() - start < 1.0:
    e = json.loads(event)
    ok = e["amount"] > 0 and e["city"] and len(e["items"]) > 0
    n += 1
per_core = n / (time.perf_counter() - start)
print(f"Measured: one core parses and checks about {inr(per_core)} events per second")

# ---- step 2: estimate
events_day = ORDERS_PER_DAY * EVENTS_PER_ORDER
avg_eps = events_day / 86_400
peak_eps = avg_eps * PEAK_FACTOR
cores = peak_eps / (per_core * HEADROOM)
nodes = math.ceil(cores / CORES_PER_NODE)
storage_tb_day = events_day * BYTES_PER_EVENT / 1024**4

print(f"\nEvents per day      : {events_day / 1e7:.0f} crore")
print(f"Average events / s  : {inr(avg_eps)}")
print(f"Peak events / s     : {inr(peak_eps)}")
print(f"Cores needed at peak: {cores:.1f}  (at {HEADROOM:.0%} target load)")
print(f"Nodes ({CORES_PER_NODE} cores each): {nodes}")
print(f"Raw event storage   : {storage_tb_day:.2f} TB per day, {storage_tb_day * 365:.0f} TB per year")
print("\nReal services do far more work per event (network, database, logging),")
print("so production capacity is usually 10-100x higher than this lower bound.")

# In a terminal:  python capacity_estimator.py

An autoscaler deciding, one line at a time

For three hours of traffic, the autoscaler picks enough servers to keep each one at about 60% load, never fewer than 2. Watch servers follow the load up and down.

Worked examples: real problems, solved and traced in memory

Four scaling and estimation problems, from a first estimate to choosing the best cluster size. Each runs live and is traced.

2 crore users × 5 visits × 8 calls, peak 4×

requests/day: 80 crore
average: 9,259/s  peak: 37,037/s

120 GB/day, +5% per month, 12 months, 3 copies

data for the year: 56.0 TB
with 3 copies   : 167.9 TB
daily volume at year end: 216 GB

6 hourly loads

servers per hour: [3, 8, 24, 30, 14, 3]
autoscaled server-hours: 82
fixed at peak          : 180

sizes 8, 16, 32, 48, 64 · a = 0.03 · b = 0.0008 · Rs 50/node-hour

 8 nodes:  63.8 jobs/min, Rs  104.6 per 1,000 jobs
16 nodes:  97.4 jobs/min, Rs  136.8 per 1,000 jobs
32 nodes: 117.5 jobs/min, Rs  227.0 per 1,000 jobs
48 nodes: 113.9 jobs/min, Rs  351.2 per 1,000 jobs
64 nodes: 104.7 jobs/min, Rs  509.6 per 1,000 jobs
fastest: 32 nodes | cheapest per job: 8 nodes

Check yourself

Quick revision

Interview and viva questions

Practice problems

Estimation and scaling exercises. The last two are business tasks.

4 crore/day

463

print(round(4_00_00_000 / 86_400))

1200, 6

7200

print(1200 * 6)

5 crore × 2 KB

95 GB

print(round(5_00_00_000 * 2 / 1024 / 1024), "GB")

9000, 500, 0.6

30

import math
print(math.ceil(9000 / (500 * 0.6)))

20,000 × 1.5 KB

29.3 MB/s = 234 Mbps

mb = 20000 * 1.5 / 1024
print(round(mb, 1), "MB/s =", round(mb * 8), "Mbps")

3000/s, 40 ms

busy: 120.0 | pool with 50% safety: 180

busy = 3000 * 0.040
print("busy:", busy, "| pool with 50% safety:", round(busy * 1.5))

40 shards, 0.5%

18.2 %

print(round((1 - (1 - 0.005) ** 40) * 100, 1), "%")

40 × 4 h + 10 × 20 h

fixed: Rs 28800 | autoscaled: Rs 10800 | saving: 62 %

fixed = 40 * 24 * 30
auto = (40 * 4 + 10 * 20) * 30
print("fixed: Rs", fixed, "| autoscaled: Rs", auto, "| saving:", round((1 - auto / fixed) * 100), "%")

18 lakh / 9 crore, 30 lakh / 18 crore

Rs 20.00 per 1,000 orders
Rs 16.67 per 1,000 orders

for cost, orders in [(18_00_000, 9_00_00_000), (30_00_000, 18_00_00_000)]:
    print(f"Rs {cost / orders * 1000:.2f} per 1,000 orders")

Real-world scenario questions (asked in company interviews)

Scaling and estimation questions from interviews and production incidents. Decide your answer first, then open the model answer.