Scale out in practice: how big should a cluster be?

“How many servers should we buy?”

Scaling out and cluster sizing

Words you'll meet in this lesson

The sizing formulas

Rules of thumb vary by company and workload. These numbers are a sensible starting point, not a standard.

Sizing at three levels

Ways to grow a cluster

Few big nodes vs many small nodes (same total capacity)

Size TiffinGo's cluster for the next three years

1. Grow the data to the end of the planning period

   data_tb = 200 * 1.4 ** 3        # 200 TB growing 40% a year, for 3 years
   print(round(data_tb), "TB in year 3")

   549 TB in year 3
   Growth compounds: 200 → 280 → 392 → 549 TB. Sizing for today's data only means buying again in a few months.

2. Apply compression (if any)

   TiffinGo will store its clicks as compressed Parquet files, about 3× smaller: 549 ÷ 3 ≈ 183 TB on disk. Orders stay as they are. For simplicity, say 300 TB must be stored.

   Compression ratios depend on the data. Measure on a sample before you trust a number.

3. Multiply by replication

   print(300 * 3, "TB of replicas")

   900 TB of replicas
   Each block is on three DataNodes, on at least two racks. Every TB of business data uses 3 TB of disk.

4. Add temp space and headroom

   raw = 900 / (1 - 0.25 - 0.25)   # 25% temp space, 25% free headroom
   print(raw, "TB raw disk")

   1800.0 TB raw disk
   Half of each disk is reserved for in-between job data and safety margin. Running HDFS close to 100% full causes failed writes and slow jobs.

5. Turn disk into nodes

   import math
   print(math.ceil(1800 / 144), "storage nodes of 144 TB")

   13 storage nodes of 144 TB
   Always round up: 12.5 nodes means 13.

6. Check compute, then pick the bigger number

   The nightly report is 9,000 task-minutes; each node runs 16 tasks at once; the window is 2 hours. 9,000 ÷ (120 × 16) = 4.7, so 5 nodes are enough for compute. Storage needs 13, so storage wins.

   Add 2 spare nodes so that a failure or two doesn't break the plan: 15 nodes, which fit in 2 racks of 8 (better for rack awareness than 1 rack). Kabir has his answer, with every assumption written down.

Adding nodes to a full cluster: the dashboard, and what the blocks do

TiffinGo's cluster is getting full. Watch the admin add three nodes. Left: the dashboard. Right: where the blocks are.

Try it yourself: does a bigger cluster really run faster?

The same 4 GB file (32 blocks) is loaded on clusters of 3, 6 and 9 nodes. Each node can run 2 tasks at a time. Run the job on each size and compare the times in Job runs.

Where sizing decisions are made

Examples are simplified; figures in this lesson are illustrative. Real sizing also considers licences, support contracts, staff and data-centre limits.

A cluster sizing calculator

This function turns business requirements into a number of storage nodes, using the formulas from Part 2. Change the inputs and run it again, then try the challenges.

import math

def storage_nodes(data_tb, replication=3, temp=0.25, headroom=0.25, node_tb=144):
    raw_tb = data_tb * replication / (1 - temp - headroom)
    return raw_tb, math.ceil(raw_tb / node_tb)

for data in [50, 200, 500]:
    raw, nodes = storage_nodes(data)
    print(f"{data:>4} TB data -> {raw:>6.0f} TB raw -> {nodes:>3} nodes")

• def storage_nodes(data_tb, replication=3, temp=0.25, headroom=0.25, node_tb=144):Default values hold the usual assumptions, so a caller only passes what is different, for example storage_nodes(200, node_tb=240).
• raw_tb = data_tb * replication / (1 - temp - headroom)With the defaults, this is data × 3 ÷ 0.5 = data × 6.
• return raw_tb, math.ceil(raw_tb / node_tb)Return two values at once (a tuple). math.ceil rounds up: you cannot buy half a node.

Growing data year by year, one line at a time

Watch data grow 40% each time round the loop, and see the function called once per year to turn it into nodes. Each call gets its own frame.

Worked examples: real problems, solved and traced in memory

Three sizing problems from real platform planning: storage only, a deadline, and a three-year plan that adds nodes each year. Each runs live and is traced in memory.

80 TB · ×3 · ÷0.5 · 96 TB/node · 10 nodes/rack

Raw disk: 480.0 TB
Nodes: 5
Racks: 2

6,000 tasks × 5 min · 30 nodes × 12 slots · +20 min · 90 min window

30 nodes: 105 min
Need 36 nodes: 90 min

200 TB · +40%/year · ×6 · 144 TB/node · +2 spares · 3 years

Today: 11 nodes
Year 1: 280 TB, need 14, buy 3
Year 2: 392 TB, need 19, buy 5
Year 3: 549 TB, need 25, buy 6

Check yourself

Quick revision

Interview and viva questions

Practice problems

Sizing questions of the kind asked in data engineering interviews and in real capacity reviews. Work them out on paper first, then run the code.

50 TB · 3 · 0.25 · 0.25

300.0 TB

print(50 * 3 / (1 - 0.25 - 0.25), "TB")

1000 TB raw · 12 × 16 TB

6 nodes

import math
print(math.ceil(1000 / (12 * 16)), "nodes")

500 tasks · 25 × 8 slots

3 waves

import math
print(math.ceil(500 / (25 * 8)), "waves")

300 TB · 4× · ×6 · 144 TB

raw logs -> 13 nodes
Parquet -> 4 nodes

import math
for label, tb in [("raw logs", 300), ("Parquet", 300 / 4)]:
    print(label, "->", math.ceil(tb * 6 / 144), "nodes")

capacity 400 · 150 TB · +35%/year

Year 4 -> 498 TB

data, year = 150, 0
while data <= 400:
    data *= 1.35
    year += 1
print("Year", year, "->", round(data), "TB")

200 TB · 39 nodes × 100 MB/s

14.9 hours

mb = 200 * 1024 * 1024
rate = 39 * 100            # MB per second, all nodes together
print(round(mb / rate / 3600, 1), "hours")

40 nodes · Rs 35/h · 2 h vs 24 h · 30 days

Always on : Rs 10.08 lakh
Per job   : Rs 0.84 lakh
Saved     : Rs 9.24 lakh

rate, nodes, days = 35, 40, 30
always = nodes * 24 * rate * days
per_job = nodes * 2 * rate * days
print(f"Always on : Rs {always / 1e5:.2f} lakh")
print(f"Per job   : Rs {per_job / 1e5:.2f} lakh")
print(f"Saved     : Rs {(always - per_job) / 1e5:.2f} lakh")

120 TB · 30% × 2 years · 3 · 0.25 · 0.25 · 144 TB · +2 · 12/rack

Data in 2 years : 203 TB
Raw disk needed : 1217 TB
Nodes (+2 spare): 11
Racks           : 2

import math
data = 120 * 1.3 ** 2
raw = data * 3 / (1 - 0.25 - 0.25)
nodes = math.ceil(raw / 144) + 2
racks = max(2, math.ceil(nodes / 12))
print(f"Data in 2 years : {data:.0f} TB")
print(f"Raw disk needed : {raw:.0f} TB")
print(f"Nodes (+2 spare): {nodes}")
print(f"Racks           : {racks}")

Real-world scenario questions (asked in company interviews)

Sizing and scaling situations that come up in data engineering interviews. Decide your answer first, then compare.