Cluster sizing with real numbers

“How many servers should we buy?”

Cluster sizing

Words you'll meet in this lesson

From our lab (TestDFSIO, 64 MB files, replication 3, 3 workers on one laptop)

Architecture at a glance: the cluster we are sizing

The UPI example, sized. Typical hardware values (disks, RAM, NICs) are assumptions for illustration.

Sizing at three levels

The levers that change a sizing

Roles to size

Animated example: sizing the payments cluster

Benchmark first, then size

Run on the Module 8 lab with dfsio_lab.sh (in “Run it on your PC”), then do the arithmetic for the payments example.

1. Benchmark writes with one task

   yarn org.apache.hadoop.fs.TestDFSIO -libjars $T -write -nrFiles 1 -fileSize 64MB

   Total MBytes processed: 64 Throughput mb/sec: 113.48 Test exec time sec: 17.6
   One map task writes one 64 MB file with three replicas. 113 MB/s is what one writer gets from our laptop's disk, through the DataNode pipeline.

2. Add writers

   write, 3 files: Throughput mb/sec: 61.82
   write, 6 files: Throughput mb/sec: 34.99

   Per task the number drops; the total grows from about 113 to 185 to 210 MB/s and then flattens. Three “workers” sharing one disk can't add disk speed. On real servers, each node adds its own disks and the total keeps growing.

3. Benchmark reads

   read, 1 file:  Throughput mb/sec: 320
   read, 3 files: Throughput mb/sec: 175.34
   read, 6 files: Throughput mb/sec: 130.3

   Reads look faster because the files were written seconds earlier and are still in memory (page cache). For an honest read benchmark, use files much larger than RAM or drop the caches first.

4. Storage for the payments company

   2 TB × 0.25 = 0.5 TB/day × 1,095 days = 547.5 TB
   × 3 copies = 1,642.5 TB × 1.25 temp ÷ 0.70 fill × 1.3 growth = 3,812.9 TB
   ÷ 96 TB per worker (12 × 8 TB) = 39.7 → 40 workers

   Storage decides the cluster size. The compression ratio and disk per node are assumptions; measure the ratio on a sample of your real data (Lesson 10.2 measured about 0.14 for Parquet + zstd on our CSV, so 0.25 is a cautious choice).

5. Check NameNode heap and compute

   files: 2,048 a day × 1,095 = 2,242,560; blocks: 4,485,120
   objects × 150 B ≈ 0.9 GB  (rule of thumb; plan a much larger heap)
   nightly work: 60 TB ÷ 20 MB/s = 3,145,728 container-seconds
   ÷ 10,800 s (3 h) = 291 containers at once; available 40 × 30 = 1,200

   Both fit with room to spare. If either did not, it would add nodes (compute), larger files (heap) or a second namespace (Lesson 7.2).

6. Masters, edge and network

   masters: 3 servers (2 NN, 2 RM, 3 JN, 3 ZK, JobHistory), one per rack
   edge: 2 servers
   racks: 45 servers ÷ 20 per rack = 3 racks
   oversubscription: 20 × 10 Gb ÷ 80 Gb uplink = 2.5 : 1

   Spreading masters across racks means a single rack failure can't take the NameNode pair or the JournalNode quorum down (Lesson 7.1).

Why more writers didn't mean more speed

Six steps of our TestDFSIO runs: what the tool printed and what was happening on the one shared disk.

Sizing at work

Simplified; organisations differ in tools and policies. Hardware figures in this lesson are assumptions for illustration.

A cluster sizing calculator

The assumptions are at the top. Change one and see which part of the cluster moves. Storage usually decides the number of workers; compute checks that the nightly work fits the window.

import math
# --- assumptions (change these) ---
RAW_TB_DAY, RATIO, DAYS = 2.0, 0.25, 3 * 365     # raw data per day, Parquet+zstd size ratio, retention
REPL, TEMP, FILL, GROWTH = 3, 1.25, 0.70, 1.3     # replication, +25% temp/shuffle, keep disks 70% full, +30%
DISK_TB_PER_NODE = 12 * 8                         # 12 disks of 8 TB
NIGHTLY_SCAN_TB, MB_S_PER_CONTAINER, WINDOW_H = 60, 20, 3

stored = RAW_TB_DAY * RATIO * DAYS                # TB kept, one copy
data = stored * REPL
disk = data * TEMP / FILL * GROWTH
workers = math.ceil(disk / DISK_TB_PER_NODE)
print(f"one copy {stored:.1f} TB -> x{REPL} = {data:.1f} TB -> disk to buy {disk:.1f} TB -> {workers} workers")

files = RAW_TB_DAY * RATIO * 1024 * 1024 / 256 * DAYS   # 256 MB files
blocks = files * 2                                      # 128 MB blocks
print(f"{files:,.0f} files, {blocks:,.0f} blocks -> about {(files + blocks) * 150 / 2**30:.1f} GB of NameNode heap")

containers = NIGHTLY_SCAN_TB * 1024 * 1024 / MB_S_PER_CONTAINER / (WINDOW_H * 3600)
print(f"nightly work needs {containers:.0f} containers at once; {workers} workers x 30 = {workers * 30}")

• disk = data * TEMP / FILL * GROWTHTemp space for shuffle and staging, headroom so disks never pass 70%, and a year of growth.
• blocks = files * 2 # 128 MB blocksHeap depends on the number of files and blocks, not on bytes (Lesson 7.5).
• containers = NIGHTLY_SCAN_TB * 1024 * 1024 / MB_S_PER_CONTAINER / (WINDOW_H * 3600)Total work in container-seconds divided by the seconds you have.

Run it on your PC: benchmark your own cluster

TestDFSIO ships with Hadoop in the MapReduce tests jar. Each map task writes (or reads) one file; the result is the throughput per task. Run it inside the Module 8 lab's namenode container. It takes about 3 minutes and removes its files with -clean at the end.

# dfsio_lab.sh: measure what our cluster can write and read (TestDFSIO), the input to any sizing exercise.
# Run inside lab8-namenode-1.
T=$(ls /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar)
export HADOOP_CLASSPATH=$T
for files in 1 3 6; do
  for op in write read; do
    yarn org.apache.hadoop.fs.TestDFSIO -libjars $T -$op -nrFiles $files -fileSize 64MB -resFile /tmp/dfsio.txt > /tmp/dfsio.log 2>&1
    echo "== $op, $files file(s) x 64 MB in parallel"
    grep -E "Total MBytes processed|Throughput mb/sec|Average IO rate|Test exec time" /tmp/dfsio.log | sed -E 's/^.*TestDFSIO:[0-9]+ - +/   /'
  done
done
yarn org.apache.hadoop.fs.TestDFSIO -libjars $T -clean > /dev/null 2>&1

# On your PC, in the mr-lab folder:
#   docker cp dfsio_lab.sh lab8-namenode-1:/tmp/
#   docker exec lab8-namenode-1 bash /tmp/dfsio_lab.sh

From terabytes a day to workers

Follow the storage calculation one multiplication at a time.

Worked examples: real problems, solved and traced in memory

Four sizing problems from different industries. Every assumption is written down, because a sizing is only as good as its assumptions.

800 GB/day · 2 years · ORC 20% · 3 copies

116.8 TB stored, 350.4 TB with copies, 625.7 TB disk -> 6 workers

4,00,000 files/day × 300 KB · 5 years

small files: 204 GB heap; packed: 0.26 GB heap

45 TB · 3 hours · 25 MB/s per container · 900 containers

needs 175 containers at once, has 900 -> 19% of the cluster

300 GB/day in year 1 · +40%/year · keep forever

year 1:   109.5 TB kept ->   7 workers
year 2:   262.8 TB kept ->  15 workers
year 3:   477.4 TB kept ->  27 workers
year 4:   777.9 TB kept ->  44 workers
year 5:  1198.5 TB kept ->  67 workers

Check yourself

Quick revision

Interview and viva questions

Practice problems

Sizing exercises. Write your assumptions first, then calculate.

10 TB

replication 3: 30 TB   RS-6-3: 15.0 TB

print("replication 3:", 10 * 3, "TB   RS-6-3:", 10 * 1.5, "TB")

per-task throughput × tasks

185.5 209.9 MB/s

print(round(3 * 61.82, 1), round(6 * 34.99, 1), "MB/s")

2 TB/day, no compression

15252 TB -> 159 workers

import math
disk = 2.0 * 1095 * 3 * 1.25 / 0.70 * 1.3
print(f"{disk:.0f} TB -> {math.ceil(disk / 96)} workers")

224 GB, 30 vcores

7.5 GB per vcore

print(f"{224 / 30:.1f} GB per vcore")

0.5 TB × 3 in 2 hours

218 MB/s across the cluster

mb = 0.5 * 1024 * 1024 * 3
print(f"{mb / 7200:.0f} MB/s across the cluster")

45 servers, 20 per rack

3 racks

import math
print(math.ceil(45 / 20), "racks")

₹6 crore / 3 years vs ₹1,500 × 40 nodes per hour

break-even at 0.9 cluster-hours a day

own_per_day = 6e7 / (3 * 365)
cloud_per_hour = 1500 * 40
print(f"break-even at {own_per_day / cloud_per_hour:.1f} cluster-hours a day")

2.0, 0.25, 1095, 3

40 20

import math
def size(raw_tb_day, ratio, days, copies, node_tb=96):
    disk = raw_tb_day * ratio * days * copies * 1.25 / 0.70 * 1.3
    return math.ceil(disk / node_tb)
print(size(2.0, 0.25, 1095, 3), size(2.0, 0.25, 1095, 1.5))

Real-world scenario questions (asked in company interviews)

Sizing questions are common for experienced roles. Interviewers look for a method and stated assumptions, not a memorised number.