Why companies run many clusters

One cluster was never going to be enough

A cluster fleet

Words you'll meet in this lesson

From our lab (Hadoop 3.5.0, two clusters)

Architecture at a glance

A typical fleet of an Indian bank or marketplace. Clusters are separate failure domains; the services along the bottom are shared by all of them.

Sizes are illustrative. Shared services are what make a fleet manageable; without them every cluster becomes its own island.

Many clusters at three levels

Kinds of clusters and fleet patterns

Fleet patterns

Federation or another cluster?

Animated example: from one cluster to a fleet

Building a two-cluster fleet in the lab

Our “prod” is the Module 8 cluster (lab8, with YARN). Our “DR” is the federation cluster of Lesson 7.2 (lab72). Scripts are in “Run it on your PC”.

1. First attempt: connect everything

   for c in namenode worker1 worker2 worker3 resourcemanager historyserver; do
     docker network connect lab5_default lab8-$c-1
   done
   hadoop distcp /prod/canteen hdfs://lab5-namenode-1:8020/dr/

   Retrying connect to server: resourcemanager/172.19.0.10:8032. Already tried 9 time(s) java.net.ConnectException: Call From namenode/172.22.0.2 to resourcemanager:8032 failed on connection exception: java.net.ConnectException: Connection refused
   A container on two networks answers to its name on both. The client got the ResourceManager's address on the second network, where nothing listens. Both clusters also had a host called namenode.

2. Fix: unique names, only workers join

   docker compose -p lab72 start                     # DR: nn-canteen, nn-hostel
   for c in worker1 worker2 worker3; do docker network connect lab72_default lab8-$c-1; done
   docker exec lab8-worker1-1 getent hosts namenode resourcemanager nn-canteen

   172.22.0.2 namenode 172.22.0.3 resourcemanager 172.21.0.2 nn-canteen
   The DR cluster's names (nn-canteen, nn-hostel) do not clash with prod's. Prod names still resolve to prod's network.

3. Take inventory of the fleet

   python3 /tmp/fleet.py

   cluster version clusterId live DNs files used prod (lab8) 3.5.0 CID-01fef50a-2caf-4bfb-9422-ad32ed9b0cb4 3 1680 4769 MB dr (lab72) ns canteen 3.5.0 campus-cluster 3 15 541 MB dr (lab72) ns hostel 3.5.0 campus-cluster 3 4 541 MB
   Two clusters, three NameNodes. The two DR namespaces share a cluster ID because they are federated over the same DataNodes (Lesson 7.2); prod has its own, so it fails independently.

4. What the fleet needs next

   copy data prod → DR            DistCp, snapshots           Lesson 12.2
   size each cluster              measured throughput          Lesson 12.3
   watch every cluster            JMX/REST + alerts            Lesson 12.4
   one identity, one rule book    Kerberos, Ranger             Lesson 12.5
   known fixes for known faults   runbooks                     Lesson 12.6

   The rest of this module turns our two-cluster lab into a small but properly operated fleet.

Two clusters, one network mistake

Six steps from our lab: the wrong way to join clusters, the fix, and the first fleet report.

Fleets at work

Simplified; organisations differ in tools, policies and regulations.

A fleet inventory and a cluster decision helper

The first part mimics fleet.py: one record per NameNode, then a summary per cluster ID (federated namespaces share one). The second part turns the reasons from this lesson into a simple decision helper.

fleet = [  # what fleet.py printed in our lab
    {"name": "prod (lab8)", "cluster": "CID-01fef50a", "version": "3.5.0", "dns": 3, "files": 1680, "used_mb": 4769},
    {"name": "dr ns canteen", "cluster": "campus-cluster", "version": "3.5.0", "dns": 3, "files": 15, "used_mb": 541},
    {"name": "dr ns hostel", "cluster": "campus-cluster", "version": "3.5.0", "dns": 3, "files": 4, "used_mb": 541},
]
clusters = {}
for nn in fleet:
    c = clusters.setdefault(nn["cluster"], {"namenodes": 0, "files": 0, "used_mb": nn["used_mb"], "dns": nn["dns"]})
    c["namenodes"] += 1
    c["files"] += nn["files"]
for cid, c in clusters.items():
    print(f"{cid:15} {c['namenodes']} NameNode(s)  {c['dns']} DataNodes  {c['files']:>5} files  {c['used_mb']:>5} MB")
versions = {nn["version"] for nn in fleet}
print("versions in the fleet:", versions, "-> no drift" if len(versions) == 1 else "-> DRIFT")

• c = clusters.setdefault(nn["cluster"], {"namenodes": 0, "files": 0, "used_mb": nn["used_mb"], "dns": nn["dns"]})Federated namespaces share DataNodes and the cluster ID, so we count their storage once.
• versions = {nn["version"] for nn in fleet}A fleet report always checks for version drift: one forgotten cluster on an old release is a classic incident.

Run it on your PC: a two-cluster fleet

Start the Module 8 lab (prod) and the Lesson 7.2 federation lab (DR). join_clusters.sh connects only the prod workers to the DR network, then fleet.py runs on a worker, which can see both clusters. Run docker cp fleet.py lab8-worker1-1:/tmp/ first.

# fleet.py: an inventory of every cluster we run - the first thing a platform team builds.
import json, urllib.request
NAMENODES = {"prod (lab8)": "http://namenode:9870",
             "dr (lab72) ns canteen": "http://nn-canteen:9870",
             "dr (lab72) ns hostel": "http://nn-hostel:9870"}
def jmx(base, name):
    return json.load(urllib.request.urlopen(f"{base}/jmx?qry=Hadoop:service=NameNode,name={name}", timeout=5))["beans"][0]
print(f"{'cluster':24} {'version':8} {'clusterId':38} {'live DNs':>8} {'files':>7} {'used':>9}")
for name, base in NAMENODES.items():
    info, st = jmx(base, "NameNodeInfo"), jmx(base, "FSNamesystemState")
    print(f"{name:24} {info['Version'].split(',')[0]:8} {info['ClusterId']:38} {st['NumLiveDataNodes']:>8} "
          f"{st['FilesTotal']:>7} {st['CapacityUsed'] / 2**20:>7.0f} MB")

# ---- join_clusters.sh (run on your PC) ----
# # join_clusters.sh: let the lab8 workers (where DistCp's map tasks run) reach the second cluster.
# # Only the workers join: the clusters both use names like "namenode", and a container on two networks
# # can answer to the wrong address. Run on your PC.
# docker compose -p lab72 start                       # the Lesson 7.2 federation cluster = our "DR" site
# for c in worker1 worker2 worker3; do docker network connect lab72_default lab8-$c-1; done
# docker exec lab8-worker1-1 getent hosts namenode resourcemanager nn-canteen
# docker cp lab8-namenode-1:/tmp/mr/mr.jar /tmp/mr.jar && docker cp /tmp/mr.jar lab8-worker1-1:/tmp/mr.jar

# Then:
#   docker cp fleet.py lab8-worker1-1:/tmp/
#   docker exec lab8-worker1-1 python3 /tmp/fleet.py

Grouping NameNodes into clusters

Three NameNodes, two clusters: the cluster ID decides which ones belong together.

Worked examples: real problems, solved and traced in memory

Four fleet decisions from different industries: when to add a cluster, what DR costs, and how many clusters to run.

problem: full HDFS, not CPU

HDFS filled            -> space quota
CPU contention         -> queue

3 PB · 25% critical · replication 3 · 80% fill

critical data 0.75 PB -> 2.81 PB of raw disk in the DR cluster

in place 6 h + 20% × 4 h · blue-green 2 h

in place 6.8 h expected downtime, blue-green 2 h (plus a second cluster for a month)

5 needs

['dev', 'dr', 'prod', 'sandbox'] -> 4 clusters, plus staging for upgrades = 5

Check yourself

Quick revision

Interview and viva questions

Practice problems

Fleet exercises using our lab numbers and realistic company scenarios.

3 lines, 2 cluster IDs

2 clusters, 3 NameNodes

ids = ["CID-01fef50a", "campus-cluster", "campus-cluster"]
print(len(set(ids)), "clusters,", len(ids), "NameNodes")

1,680 · 15 · 4

prod: 1680 dr: 19

print("prod:", 1680, "dr:", 15 + 4)

3 needs

CPU at night         -> queue
fills the disks      -> space quota
city-wide power cut  -> DR cluster

for need, fix in [("CPU at night", "queue"), ("fills the disks", "space quota"), ("city-wide power cut", "DR cluster")]:
    print(f"{need:20} -> {fix}")

3 clusters

{'namenode': ['lab8', 'lab5'], 'datanode': ['lab5', 'lab72']}

clusters = {"lab8": ["namenode", "resourcemanager", "worker1"], "lab5": ["namenode", "datanode"],
            "lab72": ["nn-canteen", "nn-hostel", "datanode"]}
seen = {}
for c, hosts in clusters.items():
    for h in hosts:
        seen.setdefault(h, []).append(c)
print({h: cs for h, cs in seen.items() if len(cs) > 1})

2 PB · shares

9.96 PB raw

data = 2.0
shares = {"prod": 1.0, "dr": 0.5, "sandbox": 0.15, "staging": 0.01}
print(f"{sum(data * s * 3 for s in shares.values()):.2f} PB raw")

5 clusters

majority 3.5.0 | drifted: ['dev', 'sandbox']

from collections import Counter
v = {"prod": "3.5.0", "dr": "3.5.0", "staging": "3.5.0", "dev": "3.4.1", "sandbox": "3.3.6"}
major = Counter(v.values()).most_common(1)[0][0]
print("majority", major, "| drifted:", [c for c, x in v.items() if x != major])

₹40 crore · 1% · ₹1.2 crore

expected loss Rs 0.4 crore/year vs DR Rs 1.2 crore/year

expected_loss = 40 * 0.01
print(f"expected loss Rs {expected_loss:.1f} crore/year vs DR Rs 1.2 crore/year")

50 nodes · 3 h · ₹60 vs ₹90

always-on Rs 2,160,000, ephemeral Rs 438,750 per month

always = 50 * 24 * 30 * 60
ephemeral = 50 * (3 + 0.25) * 30 * 90
print(f"always-on Rs {always:,.0f}, ephemeral Rs {ephemeral:,.0f} per month")

Real-world scenario questions (asked in company interviews)

Fleet questions appear in senior data-engineering and platform interviews.