HDFS federation

When one NameNode's memory is not enough

HDFS federation

Words you'll meet in this lesson

What we saw in the lab (Hadoop 3.5.0)

Federation at three levels

HA vs federation, and two ways to unify the view

Unifying the namespaces

Two NameNodes, three shared DataNodes

Our lab: NameNodes nn-canteen (nameservice nsc) and nn-hostel (nsh), and 3 DataNodes, all apache/hadoop:3.5.0. Commands run on datanode-1; full script in “Run it on your PC”.

1. Configure two nameservices, format with one cluster ID

   HDFS-SITE.XML_dfs.nameservices=nsc,nsh
   HDFS-SITE.XML_dfs.namenode.rpc-address.nsc=nn-canteen:8020
   HDFS-SITE.XML_dfs.namenode.rpc-address.nsh=nn-hostel:8020
   # on each NameNode, once:
   hdfs namenode -format -nonInteractive -clusterId campus-cluster

   nn-canteen: blockpoolID=BP-1200013783-172.21.0.3-1792598083981 clusterID=campus-cluster nn-hostel: blockpoolID=BP-343349822-172.21.0.2-1792598083339 clusterID=campus-cluster
   Same cluster ID, different block pool IDs. A DataNode accepts NameNodes only from its own cluster ID, so getting this right at format time is essential.

2. One tree for the client

   <name>fs.defaultFS</name><value>viewfs://campus</value>
   <name>fs.viewfs.mounttable.campus.link./canteen</name><value>hdfs://nn-canteen:8020/canteen</value>
   <name>fs.viewfs.mounttable.campus.link./hostel</name><value>hdfs://nn-hostel:8020/hostel</value>

   $ hdfs dfs -ls / Found 3 items -r-xr-xr-x - hadoop users 0 2026-10-21 15:58 /canteen -r-xr-xr-x - hadoop users 0 2026-10-21 15:58 /hostel -r-xr-xr-x - hadoop users 0 2026-10-21 15:58 /tmp $ hdfs dfs -ls /canteen /hostel -rw-r--r-- 3 hadoop supergroup 2488895 2026-10-21 15:58 /canteen/orders.csv -rw-r--r-- 3 hadoop supergroup 538894 2026-10-21 15:58 /hostel/rooms.csv
   The root / is not stored anywhere: ViewFS builds it from the mount table (hence the read-only r-x entries). Each mount point leads to a different NameNode.

3. Each NameNode knows only its own files

   hdfs dfs -ls -R hdfs://nn-canteen:8020/ ; echo ---; hdfs dfs -ls -R hdfs://nn-hostel:8020/

   drwxr-xr-x - hadoop supergroup 0 2026-10-21 15:58 hdfs://nn-canteen:8020/canteen -rw-r--r-- 3 hadoop supergroup 2488895 2026-10-21 15:58 hdfs://nn-canteen:8020/canteen/orders.csv --- drwxr-xr-x - hadoop supergroup 0 2026-10-21 15:58 hdfs://nn-hostel:8020/hostel -rw-r--r-- 3 hadoop supergroup 538894 2026-10-21 15:58 hdfs://nn-hostel:8020/hostel/rooms.csv drwxr-xr-x - hadoop supergroup 0 2026-10-21 15:55 hdfs://nn-hostel:8020/tmp
   Two completely separate directory trees, each in its own NameNode's memory.

4. Inside a DataNode: one folder per block pool

   ls /tmp/hadoop-hadoop/dfs/data/current/ ; du -sh /tmp/hadoop-hadoop/dfs/data/current/BP-*
   for nn in nn-canteen nn-hostel; do hdfs dfsadmin -fs hdfs://$nn:8020 -report | grep -m1 'Live datanodes'; done

   BP-1200013783-172.21.0.3-1792598083981 BP-343349822-172.21.0.2-1792598083339 VERSION 2.5M /tmp/hadoop-hadoop/dfs/data/current/BP-1200013783-172.21.0.3-1792598083981 572K /tmp/hadoop-hadoop/dfs/data/current/BP-343349822-172.21.0.2-1792598083339 nn-canteen: Live datanodes (3): nn-hostel: Live datanodes (3):
   The same disk holds both pools: 2.5 MB of canteen orders, 572 KB of hostel rooms. Both NameNodes see all three DataNodes, which heartbeat to each separately.

5. No rename across namespaces

   hdfs dfs -mv /canteen/orders.csv /hostel/orders.csv

   mv: Renames across Mount points not supported
   A rename is a metadata change inside one NameNode. Two NameNodes can't do it atomically together, so you must copy and delete. Plan the namespace split so that jobs don't need to move data across it.

6. Switch one NameNode off

   docker stop lab72-nn-canteen-1
   hdfs dfs -ls /hostel
   hdfs dfs -ls /canteen

   -rw-r--r-- 3 hadoop supergroup 538894 2026-10-21 15:58 /hostel/rooms.csv -ls: java.net.UnknownHostException: nn-canteen
   Failure isolation: the hostel namespace keeps working. The canteen namespace is unavailable (in our Docker lab the stopped container's name no longer resolves). In production, each nameservice would also be HA.

One client, two NameNodes, shared DataNodes

Six steps from our lab: writing to two mount points, looking at the DataNode, and losing one NameNode.

Federation at work

Simplified; organisations differ in tools and policies.

A mount table that routes paths to NameNodes

ViewFS is essentially a longest-prefix lookup: the path's mount point decides which NameNode answers. This program routes a few paths and refuses a rename across mount points, like our real cluster did.

mounts = {"/canteen": "hdfs://nn-canteen:8020/canteen",
          "/hostel":  "hdfs://nn-hostel:8020/hostel",
          "/tmp":     "hdfs://nn-hostel:8020/tmp"}

def resolve(path):
    best = max((m for m in mounts if path == m or path.startswith(m + "/")), key=len, default=None)
    if best is None:
        raise FileNotFoundError(path + " is not under any mount point")
    return mounts[best] + path[len(best):]

def rename(src, dst):
    s, d = resolve(src), resolve(dst)
    if s.split("/")[2] != d.split("/")[2]:
        return "mv: Renames across Mount points not supported"
    return f"renamed inside {s.split('/')[2]}"

for p in ["/canteen/orders.csv", "/hostel/rooms.csv", "/tmp/x"]:
    print(f"{p:22} -> {resolve(p)}")
print(rename("/canteen/orders.csv", "/hostel/orders.csv"))
print(rename("/hostel/rooms.csv", "/tmp/rooms.csv"))

• best = max((m for m in mounts if path == m or path.startswith(m + "/")), key=len, default=None)The longest matching mount point wins, so nested mounts (e.g. /data and /data/logs) work.
• if s.split("/")[2] != d.split("/")[2]:Different NameNodes behind source and target: no single NameNode can rename atomically. Our cluster printed exactly this message.
• print(rename("/hostel/rooms.csv", "/tmp/rooms.csv"))Interesting: /hostel and /tmp are on the same NameNode here, so the logic allows it. Real ViewFS is stricter by default and refuses renames between any two different mount points.

Run it on your PC: two NameNodes, three DataNodes

Create a folder hdfs-fed-lab with config, docker-compose.yaml, nnf.sh, and a subfolder client-conf containing the client core-site.xml (ViewFS mount table) shown below. Run chmod 777 client-conf, start with docker compose -p lab72 up -d --scale datanode=3, wait 45 seconds, then run federation_lab.sh.

x-hadoop: &hadoop
  image: apache/hadoop:3.5.0
  env_file: [./config]
  volumes: [./nnf.sh:/opt/nnf.sh:ro, ./client-conf:/opt/client-conf]
services:
  nn-canteen:
    <<: *hadoop
    hostname: nn-canteen
    command: ["bash", "/opt/nnf.sh"]
    environment: { HDFS-SITE.XML_dfs.nameservice.id: nsc }
  nn-hostel:
    <<: *hadoop
    hostname: nn-hostel
    command: ["bash", "/opt/nnf.sh"]
    environment: { HDFS-SITE.XML_dfs.nameservice.id: nsh }
  datanode:
    <<: *hadoop
    command: ["hdfs", "datanode"]
    depends_on: [nn-canteen, nn-hostel]

# ---- config ----
CORE-SITE.XML_fs.defaultFS=hdfs://nn-canteen:8020
HDFS-SITE.XML_dfs.nameservices=nsc,nsh
HDFS-SITE.XML_dfs.namenode.rpc-address.nsc=nn-canteen:8020
HDFS-SITE.XML_dfs.namenode.rpc-address.nsh=nn-hostel:8020
HDFS-SITE.XML_dfs.namenode.http-address.nsc=nn-canteen:9870
HDFS-SITE.XML_dfs.namenode.http-address.nsh=nn-hostel:9870
HDFS-SITE.XML_dfs.namenode.name.dir=/tmp/hadoop-hadoop/dfs/name
HDFS-SITE.XML_dfs.replication=3

# ---- nnf.sh ----
#!/bin/bash
# Format this NameNode once with the SHARED cluster ID, then run it.
NAME=/tmp/hadoop-hadoop/dfs/name
[ -d $NAME/current ] || hdfs namenode -format -nonInteractive -clusterId campus-cluster
exec hdfs namenode

# ---- client-conf/core-site.xml ----
<?xml version="1.0"?>
<!-- Client view of the federated cluster: one tree, two NameNodes behind it (ViewFS mount table). -->
<configuration>
  <property><name>fs.defaultFS</name><value>viewfs://campus</value></property>
  <property><name>fs.viewfs.mounttable.campus.link./canteen</name><value>hdfs://nn-canteen:8020/canteen</value></property>
  <property><name>fs.viewfs.mounttable.campus.link./hostel</name><value>hdfs://nn-hostel:8020/hostel</value></property>
  <property><name>fs.viewfs.mounttable.campus.link./tmp</name><value>hdfs://nn-hostel:8020/tmp</value></property>
</configuration>

#!/usr/bin/env bash
# federation_lab.sh: two NameNodes (canteen, hostel) share the same 3 DataNodes.
P=lab72
C="docker exec $P-datanode-1"
V='HADOOP_CONF_DIR=/opt/client-conf'          # client config with the ViewFS mount table
run() { echo "\$ $1"; $C bash -c "$1" 2>&1 | grep -v -E "WARN|^\s*$"; echo; }
$C bash -c 'cp /opt/hadoop/etc/hadoop/hdfs-site.xml /opt/hadoop/etc/hadoop/log4j.properties /opt/client-conf/'
run "hdfs dfs -mkdir -p hdfs://nn-canteen:8020/canteen hdfs://nn-hostel:8020/hostel hdfs://nn-hostel:8020/tmp"
run "seq 1 200000 | sed 's/^/order,/' > /tmp/orders.csv; seq 1 50000 | sed 's/^/room,/' > /tmp/rooms.csv"
run "$V hdfs dfs -put -f /tmp/orders.csv /canteen/ && $V hdfs dfs -put -f /tmp/rooms.csv /hostel/"
run "$V hdfs dfs -ls /"
run "$V hdfs dfs -ls /canteen /hostel"
run "hdfs dfs -ls -R hdfs://nn-canteen:8020/ ; echo ---; hdfs dfs -ls -R hdfs://nn-hostel:8020/"
for n in canteen hostel; do echo "nn-$n: $(docker exec $P-nn-$n-1 grep -E 'clusterID|blockpoolID' /tmp/hadoop-hadoop/dfs/name/current/VERSION | tr '\n' ' ')"; done; echo
run "ls /tmp/hadoop-hadoop/dfs/data/current/ ; du -sh /tmp/hadoop-hadoop/dfs/data/current/BP-*"
run "for nn in nn-canteen nn-hostel; do echo \$nn: \$(hdfs dfsadmin -fs hdfs://\$nn:8020 -report 2>/dev/null | grep -m1 'Live datanodes'); done"
run "$V hdfs dfs -mv /canteen/orders.csv /hostel/orders.csv"
echo ">>> docker stop $P-nn-canteen-1"; docker stop $P-nn-canteen-1 > /dev/null; echo
run "$V hdfs dfs -ls /hostel"
run "$V timeout 40 hdfs dfs -ls /canteen 2>&1 | grep -m1 -- -ls:"
docker start $P-nn-canteen-1 > /dev/null

# Run:  bash federation_lab.sh

A DataNode registering with two NameNodes

At start-up, a DataNode reads every nameservice from its configuration and registers with each, creating one block pool folder per NameNode.

Worked examples: real problems, solved and traced in memory

Three federation calculations: when one NameNode is too small, how to split, and what federation buys in failure isolation. Each runs live and is traced in memory.

192 GB · 150 B/object · 2.6 objects per file

1.37 billion objects -> about 529 million files

5 folders, 38 crore objects

nn1: [('/warehouse', 12), ('/user', 6)] 18 crore
nn2: [('/tmp', 9), ('/logs', 7), ('/ml', 4)] 20 crore

50% A · 30% B · 20% both

federated: 30% of jobs keep running; single NameNode: 0%

Check yourself

Quick revision

Interview and viva questions

Practice problems

Federation exercises using the real IDs, sizes and messages from the lab.

BP-1200013783-172.21.0.3-1792598083981

1200013783 172.21.0.3 1792598083981

bp = "BP-1200013783-172.21.0.3-1792598083981"
_, rnd, ip, ms = bp.split("-")
print(rnd, ip, ms)

/canteen, /hostel, /tmp mounts

nn-hostel

mounts = {"/canteen": "nn-canteen", "/hostel": "nn-hostel", "/tmp": "nn-hostel"}
print(next(nn for m, nn in mounts.items() if "/tmp/x".startswith(m + "/")))

2.5 MB, 0.572 MB

canteen 81%, hostel 19%

a, b = 2.5, 0.572
print(f"canteen {a / (a + b):.0%}, hostel {b / (a + b):.0%}")

120 GB · 50 crore

needs 70 GB -> fits

need = 500_000_000 * 150 / 1024**3
print(f"needs {need:.0f} GB -> {'fits' if need <= 120 else 'does not fit: federate or reduce objects'}")

3 moves

/canteen/a -> /canteen/archive/a : rename
/canteen/a -> /hostel/a : copy + delete
/hostel/r -> /hostel/old/r : rename

moves = [("/canteen/a", "/canteen/archive/a"), ("/canteen/a", "/hostel/a"), ("/hostel/r", "/hostel/old/r")]
mount = lambda p: "/" + p.split("/")[1]
for s, d in moves:
    print(s, "->", d, ":", "rename" if mount(s) == mount(d) else "copy + delete")

two lines

[3, 3] same

lines = ["nn-canteen: Live datanodes (3):", "nn-hostel: Live datanodes (3):"]
counts = [int(l.split("(")[1].split(")")[0]) for l in lines]
print(counts, "same" if len(set(counts)) == 1 else "different")

A ₹40 lakh · B 2 × ₹14 lakh + 3 × ₹3 lakh

A: Rs 40 lakh, B: Rs 37 lakh -> federation cheaper

a = 40
b = 2 * 14 + 3 * 3
print(f"A: Rs {a} lakh, B: Rs {b} lakh ->", "federation cheaper" if b < a else "bigger server cheaper")

2 nested mounts

/data/sales/x -> nn1
/data/logs/app.log -> nn2
/data/logsheet.csv -> nn1

mounts = {"/data": "nn1", "/data/logs": "nn2"}
def ns(path):
    return mounts[max((m for m in mounts if path == m or path.startswith(m + "/")), key=len)]
for p in ["/data/sales/x", "/data/logs/app.log", "/data/logsheet.csv"]:
    print(p, "->", ns(p))

Real-world scenario questions (asked in company interviews)

Federation questions appear in senior Hadoop administrator and platform architect interviews.