NameNode and DataNode

1 The story

“Where is my file, really?”

Riya uploaded canteen_orders.csv to HDFS yesterday. Today she asks a simple question nobody in the lab can answer: on which disk, of which machine, are those bytes? Kabir opens a shell on one DataNode, digs through folders, and finds a file called blk_1073741825. He runs head on it. Out come order_id,item,amount and Riya's orders, as plain text.

Riya

So HDFS is just… files on ordinary Linux disks?

Arjun Sir

Exactly. A DataNode stores each block as an ordinary file in its own folders. What makes it a distributed file system is the NameNode: it keeps the whole directory tree, who owns what, and which blocks make up each file. And it knows, from the DataNodes' own reports, which machines hold each block.

Kabir

And if I lose the NameNode?

Arjun Sir

The bytes are still on the disks, but nobody knows which blk_ file belongs to which file. That's why the NameNode's metadata is protected so carefully, and why HA exists. Today we open both: the NameNode's metadata folder and a DataNode's block folder, on a real cluster with four DataNodes in two racks.

2 Definition

NameNode and DataNode

Formal definition (write this in exams)

HDFS has a master–worker architecture. The NameNode manages the file system namespace (the tree of directories and files with their permissions, owners, replication factor and the ordered list of blocks of each file) and regulates client access. It persists the namespace in the fsimage and edit log, and keeps it in memory together with the block map (block → DataNodes holding a replica), which it rebuilds from DataNode block reports. DataNodes, usually one per worker machine, store blocks as files on their local disks (each with a .meta checksum file), serve read and write requests from clients directly, create, delete and copy replicas when the NameNode instructs, and send heartbeats and block reports to the NameNode. File data never passes through the NameNode.

In simple words

The NameNode is the index of a huge library: it knows every book's title and on which shelves its pages are. DataNodes are the shelves that hold the pages. Readers ask the index where to go, then fetch pages straight from the shelves.

Words you'll meet in this lesson

NamespaceThe directory tree: paths, owners, permissions, replication, and each file's list of blocks.

Block mapWhich DataNodes hold each block. Kept in NameNode memory only, rebuilt from block reports.

blk_ fileA block replica on a DataNode's disk, e.g. blk_1073741825. Plain bytes of your file.

.meta fileChecksums for the block: 4 bytes for every 512 bytes of data.

Block poolThe set of blocks belonging to one namespace, e.g. BP-1507681208-…. A folder on each DataNode.

Cluster IDCreated when the NameNode is formatted; every DataNode stores it and must match.

dfsadmin -reportCapacity and status of the whole cluster and of each DataNode.

fsckChecks files and lists their blocks and replica locations.

Who keeps what (checked on Hadoop 3.5.0)

NameNode on disk	`dfs.namenode.name.dir`: `fsimage_`, `edits_`, `seen_txid`, `VERSION`
NameNode in memory	Whole namespace plus block map; a common rule of thumb is about 150 bytes per file, directory or block
DataNode on disk	`dfs.datanode.data.dir`/current/BP-…/current/finalized/…/`blk_N` + `blk_N_genstamp.meta`
Checksums	`dfs.bytes-per-checksum` = 512; CRC, 4 bytes each
Ports	NameNode RPC 8020, web 9870 · DataNode data 9866, IPC 9867, web 9864
Per cluster	One active NameNode per namespace (Lesson 7.1 adds a standby); many DataNodes

NameNode and DataNode at three levels

Pick your level. New to this? Start at Beginner and climb.

Beginner The library and its index

A big book is torn into chapters (blocks) and each chapter is photocopied three times onto different shelves (DataNodes).
The index card (NameNode) says: “Canteen orders: chapter 1 on shelves 1, 2 and 4.”
A reader reads the card, then walks to the shelf. The librarian at the index never carries pages.

Intermediate What each daemon does

NameNode: create/delete/rename files and folders, check permissions, choose DataNodes for new blocks, track replicas, order re-replication when one is lost.
DataNode: store and checksum blocks, stream data to and from clients, forward blocks in the write pipeline, delete or copy blocks on command.
Client (your hdfs dfs, Spark, Hive): asks the NameNode for metadata, then talks to DataNodes for data.

Advanced Details that matter in production

The block map is not saved in the fsimage. After a restart the NameNode waits for block reports (safe mode, Lesson 5.6).
NameNode heap grows with the number of objects, not bytes: 10 crore tiny files hurt far more than 1 PB in big files (the small-files problem, Lesson 7.5).
DataNodes spread blocks over all disks listed in dfs.datanode.data.dir; a failed disk can be tolerated (dfs.datanode.failed.volumes.tolerated).
Clusters with very large namespaces split it across several NameNodes with federation; each namespace gets its own block pool on the same DataNodes.

Exam tip: “Explain the roles of NameNode and DataNode”: NameNode = master, namespace + metadata (fsimage, edit log, block map), handles client metadata requests, directs replication; DataNode = worker, stores blocks, serves reads/writes, sends heartbeats (every 3 s) and block reports. Add: data flows client ↔ DataNode, never through the NameNode.

3 Types and ways to use it

NameNode vs DataNode, side by side

	NameNode	DataNode
Role	Master: metadata	Worker: data
How many	1 active per namespace (+ standby in HA)	Tens to thousands
Stores	Namespace in memory; fsimage + edits on disk	Block files + checksums on local disks
Hardware	Lots of RAM, reliable disks (often RAID), redundant power	Many large disks (JBOD), moderate RAM
If it fails	Whole file system unavailable (without HA)	Routine: replicas elsewhere, re-replication
Talks to	Clients (metadata), DataNodes (commands)	Clients (data), NameNode (heartbeats, reports), other DataNodes (pipeline)

Other HDFS daemons you'll see

SecondaryNameNode

Merges the edit log into a new fsimage (checkpoint). Not a backup NameNode. Lesson 5.4.

Checkpoint

Standby NameNode

A hot spare in HA that takes over automatically. Lesson 7.1.

High availability

JournalNodes

Store the shared edit log for HA NameNodes (usually 3 or 5).

Router (RBF)

Router-based federation: one view over several namespaces.

Federation

When something is wrong with HDFS, ask two questions first: is it a metadata problem (NameNode: permissions, safe mode, quota, heap) or a data problem (DataNodes: dead nodes, full disks, missing blocks)?

4 Step by step

Open the NameNode and a DataNode on a real cluster

Module 5 uses a slightly bigger practice cluster than Lesson 4.3: 4 DataNodes in 2 racks plus a SecondaryNameNode. Every output on this page and in the rest of Module 5 comes from this cluster.

Start the Module 5 lab cluster

services:
  namenode:
    image: apache/hadoop:3.5.0
    hostname: namenode
    command: ["hdfs", "namenode"]
    ports:
      - 9870:9870
    env_file:
      - ./config
    environment:
      ENSURE_NAMENODE_DIR: "/tmp/hadoop-hadoop/dfs/name"
    volumes:
      - ./topology.sh:/opt/topology.sh:ro
  secondarynamenode:
    image: apache/hadoop:3.5.0
    hostname: secondarynamenode
    command: ["hdfs", "secondarynamenode"]
    env_file:
      - ./config
  datanode:
    image: apache/hadoop:3.5.0
    command: ["hdfs", "datanode"]
    env_file:
      - ./config

CORE-SITE.XML_fs.defaultFS=hdfs://namenode
CORE-SITE.XML_net.topology.script.file.name=/opt/topology.sh
HDFS-SITE.XML_dfs.namenode.rpc-address=namenode:8020
HDFS-SITE.XML_dfs.replication=3
HDFS-SITE.XML_dfs.namenode.secondary.http-address=secondarynamenode:9868
HDFS-SITE.XML_dfs.namenode.checkpoint.period=120
HDFS-SITE.XML_dfs.namenode.heartbeat.recheck-interval=15000

#!/bin/bash
# Rack topology script: Hadoop passes IP addresses, we print one rack per argument.
for host in "$@"; do
  name=$(getent hosts "$host" | awk '{print $2}')
  n=$(echo "$name" | grep -o 'datanode-[0-9]*' | grep -o '[0-9]*$')
  if [ -z "$n" ]; then echo "/default-rack"
  elif [ "$n" -le 2 ]; then echo "/rack1"
  else echo "/rack2"; fi
done

chmod +x topology.sh
docker compose up -d --scale datanode=4
docker compose exec namenode hdfs dfsadmin -printTopology

Rack: /rack1 172.19.0.4:9866 (lab5-datanode-2.lab5_default) In Service 172.19.0.5:9866 (lab5-datanode-1.lab5_default) In Service Rack: /rack2 172.19.0.7:9866 (lab5-datanode-4.lab5_default) In Service 172.19.0.6:9866 (lab5-datanode-3.lab5_default) In Service

Two settings are deliberately shortened so you can watch things happen: checkpoints every 2 minutes (default 1 hour, Lesson 5.4) and the dead-node check every 15 s (default 5 minutes, Lesson 5.5). Stop the Lesson 4.3 cluster first; both use port 9870.

Store a file twice: normal blocks and 1 MB blocks

docker compose cp canteen_orders.csv namenode:/tmp/canteen_orders.csv
docker compose exec namenode hdfs dfs -mkdir -p /user/hadoop/data
docker compose exec namenode hdfs dfs -put /tmp/canteen_orders.csv data/
docker compose exec namenode hdfs dfs -D dfs.blocksize=1048576 -put /tmp/canteen_orders.csv data/canteen_1mb.csv
docker compose exec namenode hdfs dfs -ls data

Found 2 items -rw-r--r-- 3 hadoop supergroup 4692009 2026-10-21 06:49 data/canteen_1mb.csv -rw-r--r-- 3 hadoop supergroup 4692009 2026-10-21 06:49 data/canteen_orders.csv

Same 4.5 MB file (from make_canteen.py, Lesson 4.3), stored as 1 block and as 5 blocks. The 3 in the listing is the replication factor.

Ask the NameNode about the cluster
```
docker compose exec namenode hdfs dfsadmin -report
```
Configured Capacity: 1082212696064 (1007.89 GB) Present Capacity: 86452563116 (80.52 GB) DFS Remaining: 86424092672 (80.49 GB) DFS Used: 28470444 (27.15 MB) DFS Used%: 0.03% Replicated Blocks: Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 … Live datanodes (4): Name: 172.19.0.5:9866 (lab5-datanode-1.lab5_default) Hostname: ab3b1bedf98a Rack: /rack1 Decommission Status : Normal Configured Capacity: 270553174016 (251.97 GB) DFS Used: 9481956 (9.04 MB) Non DFS Used: 24661946652 (22.97 GB) DFS Remaining: 21606023168 (20.12 GB) … Last contact: Wed Oct 21 06:49:41 UTC 2026 Last Block Report: Wed Oct 21 06:49:03 UTC 2026 Num of Blocks: 6
The NameNode knows every DataNode's rack, capacity, last heartbeat (“Last contact”) and how many blocks it holds. All four containers share one physical disk here, so “capacity” is counted four times; on real servers each DataNode reports its own disks.
Ask where a file's blocks are
```
docker compose exec namenode hdfs fsck /user/hadoop/data/canteen_1mb.csv -files -blocks -locations -racks
```
/user/hadoop/data/canteen_1mb.csv 4692009 bytes, replicated: replication=3, 5 block(s): OK 0. BP-1507681208-172.19.0.3-1792565333483:blk_1073741826_1002 len=1048576 Live_repl=3 [/rack1/172.19.0.5:9866, /rack2/172.19.0.6:9866, /rack2/172.19.0.7:9866] 1. BP-1507681208-172.19.0.3-1792565333483:blk_1073741827_1003 len=1048576 Live_repl=3 [/rack1/172.19.0.5:9866, /rack1/172.19.0.4:9866, /rack2/172.19.0.7:9866] 2. BP-1507681208-172.19.0.3-1792565333483:blk_1073741828_1004 len=1048576 Live_repl=3 [/rack2/172.19.0.6:9866, /rack2/172.19.0.7:9866, /rack1/172.19.0.5:9866] 3. BP-1507681208-172.19.0.3-1792565333483:blk_1073741829_1005 len=1048576 Live_repl=3 [/rack2/172.19.0.6:9866, /rack2/172.19.0.7:9866, /rack1/172.19.0.5:9866] 4. BP-1507681208-172.19.0.3-1792565333483:blk_1073741830_1006 len=497705 Live_repl=3 [/rack1/172.19.0.5:9866, /rack1/172.19.0.4:9866, /rack2/172.19.0.6:9866] Status: HEALTHY Number of data-nodes: 4 Number of racks: 2
This is the NameNode's map for one file: block ID, length, and the three DataNodes (with racks) holding each replica. BP-… is the block pool, _1002 the generation stamp.
Look inside the NameNode's folder
```
docker compose exec namenode ls -l /tmp/hadoop-hadoop/dfs/name/current
docker compose exec namenode cat /tmp/hadoop-hadoop/dfs/name/current/VERSION
```
-rw-r--r-- 1 hadoop users 1048576 Oct 21 06:49 edits_inprogress_0000000000000000001 -rw-r--r-- 1 hadoop users 401 Oct 21 06:48 fsimage_0000000000000000000 -rw-r--r-- 1 hadoop users 62 Oct 21 06:48 fsimage_0000000000000000000.md5 -rw-r--r-- 1 hadoop users 2 Oct 21 06:49 seen_txid -rw-r--r-- 1 hadoop users 215 Oct 21 06:48 VERSION #Wed Oct 21 06:48:53 UTC 2026 namespaceID=1092532876 blockpoolID=BP-1507681208-172.19.0.3-1792565333483 storageType=NAME_NODE cTime=1792565333485 clusterID=CID-570b3f1b-005e-4977-a8e0-3476c40d28cc layoutVersion=-67
No file contents at all: a 401-byte empty fsimage from the format, and the edit log recording every change since (Lesson 5.4). The metadata of all our files fits in a few kilobytes.
Look inside a DataNode's folder
```
docker compose exec --index 1 datanode bash -c 'find /tmp/hadoop-hadoop/dfs/data -name "blk_*" | sort | xargs ls -l'
docker compose exec --index 1 datanode cat /tmp/hadoop-hadoop/dfs/data/current/VERSION
```
-rw-r--r-- 1 hadoop users 4692009 … /current/BP-1507681208-172.19.0.3-1792565333483/current/finalized/subdir0/subdir0/blk_1073741825 -rw-r--r-- 1 hadoop users 36667 … /subdir0/subdir0/blk_1073741825_1001.meta -rw-r--r-- 1 hadoop users 1048576 … /subdir0/subdir0/blk_1073741826 -rw-r--r-- 1 hadoop users 8199 … /subdir0/subdir0/blk_1073741826_1002.meta … -rw-r--r-- 1 hadoop users 497705 … /subdir0/subdir0/blk_1073741830 -rw-r--r-- 1 hadoop users 3899 … /subdir0/subdir0/blk_1073741830_1006.meta #Wed Oct 21 06:49:01 UTC 2026 datanodeUuid=8c5d6579-638f-40fd-b59a-17ed6563d848 storageType=DATA_NODE cTime=0 clusterID=CID-570b3f1b-005e-4977-a8e0-3476c40d28cc layoutVersion=-57 storageID=DS-291fa99b-6a48-43ad-9769-bad688411060
Each replica is two ordinary files: the data and its checksums. Same clusterID as the NameNode, or the DataNode would refuse to start (Lesson 4.3). Paths shortened with “…”.
Read a block file directly
```
head -3 blk_1073741826
```
order_id,item,amount 1001,samosa,18 1002,vada pav,20
The first block of the 1 MB-block copy starts with the CSV header. HDFS doesn't encode or compress your data: a block is a slice of your file's bytes.

Never edit, move or delete files inside DataNode or NameNode folders by hand. Use HDFS commands; otherwise checksums, the block map and the metadata disagree, and HDFS reports corrupt or missing blocks.

5 See it happen

Following one file from its name to the disk

On the left, the commands from Part 4. On the right, which daemon answers and what it holds.

From path to bytes 6 steps

What you see

Behind the screen

Try it yourself: the NameNode's view

Open the NameNode metadata tab under the cluster: it is the playground's version of fsck -files -blocks -locations.

6 Real-world use

NameNodes and DataNodes in production

1. NameNode sizing is about objects

Large companies size NameNode memory by the number of files, directories and blocks, not by petabytes. Teams that create millions of tiny files are asked to compact them because they cost NameNode heap.

Objects→Heap needed→Server size

2. Workers packed with disks

DataNode servers commonly hold 12 or more large disks, each listed in dfs.datanode.data.dir. A dead disk is swapped while the node keeps serving from the others.

Disk fails→Node keeps running→Hot-swap

3. Protecting the metadata

Admins write the NameNode metadata to more than one disk (a comma-separated dfs.namenode.name.dir), run HA, and back up the fsimage regularly. Losing it means losing the meaning of every block.

fsimage + edits→Several copies→Backups

4. Daily health checks

On-call engineers start the day with the NameNode UI or hdfs dfsadmin -report: live and dead DataNodes, remaining capacity, missing and under-replicated blocks.

Report→Spot problems→Act before users notice

Simplified; organisations differ in tools and policies.

7 Run the code

A tiny NameNode in Python

This program keeps the two NameNode structures from Part 2: the namespace (path → blocks) and the block map (block → DataNodes). Then it answers the question fsck answers: where are this file's blocks?

namespace = {
    "/user/hadoop/data/canteen_orders.csv": {"owner": "hadoop", "repl": 3,
                                             "blocks": ["blk_1073741825"]},
    "/user/hadoop/data/canteen_1mb.csv": {"owner": "hadoop", "repl": 3,
                                          "blocks": ["blk_1073741826", "blk_1073741827"]},
}
block_map = {
    "blk_1073741825": ["datanode-1", "datanode-2", "datanode-4"],
    "blk_1073741826": ["datanode-1", "datanode-3", "datanode-4"],
    "blk_1073741827": ["datanode-1", "datanode-2", "datanode-4"],
}

def locate(path):
    meta = namespace[path]
    print(path, "owner", meta["owner"], "replication", meta["repl"])
    for blk in meta["blocks"]:
        print("  ", blk, "->", ", ".join(block_map.get(blk, [])) or "MISSING")

locate("/user/hadoop/data/canteen_1mb.csv")

/user/hadoop/data/canteen_1mb.csv owner hadoop replication 3 blk_1073741826 -> datanode-1, datanode-3, datanode-4 blk_1073741827 -> datanode-1, datanode-2, datanode-4

namespace = {Saved in the fsimage and edit log: survives a restart.
block_map = {Kept only in memory and rebuilt from DataNode block reports after a restart.
print(" ", blk, "->", ", ".join(block_map.get(blk, [])) or "MISSING")If no DataNode has reported a block, its location list is empty: a missing block.

Run it on your PC: find the real block files of an HDFS file

This script asks the NameNode (via fsck) where each block of a file is, finds the matching Docker container for each DataNode IP, and checks the block file on that container's disk. Save it as where_is_my_file.sh in the hdfs-lab folder.

From an HDFS path to block files on DataNode disks

#!/usr/bin/env bash
# where_is_my_file.sh: show the real block files behind one HDFS file
# usage: bash where_is_my_file.sh /user/hadoop/data/canteen_orders.csv
file=$1
docker compose exec -T namenode hdfs fsck "$file" -files -blocks -locations 2>/dev/null |
  grep -o 'blk_[0-9]*_[0-9]* len=[0-9]*\|[0-9.]*:9866' | while read -r item; do
    case "$item" in
      blk_*) blk=$(echo "$item" | cut -d_ -f1-2); echo "$item";;
      *) ip=${item%:9866}
         c=$(docker ps --format '{{.Names}}' | while read n; do
               docker inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} $n" "$n"; done |
             awk -v ip="$ip" '$1==ip {print $2}')
         path=$(docker exec "$c" find /tmp/hadoop-hadoop/dfs/data -name "$blk" 2>/dev/null)
         size=$(docker exec "$c" stat -c %s "$path")
         echo "   on $c  ($ip)  file size $size bytes";;
    esac
  done

# In the hdfs-lab folder:
#   bash where_is_my_file.sh /user/hadoop/data/canteen_orders.csv

blk_1073741825_1001 len=4692009 on lab5-datanode-1 (172.19.0.4) file size 4692009 bytes on lab5-datanode-2 (172.19.0.5) file size 4692009 bytes on lab5-datanode-4 (172.19.0.7) file size 4692009 bytes

Real output from the lab cluster. Three DataNodes, three identical 4,692,009-byte files: the replicas. The IPs on your machine will differ. On a real cluster you would use ssh to each host instead of docker exec.

8 Line by line

A read request, as the NameNode sees it

Follow a client asking for one file. Watch how the NameNode combines the namespace and the block map, and notice it never touches the data.

Worked examples: real problems, solved and traced in memory

Three calculations platform teams really make for HDFS masters and workers: NameNode heap, how many DataNodes a data set needs, and how evenly blocks are spread. Each runs live and is traced in memory.

Example 1BeginnerCapacity · NameNode

How much heap for the NameNode?

Problem statement

An insurance company stores 40 lakh files averaging 300 MB each (128 MB blocks) in 50,000 directories. Using the rule of thumb of 150 bytes per object, estimate NameNode heap. Then double it for safety and growth.

Input

40,00,000 files · 300 MB each · 128 MB blocks · 50,000 dirs · 150 B per object

Expected output

objects 16,050,000  heap ~2.2 GB  with headroom ~4.5 GB

Approach

Blocks per file = ceil(300 / 128) = 3.
Objects = files + directories + blocks.
Heap = objects × 150 bytes, then × 2 for headroom.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 2IntermediateCapacity · DataNodes

How many DataNodes for 500 TB?

Problem statement

A team expects 500 TB of data with replication 3. Each worker has 12 disks of 16 TB. Keep 25% free for growth and temporary data. How many DataNodes are needed?

Input

500 TB · replication 3 · 12 × 16 TB per node · 25% free

Expected output

raw 1500 TB, 144 TB usable per node -> 11 DataNodes

Approach

Raw space = data × replication.
Usable per node = disks × size × 0.75.
Nodes = ceil(raw / usable per node).

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 3AdvancedOperations · balance

Is data spread evenly over DataNodes?

Problem statement

From dfsadmin -report you have DFS Used per DataNode. Flag any node more than 10 percentage points above or below the cluster average utilisation (the same idea the HDFS balancer uses).

Input

capacity 100 TB each · used: dn1 62, dn2 58, dn3 81, dn4 35, dn5 60 TB

Expected output

cluster average 59.2%
dn1: 62% ok
dn2: 58% ok
dn3: 81% OVER
dn4: 35% UNDER
dn5: 60% ok

Approach

Utilisation = used / capacity per node.
Average = total used / total capacity.
Flag nodes outside average ± 10 points.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

9 Quiz and revision

Check yourself

Quick revision

Lesson 5.1 in one screen

HDFS = one NameNode (master, metadata) + many DataNodes (workers, data).
NameNode: namespace (tree, owners, permissions, replication, file → blocks) + block map (block → DataNodes).
Saved on disk: fsimage + edit log. Not saved: block map (rebuilt from block reports).
DataNode: each replica = blk_N + blk_N_genstamp.meta (checksums, 4 B per 512 B).
Data flows client ↔ DataNode, never through the NameNode.
NameNode heap ≈ 150 B per file/dir/block (rule of thumb): object count matters, not bytes.
Commands: hdfs dfsadmin -report, hdfs fsck PATH -files -blocks -locations.
Ports: NN 8020 / 9870 · DN 9866 / 9867 / 9864.

Interview and viva questions

What is the difference between NameNode and DataNode?

The NameNode is the master that stores metadata: the directory tree, permissions, which blocks make up each file and where they are. DataNodes are workers that store the actual blocks on local disks, serve reads and writes, and report to the NameNode with heartbeats and block reports.

Does the NameNode store data?

No. It stores only metadata. File contents are stored as blocks on DataNodes, and clients read and write those blocks directly from and to DataNodes.

Where does a DataNode store blocks?

In the folders listed in dfs.datanode.data.dir, under current/<block pool ID>/current/finalized/, as files named blk_<id> with a matching blk_<id>_<generation stamp>.meta checksum file.

What happens if the NameNode fails?

Without high availability, the whole file system becomes unavailable until the NameNode is restarted (the data stays on DataNodes but cannot be located). With HA, a standby NameNode takes over automatically.

How does the NameNode know where blocks are?

From block reports: each DataNode reports all its blocks when it starts and periodically afterwards (every 6 hours by default), plus incremental reports when blocks are added or removed.

How much memory does a NameNode need?

It depends on the number of files, directories and blocks. A common rule of thumb is about 150 bytes per object, so 10 crore objects need roughly 14 GB of heap, plus headroom.

Can there be more than one NameNode?

Yes: in HA there is one active and one or more standby NameNodes for the same namespace, and with federation several independent NameNodes each manage part of the namespace on the same DataNodes.

Practice problems

Exercises on metadata, block files and DataNode reports, using numbers from the real cluster.

P1Easy

Checksum file size

A block has 1,048,576 bytes. With 4 bytes of checksum per 512 bytes and a 7-byte header, how big is its .meta file?

Input

1,048,576 bytes · 512 B per checksum · 4 B · 7 B header

Expected output

8199 bytes

Show solution and explanation

import math
print(math.ceil(1048576 / 512) * 4 + 7, "bytes")

8199 bytes

How it works: 8,199 bytes, exactly the size of blk_1073741826_1002.meta on the DataNode.

P2Easy

Replicas on disk

6 blocks with replication 3 are spread over 4 DataNodes. How many block files exist in total, and how many on average per node?

Input

6 blocks · replication 3 · 4 DataNodes

Expected output

18 replicas, 4.5 per node on average

Show solution and explanation

total = 6 * 3
print(total, "replicas,", total / 4, "per node on average")

18 replicas, 4.5 per node on average

How it works: 18 replicas, 4.5 per node on average (the real report showed 3, 6, 5 and 4).

P3Easy

Parse a report line

From "Num of Blocks: 6" and "DFS Used: 9481956 (9.04 MB)", print the number of blocks and MB used.

Input

Num of Blocks: 6 · DFS Used: 9481956 (9.04 MB)

Expected output

6 blocks, 9.04 MB

Show solution and explanation

a = "Num of Blocks: 6"
b = "DFS Used: 9481956 (9.04 MB)"
blocks = int(a.split(":")[1])
used = int(b.split(":")[1].split()[0])
print(blocks, "blocks,", round(used / 2**20, 2), "MB")

6 blocks, 9.04 MB

How it works: 6 blocks and 9.04 MB, as in the report for datanode-1.

P4Medium

Heap for small files

The same 10 TB is stored as 1 GB files or as 100 KB files (one block per file). Estimate NameNode heap for each at 150 bytes per object (file + block).

Input

10 TB · 1 GB vs 100 KB files

Expected output

  1 GB files: 10,240 files -> 2.9 MB heap
100 KB files: 107,374,182 files -> 30,720.0 MB heap

Show solution and explanation

tb = 10 * 2**40
for size, label in [(2**30, "1 GB"), (100 * 2**10, "100 KB")]:
    files = tb // size
    print(f"{label:>6} files: {files:,} files -> {files * 2 * 150 / 2**20:,.1f} MB heap")

1 GB files: 10,240 files -> 2.9 MB heap 100 KB files: 107,374,182 files -> 30,720.0 MB heap

How it works: About 3 MB of heap for 1 GB files versus about 31 GB for 100 KB files: same data, ten thousand times more metadata.

P5Medium

Missing blocks after failures

Given each block's DataNodes and a set of dead nodes, list blocks with no live replica.

Input

dead: dn2, dn5

Expected output

['b2']

Show solution and explanation

blocks = {"b1": ["dn1", "dn2", "dn3"], "b2": ["dn2", "dn5"], "b3": ["dn4", "dn5", "dn6"]}
dead = {"dn2", "dn5"}
print([b for b, nodes in blocks.items() if not set(nodes) - dead])

['b2']

How it works: Only b2 is missing: both of its replicas were on dead nodes (it had replication 2).

P6Medium

Which nodes hold a file?

Using the real fsck output of canteen_1mb.csv, count how many replicas each DataNode IP holds.

Input

5 blocks × 3 locations (from Part 4)

Expected output

Counter({'172.19.0.5': 5, '172.19.0.6': 4, '172.19.0.7': 4, '172.19.0.4': 2})

Show solution and explanation

from collections import Counter
locs = [["5", "6", "7"], ["5", "4", "7"], ["6", "7", "5"], ["6", "7", "5"], ["5", "4", "6"]]
print(Counter("172.19.0." + ip for block in locs for ip in block))

Counter({'172.19.0.5': 5, '172.19.0.6': 4, '172.19.0.7': 4, '172.19.0.4': 2})

How it works: 172.19.0.5 holds all 5 blocks, .7 and .6 hold 4 each, .4 holds 2: 15 replicas in total.

P7Business problem

NameNode server budget

Heap needed is 96 GB. Servers come with 128, 256 or 512 GB RAM; keep at least 32 GB for the OS and other processes. Which is the smallest that fits?

Input

96 GB heap · 32 GB other · 128/256/512

Expected output

128 GB server

Show solution and explanation

need = 96 + 32
print(next(r for r in [128, 256, 512] if r >= need), "GB server")

128 GB server

How it works: 128 GB is exactly enough today, but with no room to grow. Most teams would choose 256 GB for a NameNode.

P8Advanced

Rebuild the block map

After a restart the NameNode only has the namespace. Build the block map from three block reports and print each block's replica count.

Input

dn1: b1,b2 · dn2: b1,b3 · dn3: b2,b3,b1

Expected output

b1 ['dn1', 'dn2', 'dn3'] 3 replicas
b2 ['dn1', 'dn3'] 2 replicas
b3 ['dn2', 'dn3'] 2 replicas

Show solution and explanation

reports = {"dn1": ["b1", "b2"], "dn2": ["b1", "b3"], "dn3": ["b2", "b3", "b1"]}
block_map = {}
for dn, blocks in reports.items():
    for b in blocks:
        block_map.setdefault(b, []).append(dn)
for b in sorted(block_map):
    print(b, block_map[b], len(block_map[b]), "replicas")

b1 ['dn1', 'dn2', 'dn3'] 3 replicas b2 ['dn1', 'dn3'] 2 replicas b3 ['dn2', 'dn3'] 2 replicas

How it works: b1 has 3 replicas, b2 and b3 have 2 each. This is exactly how the block map is rebuilt at start-up (Lessons 5.5 and 5.6).

Real-world scenario questions (asked in company interviews)

Interviewers use NameNode and DataNode questions to see whether you understand where metadata ends and data begins.

HDFS · conceptsFresher

“Does a 1 TB file pass through the NameNode when I upload it?”

A classic first-round question.

The interviewer is checking: the split between the metadata path and the data path.

Show a strong answer

No. The client asks the NameNode to create the file and allocate blocks; the NameNode returns DataNode addresses.
The client streams bytes to the first DataNode, which forwards them along a pipeline to the others.
The NameNode only records metadata (and later learns of finished replicas). That is why one NameNode can serve thousands of DataNodes.

Admin · memoryExperienced

NameNode heap keeps climbing, but the cluster is only 30% full

Disk usage is low, yet the NameNode needs more and more heap and garbage collection pauses are getting longer.

The interviewer is checking: do you know heap depends on object count?

Show a strong answer

Check files and blocks (NameNode UI, FilesTotal/BlocksTotal in /jmx) and find directories with millions of small files (hdfs dfs -count).
Compact small files (merge into larger Parquet/ORC files, use HAR or larger batches in ingestion).
Short term: increase heap and tune GC; long term: fix the sources of small files, add quotas, or federate the namespace.

Troubleshooting · diskExperienced

“Someone deleted blk_ files on a DataNode to free space”

A junior admin removed some old-looking blk_ files from a full DataNode disk.

The interviewer is checking: what HDFS does when replicas vanish behind its back.

Show a strong answer

Those were live replicas. The DataNode will report them missing (block scanner or next block report), and the NameNode will re-replicate from other copies if any exist.
Run hdfs fsck / -list-corruptfileblocks and check for missing blocks; files whose every replica was deleted are lost.
Prevent it: free space with HDFS commands (delete data, lower replication of cold data, add disks, run the balancer), never inside the DataNode folders.

Design · hardwareFresher

Choose hardware for master and worker nodes

You have budget for 1 master and 10 workers. What do you prioritise for each?

The interviewer is checking: that you match hardware to each role.

Show a strong answer

NameNode: plenty of RAM for the heap, reliable disks (RAID) for metadata, redundant power and network; disk capacity is not important.
DataNodes: many large disks (JBOD, no RAID), enough RAM and cores for YARN tasks, good network.
Plan for a standby NameNode (HA) as soon as the cluster matters.

NameNode and DataNode

“Where is my file, really?”

NameNode and DataNode

Words you'll meet in this lesson

Who keeps what (checked on Hadoop 3.5.0)

NameNode and DataNode at three levels

Beginner The library and its index

Intermediate What each daemon does

Advanced Details that matter in production

NameNode vs DataNode, side by side

Other HDFS daemons you'll see

SecondaryNameNode

Standby NameNode

JournalNodes

Router (RBF)

Open the NameNode and a DataNode on a real cluster

Start the Module 5 lab cluster

Store a file twice: normal blocks and 1 MB blocks

Ask the NameNode about the cluster

Ask where a file's blocks are

Look inside the NameNode's folder

Look inside a DataNode's folder

Read a block file directly

Following one file from its name to the disk

Try it yourself: the NameNode's view

NameNodes and DataNodes in production

1. NameNode sizing is about objects

2. Workers packed with disks

3. Protecting the metadata

4. Daily health checks

A tiny NameNode in Python

Run it on your PC: find the real block files of an HDFS file

From an HDFS path to block files on DataNode disks

A read request, as the NameNode sees it

Worked examples: real problems, solved and traced in memory

How much heap for the NameNode?

How many DataNodes for 500 TB?

Is data spread evenly over DataNodes?

Check yourself

Quick revision

Lesson 5.1 in one screen

Interview and viva questions

Practice problems

Checksum file size

Replicas on disk

Parse a report line

Heap for small files

Missing blocks after failures

Which nodes hold a file?

NameNode server budget

Rebuild the block map

Real-world scenario questions (asked in company interviews)

“Does a 1 TB file pass through the NameNode when I upload it?”

NameNode heap keeps climbing, but the cluster is only 30% full

“Someone deleted blk_ files on a DataNode to free space”

Choose hardware for master and worker nodes