fsimage, edit log and checkpointing

A photo and a diary

fsimage, edit log and checkpointing

Words you'll meet in this lesson

Checkpoint settings (Hadoop 3.5.0 defaults)

Checkpointing at three levels

Who can make a checkpoint?

Tools for metadata files

Read the diary and watch the photos being taken

The Module 5 lab cluster runs a SecondaryNameNode with dfs.namenode.checkpoint.period=120, so a checkpoint happens every 2 minutes instead of every hour.

1. List the NameNode's metadata files

   docker compose exec namenode ls -l /tmp/hadoop-hadoop/dfs/name/current
   docker compose exec namenode cat /tmp/hadoop-hadoop/dfs/name/current/seen_txid

   -rw-r--r-- 1 hadoop users 4003 Oct 21 06:49 edits_0000000000000000001-0000000000000000044 -rw-r--r-- 1 hadoop users 1920 Oct 21 06:51 edits_0000000000000000045-0000000000000000062 -rw-r--r-- 1 hadoop users 42 Oct 21 06:53 edits_0000000000000000063-0000000000000000064 -rw-r--r-- 1 hadoop users 42 Oct 21 06:55 edits_0000000000000000065-0000000000000000066 -rw-r--r-- 1 hadoop users 1048576 Oct 21 06:55 edits_inprogress_0000000000000000067 -rw-r--r-- 1 hadoop users 1298 Oct 21 06:53 fsimage_0000000000000000064 -rw-r--r-- 1 hadoop users 62 Oct 21 06:53 fsimage_0000000000000000064.md5 -rw-r--r-- 1 hadoop users 1298 Oct 21 06:55 fsimage_0000000000000000066 -rw-r--r-- 1 hadoop users 62 Oct 21 06:55 fsimage_0000000000000000066.md5 -rw-r--r-- 1 hadoop users 3 Oct 21 06:55 seen_txid -rw-r--r-- 1 hadoop users 215 Oct 21 06:48 VERSION 67
   Read the numbers as a timeline: transactions 1–44 (our uploads and jobs), 45–62, … The newest photo is fsimage_…066; everything after transaction 66 is in edits_inprogress_…067, which is pre-allocated as a 1 MB file. A 42-byte segment holds just the start and end markers of an empty segment.

2. See a checkpoint in the SecondaryNameNode log

   docker compose logs secondarynamenode | grep -iE "checkpoint|Downloaded file|Uploaded image"

   INFO TransferFsImage:177 - Downloaded file edits_tmp_0000000000000000065-0000000000000000066_0000000000000550729 size 0 bytes. INFO Checkpointer:324 - Checkpointer about to load edits from 1 stream(s). INFO Storage:475 - setMostRecentCheckpointInfo txid is 66, time is 1792565757250 INFO TransferFsImage:260 - Uploaded image with txid 66 to namenode at http://namenode:9870 in 0.014 seconds WARN SecondaryNameNode:586 - Checkpoint done. New Image Size: 1298
   Download edits → load → merge → upload fsimage_…066 to the NameNode. (It is logged at WARN level, but it's a normal success message.)

3. Make some changes, then read the edit log

   docker compose exec namenode hdfs dfs -mkdir data/edits_demo
   docker compose exec namenode hdfs dfs -touchz data/edits_demo/hello.txt
   docker compose exec namenode hdfs dfs -mv data/edits_demo/hello.txt data/edits_demo/namaste.txt
   docker compose exec namenode hdfs dfs -rm -r -skipTrash data/edits_demo
   docker compose exec namenode bash -c 'f=$(ls /tmp/hadoop-hadoop/dfs/name/current/edits_inprogress_*); hdfs oev -i $f -o /tmp/edits.xml; grep -o "<OPCODE>[A-Z_]*</OPCODE>" /tmp/edits.xml | sort | uniq -c'

   1 <OPCODE>OP_ADD</OPCODE> 1 <OPCODE>OP_CLOSE</OPCODE> 1 <OPCODE>OP_DELETE</OPCODE> 1 <OPCODE>OP_MKDIR</OPCODE> 1 <OPCODE>OP_RENAME_OLD</OPCODE> 1 <OPCODE>OP_START_LOG_SEGMENT</OPCODE>
   Four commands, five operations: creating a file is OP_ADD (open) plus OP_CLOSE (finished). The segment always starts with OP_START_LOG_SEGMENT.

4. Look at one record

   docker compose exec namenode bash -c 'grep -A9 "OP_RENAME" /tmp/edits.xml'

   <OPCODE>OP_RENAME_OLD</OPCODE> <DATA> <TXID>71</TXID> <LENGTH>0</LENGTH> <SRC>/user/hadoop/data/edits_demo/hello.txt</SRC> <DST>/user/hadoop/data/edits_demo/namaste.txt</DST> <TIMESTAMP>1792565829164</TIMESTAMP> <RPC_CLIENTID>52ec91db-a945-4573-8c04-3475b16572cf</RPC_CLIENTID> <RPC_CALLID>3</RPC_CALLID> </DATA>
   Transaction 71: rename hello.txt to namaste.txt. The client ID and call ID let the NameNode recognise a retried request after a failover and not apply it twice.

5. Force a checkpoint with saveNamespace

   docker compose exec namenode hdfs dfsadmin -safemode enter
   docker compose exec namenode hdfs dfsadmin -saveNamespace
   docker compose exec namenode hdfs dfsadmin -safemode leave
   docker compose exec namenode ls -l /tmp/hadoop-hadoop/dfs/name/current

   Safe mode is ON Save namespace successful Safe mode is OFF -rw-r--r-- 1 hadoop users 4003 Oct 21 06:49 edits_0000000000000000001-0000000000000000044 -rw-r--r-- 1 hadoop users 1920 Oct 21 06:51 edits_0000000000000000045-0000000000000000062 -rw-r--r-- 1 hadoop users 42 Oct 21 06:53 edits_0000000000000000063-0000000000000000064 -rw-r--r-- 1 hadoop users 42 Oct 21 06:55 edits_0000000000000000065-0000000000000000066 -rw-r--r-- 1 hadoop users 652 Oct 21 06:57 edits_0000000000000000067-0000000000000000073 -rw-r--r-- 1 hadoop users 1048576 Oct 21 06:57 edits_inprogress_0000000000000000074 -rw-r--r-- 1 hadoop users 1298 Oct 21 06:55 fsimage_0000000000000000066 -rw-r--r-- 1 hadoop users 62 Oct 21 06:55 fsimage_0000000000000000066.md5 -rw-r--r-- 1 hadoop users 1298 Oct 21 06:57 fsimage_0000000000000000073 -rw-r--r-- 1 hadoop users 62 Oct 21 06:57 fsimage_0000000000000000073.md5 -rw-r--r-- 1 hadoop users 3 Oct 21 06:57 seen_txid -rw-r--r-- 1 hadoop users 215 Oct 21 06:57 VERSION
   Our 5 operations (plus segment markers) became segment 67–73, a new photo fsimage_…073 was written, and fsimage_…064 was deleted: only 2 images are retained. The in-progress segment restarted at 74.

6. Read the photo

   docker compose exec namenode bash -c 'hdfs oiv -p Delimited -i /tmp/hadoop-hadoop/dfs/name/current/fsimage_0000000000000000073 -o /tmp/fsimage.tsv; cut -f1,2,4,7 /tmp/fsimage.tsv'

   Path Replication AccessTime FileSize / 0 1970-01-01 00:00 0 /user 0 1970-01-01 00:00 0 /user/hadoop 0 1970-01-01 00:00 0 /user/hadoop/data 0 1970-01-01 00:00 0 /user/hadoop/data/canteen_orders.csv 3 2026-10-21 06:49 4692009 /user/hadoop/data/canteen_1mb.csv 3 2026-10-21 06:49 4692009 /user/hadoop/out 0 1970-01-01 00:00 0 /user/hadoop/out/wc1 0 1970-01-01 00:00 0 /user/hadoop/out/wc1/part-r-00000 3 2026-10-21 06:49 4942122 /user/hadoop/out/wc1/_SUCCESS 3 2026-10-21 06:49 0 /user/hadoop/out/wc128 0 1970-01-01 00:00 0 /user/hadoop/out/wc128/part-r-00000 3 2026-10-21 06:49 4942122 /user/hadoop/out/wc128/_SUCCESS 3 2026-10-21 06:49 0
   The whole namespace: 13 inodes in a 1,298-byte file. edits_demo is not there: it was created and deleted before the photo, so it exists only in the diary. Directories show replication 0. (The column header comes from the tool; the selected columns are path, replication, access time and size.)

One checkpoint, step by step

The real file names from the lab, around the checkpoint at transaction 66.

Metadata files in production

Simplified; organisations differ in tools and policies.

Rebuild a namespace from a photo and a diary

This program does what a NameNode does at start-up: load the fsimage (a dictionary of paths), then replay each edit after it, in transaction order. The edits are the five real operations from Part 4.

fsimage_txid = 66
fsimage = {"/user/hadoop/data": "dir",
           "/user/hadoop/data/canteen_orders.csv": "file"}
edits = [
    (68, "OP_MKDIR", "/user/hadoop/data/edits_demo", None),
    (69, "OP_ADD", "/user/hadoop/data/edits_demo/hello.txt", None),
    (70, "OP_CLOSE", "/user/hadoop/data/edits_demo/hello.txt", None),
    (71, "OP_RENAME_OLD", "/user/hadoop/data/edits_demo/hello.txt",
         "/user/hadoop/data/edits_demo/namaste.txt"),
    (72, "OP_DELETE", "/user/hadoop/data/edits_demo", None),
]

ns = dict(fsimage)
for txid, op, path, dst in edits:
    if txid <= fsimage_txid:
        continue
    if op == "OP_MKDIR":
        ns[path] = "dir"
    elif op == "OP_ADD":
        ns[path] = "file (open)"
    elif op == "OP_CLOSE":
        ns[path] = "file"
    elif op == "OP_RENAME_OLD":
        ns[dst] = ns.pop(path)
    elif op == "OP_DELETE":
        ns = {p: k for p, k in ns.items() if p != path and not p.startswith(path + "/")}
    print(f"txid {txid}: {op:14} -> {len(ns)} paths")
print(sorted(ns))

• if txid <= fsimage_txid:Edits already contained in the fsimage are skipped. Only transactions after the image's txid are replayed.
• ns[path] = "file (open)"OP_ADD creates the file while it is being written; OP_CLOSE marks it complete.
• ns = {p: k for p, k in ns.items() if p != path and not p.startswith(path + "/")}Deleting a directory removes everything below it too.

Run it on your PC: watch your commands land in the edit log

This script makes four changes in HDFS, then decodes the newest edit log segments with hdfs oev and prints only the transactions your commands created. Save it as edit_log_report.py in the hdfs-lab folder and run it while the cluster is up.

# edit_log_report.py: make a few changes in HDFS, then read them back from the NameNode's edit log.
# Run in the hdfs-lab folder while the cluster is up.
import json
import subprocess
import xml.etree.ElementTree as ET

NAME_DIR = "/tmp/hadoop-hadoop/dfs/name/current"

def nn(*cmd):
    return subprocess.run(["docker", "compose", "exec", "-T", "namenode", *cmd],
                          capture_output=True, text=True).stdout

def last_txid():
    url = "http://namenode:9870/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo"
    info = json.loads(json.loads(nn("curl", "-s", url))["beans"][0]["JournalTransactionInfo"])
    return int(info["LastAppliedOrWrittenTxId"])

before = last_txid()
print("last transaction before our changes:", before)
for cmd in (["hdfs", "dfs", "-mkdir", "-p", "data/demo"],
            ["hdfs", "dfs", "-touchz", "data/demo/order.txt"],
            ["hdfs", "dfs", "-mv", "data/demo/order.txt", "data/demo/order_old.txt"],
            ["hdfs", "dfs", "-rm", "-r", "-skipTrash", "data/demo"]):
    nn(*cmd)

# a checkpoint may have closed the segment meanwhile, so read the newest two edit files
files = nn("bash", "-c", f"ls {NAME_DIR} | grep ^edits_ | tail -2").split()
for f in files:
    nn("hdfs", "oev", "-i", f"{NAME_DIR}/{f}", "-o", "/tmp/edits.xml")
    for rec in ET.fromstring(nn("cat", "/tmp/edits.xml")).iter("RECORD"):
        data = rec.find("DATA")
        txid = int(data.findtext("TXID"))
        path = data.findtext("PATH") or data.findtext("SRC") or ""
        dst = data.findtext("DST")
        if txid > before and path:
            print(f"txid {txid:>4}  {rec.findtext('OPCODE'):15} {path}"
                  + (f" -> {dst}" if dst else "") + f"   [{f}]")

# In the hdfs-lab folder:
#   python3 edit_log_report.py

A checkpoint, one line at a time

Follow the SecondaryNameNode: it loads the old image, applies only newer edits, and produces an image with a new transaction ID.

Worked examples: real problems, solved and traced in memory

Three checkpoint calculations that matter in operations: how fast edits accumulate, how long a restart would take right now, and whether metadata storage is safe. Each runs live and is traced in memory.

30,00,000 creates × 5 tx · 30,00,000 deletes × 1 tx · trigger 10,00,000

18,000,000 transactions/day -> count trigger 18 times/day, time trigger 24 times/day

9 days × 2 crore tx · load 4 min · 60,000 tx/s

broken checkpoints  :   180,000,000 edits ->     54 min
healthy             :     1,000,000 edits ->      4 min

"file:///data/1/nn,file:///data/1/nn-copy,file:///tmp/nn" · disks: /data/1, /tmp

['/data/1/nn', '/data/1/nn-copy', '/tmp/nn']
several directories on the same disk
/tmp/nn is under /tmp

Check yourself

Quick revision

Interview and viva questions

Practice problems

Exercises on transactions, checkpoint timing and metadata files, using real file names from the lab.

45 to 62

18

print(62 - 45 + 1)

064 · 066 · 073

fsimage_0000000000000000073

print("fsimage_%019d" % max([64, 66, 73]))

1,000 files · 2 ops each

2000

print(1000 * 2)

image 73 · segments … · in-progress 74–81

[(74, 81)] -> 8 transactions

image = 73
segments = [(1, 44), (45, 62), (63, 64), (65, 66), (67, 73), (74, 81)]
replay = [s for s in segments if s[1] > image]
print(replay, "->", sum(b - max(a, image + 1) + 1 for a, b in replay), "transactions")

64 · 66 · 73 · retain 2

[66, 73]

print(sorted([64, 66, 73])[-2:])

250 tx/s

time trigger after 3600 s

t = 1_000_000 / 250
print("count" if t < 3600 else "time", "trigger after", min(t, 3600), "s")

170 extra minutes · 400 people · Rs 1,200/h

Rs 13.6 lakh

extra_h = (180 - 10) / 60
print(f"Rs {extra_h * 400 * 1200 / 1e5:.1f} lakh")

samples every hour

ALERT: no checkpoint for 3 h, 3,499,500 edits pending

samples = [(0, 500, 900), (1, 500, 1_200_000), (2, 500, 2_400_000), (3, 500, 3_500_000)]
first = next(t for t, cp, cur in samples if cp == samples[-1][1])
t_now, cp, cur = samples[-1]
if t_now - first > 2 and cur > cp:
    print(f"ALERT: no checkpoint for {t_now - first} h, {cur - cp:,} edits pending")

Real-world scenario questions (asked in company interviews)

Interviewers love the SecondaryNameNode because its name misleads so many candidates.