Storage for big data: block, file and object storage (and HDFS)

“2 PB in S3 and no file servers”

Block, file and object storage

Words you’ll meet in this lesson

Typical numbers to reason with (approximate)

Storage at three levels

Four kinds of storage compared

Side by side

From one database disk to a petabyte lake

1. Block storage: a disk for one server

   A PostgreSQL database for TiffinGo’s live orders runs on a cloud disk (block storage). When an order’s status changes, the database rewrites one small block in place, in well under a millisecond.

   Fast, in-place, small updates are exactly what databases need. But the disk is attached to one server and is the most expensive storage per GB.

2. File storage: shared folders

   The finance team keeps monthly reports in /finance/2026/ on a shared drive. Anyone with access can open, edit and rename files by path.

   Familiar and convenient for people. At the scale of billions of files and petabytes, managing one giant folder tree becomes slow and costly.

3. HDFS: one 300 MB file becomes blocks

   import math
   file_mb, block_mb, replicas = 300, 128, 3
   blocks = math.ceil(file_mb / block_mb)
   last = file_mb - (blocks - 1) * block_mb
   print("Blocks:", blocks, "| last block:", last, "MB")
   print("Raw space used:", file_mb * replicas, "MB")

   Blocks: 3 | last block: 44 MB Raw space used: 900 MB
   HDFS cuts the file into 128 MB blocks (the last one is smaller), and stores 3 copies of each on different DataNodes. The NameNode remembers which blocks make up the file and where they are.

4. Object storage: keys, not folders

   bucket = {}
   bucket["orders/date=2026-10-19/part-0.parquet"] = b"..."
   bucket["orders/date=2026-10-20/part-0.parquet"] = b"..."
   bucket["reviews/2026/10/19/O1043.jpg"] = b"..."
   prefix = "orders/"
   print([k for k in bucket if k.startswith(prefix)])

   ['orders/date=2026-10-19/part-0.parquet', 'orders/date=2026-10-20/part-0.parquet']
   Listing a “folder” is really asking for every key that starts with a prefix. The key naming scheme decides how easily you can find, partition and delete data later.

5. Copies vs erasure coding

   data_tb = 100
   print("3 replicas     :", data_tb * 3, "TB raw")
   print("Erasure 6 + 3  :", data_tb * (6 + 3) / 6, "TB raw")

   3 replicas : 300 TB raw Erasure 6 + 3 : 150.0 TB raw
   Three full copies need 300 TB of disk for 100 TB of data. Erasure coding (6 data + 3 parity pieces) survives losing any 3 pieces and needs only 150 TB. Cloud object stores use techniques like this behind the scenes.

6. The cost of many tiny files

   files, avg_kb = 2_000_000, 40
   per_file_ms = 10
   print("Total data :", round(files * avg_kb / 1024 / 1024, 1), "GB")
   print("Open + list overhead:", round(files * per_file_ms / 1000 / 60), "minutes")

   Total data : 76.3 GB Open + list overhead: 333 minutes
   Only 76 GB of data, yet over five hours of pure overhead just opening and listing two million files one by one (at an illustrative 10 ms each). Compacting them into a few hundred large files makes the overhead disappear.

One photo: what Riya sees, and what the storage does

Step through the six frames. Left: Riya uploading and viewing a review photo. Right: how object storage stores, serves and ages it.

Real-time scenarios: storage under load

Three live scenarios: a file written into HDFS that heals itself after a node failure, the small-files problem and its fix, and an advanced look at why renaming is unsafe on object storage.

Writing a 300 MB file into HDFS, and surviving a dead node

Kabir uploads a 300 MB orders file to a small HDFS cluster with four DataNodes. Watch it split into blocks, get copied three times, and heal itself when a machine fails.

The small-files problem: 86,400 files a day

A streaming job at TiffinGo saves rider pings every second, one small file each time. The data is tiny; the trouble is the number of files.

Why “rename” is dangerous on object storage

Many big data tools were designed for HDFS, where renaming a folder is instant. Moved to cloud object storage unchanged, the same habit becomes slow and unsafe. Here’s why, and what modern platforms do instead.

Where each kind of storage is used

A tiny object store in a dictionary

Object storage is a giant key → bytes map with metadata. This program builds one with a Python dictionary: put, get and list by prefix. Then try the challenges: split a file into HDFS blocks, survive a node failure, compare replication with erasure coding, cost the small-files problem and plan storage tiers.

bucket = {}

def put(key, data, **meta):
    bucket[key] = {"data": data, "size": len(data), **meta}

def get(key):
    return bucket[key]["data"]

def list_prefix(prefix):
    return sorted(k for k in bucket if k.startswith(prefix))

put("orders/date=2026-10-19/part-0.csv", b"O1,Pune,420\nO2,Thane,250\n", type="csv")
put("orders/date=2026-10-20/part-0.csv", b"O3,Pune,180\n", type="csv")
put("reviews/2026/10/19/O1.jpg", b"...jpeg bytes...", type="image", order="O1")

print("All orders keys:", list_prefix("orders/"))
print("One day only   :", list_prefix("orders/date=2026-10-19/"))
print("Read back      :", get("orders/date=2026-10-20/part-0.csv"))
print("Metadata       :", {k: v for k, v in bucket["reviews/2026/10/19/O1.jpg"].items() if k != "data"})

• def put(key, data, **meta):Storing is just “this key → these bytes, plus some metadata”. There is no folder to create first.
• return sorted(k for k in bucket if k.startswith(prefix))A “folder listing” is really a filter on key prefixes. This is why key naming matters so much.
• put("reviews/2026/10/19/O1.jpg", b"...jpeg bytes...", type="image", order="O1")Metadata travels with the object: type, order id, size.

Run it on your PC: a mini object store and the small-files test

This program builds a tiny object store in a folder: objects are saved under hashed names with a metadata file, and “folders” exist only as key prefixes. Then it writes the same data as 5,000 small files and as one big file and times reading both. Only the standard library is used. Save it as storage_lab.py in an empty folder and run it.

"""A tiny object store on your disk, and the small-files problem measured for real.
Only the Python standard library is used."""
import hashlib, json, os, time

STORE = "my_bucket"


def put(key, data: bytes, **metadata):
    """Store bytes under a key. 'Folders' are just part of the key."""
    path = os.path.join(STORE, "objects", hashlib.sha1(key.encode()).hexdigest())
    os.makedirs(os.path.dirname(path), exist_ok=True)
    with open(path, "wb") as f:
        f.write(data)
    meta = {"key": key, "size": len(data), "etag": hashlib.md5(data).hexdigest(), **metadata}
    with open(path + ".meta.json", "w") as f:
        json.dump(meta, f)


def get(key):
    path = os.path.join(STORE, "objects", hashlib.sha1(key.encode()).hexdigest())
    with open(path, "rb") as f:
        return f.read()


def list_prefix(prefix):
    keys = []
    for name in os.listdir(os.path.join(STORE, "objects")):
        if name.endswith(".meta.json"):
            with open(os.path.join(STORE, "objects", name)) as f:
                k = json.load(f)["key"]
            if k.startswith(prefix):
                keys.append(k)
    return sorted(keys)


put("reviews/2026/10/19/O1043.jpg", b"\xff\xd8 fake image bytes", content_type="image/jpeg", order="O1043")
put("reviews/2026/10/19/O1044.jpg", b"\xff\xd8 another image", content_type="image/jpeg", order="O1044")
put("orders/date=2026-10-19/part-0.csv", b"O1043,Pune,420\nO1044,Thane,250\n", content_type="text/csv")
print("Keys under reviews/2026/10/19/:", list_prefix("reviews/2026/10/19/"))
print("Read back:", get("orders/date=2026-10-19/part-0.csv").decode().strip().splitlines())
print("Note: there are no real folders on disk, only hashed object names + metadata files.\n")

# ---- the small-files problem: same bytes, 5,000 files vs 1 file
os.makedirs("small_files", exist_ok=True)
row = b"O1043,Pune,thali,220,2026-10-19\n"
for i in range(5000):
    with open(f"small_files/part-{i:05d}.csv", "wb") as f:
        f.write(row * 20)
with open("one_big_file.csv", "wb") as f:
    f.write(row * 20 * 5000)

t = time.perf_counter()
total = 0
for name in os.listdir("small_files"):
    with open(os.path.join("small_files", name), "rb") as f:
        total += f.read().count(b"\n")
small_s = time.perf_counter() - t

t = time.perf_counter()
with open("one_big_file.csv", "rb") as f:
    big_rows = f.read().count(b"\n")
big_s = time.perf_counter() - t

print(f"5,000 small files: {total // 100000} lakh rows read in {small_s * 1000:.0f} ms")
print(f"1 big file       : {big_rows // 100000} lakh rows read in {big_s * 1000:.0f} ms")
print(f"Same data, about {small_s / big_s:.0f}x slower as small files on this machine")

# In a terminal:  python storage_lab.py

Re-replicating blocks after a failure, one line at a time

Two blocks have three copies each. Node DN2 dies. Watch the program find each block that lost a copy and pick a new node for it.

Worked examples: real problems, solved and traced in memory

Four storage calculations a data engineer actually makes, from HDFS blocks to a tiered storage bill. Each runs live and is traced.

orders database, finance team reports, data lake, CCTV footage, VM boot disk

orders database        -> block storage
finance team reports   -> file storage
data lake              -> object storage
CCTV footage           -> object storage
VM boot disk           -> block storage

400 GB/day × 180 days, 3 replicas, 25% free, 48 TB per node

Data kept : 70 TB
With 3 copies: 211 TB
With 25% free: 281 TB -> 6 nodes of 48 TB

5 days of files, 2 parts per day

Total keys: 10
Week keys : 6
silver/orders/date=2026-10-19/part-00000.parquet

1 TB/day, 730 days, tiers 30 d / 365 d

TB per tier : {'hot': 30, 'cool': 335, 'archive': 365}
Tiered bill : Rs 425500
All hot     : Rs 1314000
Saving      : 68%

Check yourself

Quick revision

Interview and viva questions

Practice problems

Storage calculations from HDFS blocks to cloud bills. The last two are business tasks.

1000 MB

8

import math
print(math.ceil(1000 / 128))

20 TB, 3 replicas

60 TB

print(20 * 3, "TB")

4 keys

['orders/a.csv', 'orders/b.csv']

keys = ["orders/a.csv", "reviews/x.jpg", "orders/b.csv", "logs/app.log"]
print([k for k in keys if k.startswith("orders/")])

[40, 52, 38, 61, 45]

0.046 MB -> compact

sizes_kb = [40, 52, 38, 61, 45]
avg_mb = sum(sizes_kb) / len(sizes_kb) / 1024
print(round(avg_mb, 3), "MB ->", "compact" if avg_mb < 64 else "ok")

B1: 3, B2: 2, B3: 3, B4: 1

['B2', 'B4']

copies = {"B1": 3, "B2": 2, "B3": 3, "B4": 1}
print([b for b, c in copies.items() if c < 3])

100 TB

3 replicas 300 TB  (+200%)
6+3        150 TB  (+50%)
10+4       140 TB  (+40%)

for name, d, p in [("3 replicas", 1, 2), ("6+3", 6, 3), ("10+4", 10, 4)]:
    raw = 100 * (d + p) / d
    print(f"{name:10} {raw:.0f} TB  (+{raw - 100:.0f}%)")

10 lakh × 1 MB

   1 MB files:  1000000 files, NameNode RAM ≈ 286.1 MB
 128 MB files:     7812 files, NameNode RAM ≈ 2.2 MB

import math
total_mb = 10_00_000
for size in [1, 128]:
    files = total_mb // size
    blocks = files * math.ceil(size / 128)
    ram_mb = (files + blocks) * 150 / 1024 / 1024
    print(f"{size:4} MB files: {files:8} files, NameNode RAM ≈ {ram_mb:.1f} MB")

600 TB

Tiered: Rs 370000
All hot: Rs 1080000
Saving: Rs 710000

tiered = 50 * 1800 + 250 * 1000 + 300 * 100
all_hot = 600 * 1800
print("Tiered: Rs", tiered)
print("All hot: Rs", all_hot)
print("Saving: Rs", all_hot - tiered)

900 TB, 3×, 30% free, 60 TB/node

Raw needed: 3857 TB
Nodes: 65

import math
raw = 900 * 3 / 0.7
print("Raw needed:", round(raw), "TB")
print("Nodes:", math.ceil(raw / 60))

Real-world scenario questions (asked in company interviews)

Storage decisions and problems from real data teams. Decide your answer first, then open the model answer.