The small files problem

Ten thousand files, three hundred kilobytes

The small files problem

Words you'll meet in this lesson

From our lab (Hadoop 3.5.0, same 338,894 bytes)

Small files at three levels

Solutions compared

Where small files come from

Measuring and fixing small files

Run on the Module 5 lab cluster with small_files_lab.sh (in “Run it on your PC”).

1. Create the same data two ways

   seq 1 10000 | while read i; do printf 'order_id,item,amount\n%d,chai,20\n' $i > o_$i.csv; done
   cat /tmp/small/*.csv > /tmp/big.csv

   10000 40M /tmp/small 338894 bytes in big.csv
   Even the local disk wastes space: 339 KB of data occupy 40 MB because each tiny file takes a 4 KB block (Big Data Fundamentals, Lesson 2.3).

2. Upload both and watch the NameNode

   hdfs dfs -put /tmp/small/*.csv /small/
   hdfs dfs -put /tmp/big.csv /big/
   curl -s 'http://localhost:9870/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'   # FilesTotal, BlocksTotal

   NameNode before : FilesTotal 41 BlocksTotal 31 put 10000 files: 281026 ms NameNode after small files: FilesTotal 10043 BlocksTotal 10031 put 1 file: 2133 ms
   4 minutes 41 seconds vs 2 seconds for the same bytes. The NameNode gained 10,002 files (10,000 + two folders) and 10,000 blocks. (Its JMX counters refresh with a delay, so they didn't show the single big file's +1 within our 6 s wait; fsck below confirms it.)

3. Same bytes, very different object counts

   hdfs dfs -count -v /small /big
   hdfs dfs -du -s /small /big
   hdfs fsck /small ; hdfs fsck /big

   DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME 1 10000 338894 /small 1 1 338894 /big 338894 1016682 /small 338894 1016682 /big Total blocks (validated): 10000 (avg. block size 33 B) Total blocks (validated): 1 (avg. block size 338894 B)
   Identical bytes on DataNode disks (HDFS blocks are only as big as their data, so no disk is wasted there). The difference is entirely in the NameNode: 10,000 blocks averaging 33 bytes.

4. Read everything back

   hdfs dfs -cat '/small/*.csv' | wc -l
   hdfs dfs -cat /big/big.csv | wc -l

   20000 read 10000 files: 15168 ms 20000 read 1 file: 1892 ms
   Same 20,000 lines. The small files needed 10,000 location lookups and 10,000 DataNode connections: 8 times slower, and that's with one client and no job framework.

5. Fix: pack them into a Hadoop Archive

   hadoop archive -archiveName orders.har -p /small /archives
   hdfs dfs -ls /archives/orders.har
   hdfs dfs -count -v har:///archives/orders.har ; hdfs dfs -cat har:///archives/orders.har/o_42.csv

   archive job took 24 s -rw-r--r-- 3 hadoop supergroup 0 2026-10-21 16:39 /archives/orders.har/_SUCCESS -rw-r--r-- 3 hadoop supergroup 834555 2026-10-21 16:39 /archives/orders.har/_index -rw-r--r-- 3 hadoop supergroup 388 2026-10-21 16:39 /archives/orders.har/_masterindex -rw-r--r-- 3 hadoop supergroup 338894 2026-10-21 16:39 /archives/orders.har/part-0 1 10000 338894 har:///archives/orders.har order_id,item,amount 42,chai,20
   The archive job (a MapReduce job, here in local mode) packed all data into one part-0 file plus an index. Through har:// the 10,000 names still exist and can be read individually.

6. Delete the originals and check the NameNode

   hdfs dfs -rm -r -skipTrash /small

   removed /small NameNode after HAR, without /small: FilesTotal 49 BlocksTotal 35
   From over 10,000 files and blocks back to 49 and 35: the archive costs the NameNode 4 files. The trade-off: archives are immutable, and each read first consults the index.

What the NameNode does for each tiny file

Six steps comparing one small-file upload with one big-file upload, then the archive fix.

Small files at work

Simplified; organisations differ in tools and policies.

Count the NameNode's work for small and big files

This model counts NameNode objects (files + blocks), NameNode calls and map tasks for the same data stored as many small files or a few big ones. The cost per object comes from the common rule of thumb of about 150 bytes of heap.

import math
BLOCK = 128 * 1024 * 1024
HEAP_PER_OBJECT = 150                       # bytes, rule of thumb

def cost(total_bytes, n_files):
    size = total_bytes / n_files
    blocks_per_file = max(1, math.ceil(size / BLOCK))
    objects = n_files * (1 + blocks_per_file)
    calls = n_files * (2 + blocks_per_file)   # create + complete + addBlock per block
    tasks = n_files * blocks_per_file         # about one split per block (one per small file)
    return objects, calls, tasks, objects * HEAP_PER_OBJECT

data = 338_894
for n in [10_000, 1]:
    o, c, t, heap = cost(data, n)
    print(f"{n:>6} files: {o:>6} objects, {c:>6} NameNode calls, {t:>6} map tasks, ~{heap / 1024:.0f} KB heap")

• objects = n_files * (1 + blocks_per_file)Each file is at least two objects: the file and its block. Our lab: +10,000 files and +10,000 blocks.
• calls = n_files * (2 + blocks_per_file) # create + complete + addBlock per blockEach file needs its own create, addBlock and complete calls, the reason 10,000 uploads took 281 s.
• tasks = n_files * blocks_per_file # about one split per block (one per small file)Frameworks create about one split per file: 10,000 tiny tasks, mostly start-up overhead.

Run it on your PC: 10,000 files vs 1, then archive them

Run this against the Module 5 lab cluster. It creates 10,000 tiny CSV files and one file with the same bytes inside the namenode container, uploads both, reads NameNode counters from /jmx, compares reads, then packs the small files into a Hadoop Archive. It takes about 6 minutes, most of it the small-file upload.

#!/usr/bin/env bash
# small_files_lab.sh: the same 10 MB of orders as 10,000 tiny files vs 1 file, and what the NameNode pays.
NN="docker exec lab5-namenode-1"
jmx() { sleep 6; $NN bash -c "curl -s 'http://localhost:9870/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'" | python3 -c "import sys,json; b=json.load(sys.stdin)['beans'][0]; print('FilesTotal', b['FilesTotal'], ' BlocksTotal', b['BlocksTotal'])"; }
run() { echo "\$ $1"; $NN bash -c "$1" 2>&1 | grep -v -E "^\s*$|WARN|INFO"; echo; }
run "hdfs dfs -rm -r -f -skipTrash /small /big /archives > /dev/null; rm -rf /tmp/small && mkdir -p /tmp/small && cd /tmp/small && seq 1 10000 | while read i; do printf 'order_id,item,amount\n%d,chai,20\n' \$i > o_\$i.csv; done; ls | wc -l; du -sh /tmp/small; cat /tmp/small/*.csv > /tmp/big.csv; ls -l /tmp/big.csv | awk '{print \$5, \"bytes in big.csv\"}'"
echo "NameNode before : $(jmx)"; echo
run "hdfs dfs -mkdir -p /small /big; s=\$(date +%s%N); hdfs dfs -put /tmp/small/*.csv /small/; echo put 10000 files: \$(( (\$(date +%s%N)-s)/1000000 )) ms"
echo "NameNode after small files: $(jmx)"; echo
run "s=\$(date +%s%N); hdfs dfs -put /tmp/big.csv /big/; echo put 1 file: \$(( (\$(date +%s%N)-s)/1000000 )) ms"
echo "NameNode after big file   : $(jmx)"; echo
run "hdfs dfs -count -v /small /big"
run "hdfs dfs -du -s /small /big"
run "s=\$(date +%s%N); hdfs dfs -cat '/small/*.csv' | wc -l; echo read 10000 files: \$(( (\$(date +%s%N)-s)/1000000 )) ms"
run "s=\$(date +%s%N); hdfs dfs -cat /big/big.csv | wc -l; echo read 1 file: \$(( (\$(date +%s%N)-s)/1000000 )) ms"
run "hdfs fsck /small | grep -E 'Total files|Total blocks|Average block size'; hdfs fsck /big | grep -E 'Total files|Total blocks|Average block size'"
echo "=== FIX 1: pack the small files into a Hadoop Archive (HAR)"
run "s=\$(date +%s); hadoop archive -archiveName orders.har -p /small /archives > /dev/null 2>&1; echo archive job took \$(( \$(date +%s)-s )) s; hdfs dfs -ls /archives/orders.har"
run "hdfs dfs -count -v har:///archives/orders.har; hdfs dfs -cat har:///archives/orders.har/o_42.csv"
run "hdfs dfs -rm -r -skipTrash /small > /dev/null && echo removed /small"
echo "NameNode after HAR, without /small: $(jmx)"

# Run:  bash small_files_lab.sh

Packing small files into one container

A container (like a SequenceFile or a HAR) stores many small files inside one big file and keeps an index of where each one starts. Follow three files being packed and one being read back.

Worked examples: real problems, solved and traced in memory

Three small-files calculations: NameNode heap, job overhead, and the payoff of compaction.

20,00,000/day · 180 days · 2 objects/file

360,000,000 files -> 101 GB of NameNode heap

10,000 × (1.5 + 0.05) s / 200 slots vs 2 × 41.5 s

small files: 77.5 s (4.2 task-hours of pure start-up)
big files  : 41.5 s

2,880 × 50 files/day · compaction 20 min × 40 nodes

heap/GC cost avoided: Rs 3,844,720  compaction cost: Rs 2,920,000

Check yourself

Quick revision

Interview and viva questions

Practice problems

Small-files exercises using the real counts and timings from the lab.

two counters

20002 objects

print((10043 - 41) + (10031 - 31), "objects")

281,026 vs 2,133 ms · 15,168 vs 1,892 ms

upload 132x slower, read 8.0x slower

print(f"upload {281026 / 2133:.0f}x slower, read {15168 / 1892:.1f}x slower")

338,894 bytes / 10,000

33.8894 bytes

print(338894 / 10000, "bytes")

5,000/hour × 8,760 hours

87,600,000 objects -> 12.2 GB

files = 5000 * 8760
print(f"{files * 2:,} objects -> {files * 2 * 150 / 1024**3:.1f} GB")

index + bytes

b'3,dosa,40\n'

part0 = b"1,chai,20\n2,poha,30\n3,dosa,40\n"
index = {"o_1.csv": (0, 10), "o_2.csv": (10, 10), "o_3.csv": (20, 10)}
o, l = index["o_3.csv"]
print(part0[o:o + l])

10,000 × 34 B

1 task(s)

import math
print(max(1, math.ceil(10000 * 34 / (128 * 1024**2))), "task(s)")

₹35 lakh vs 3 × ₹1.5 lakh

A: Rs 35 lakh, B: Rs 4.5 lakh

a, b = 35, 3 * 1.5
print(f"A: Rs {a} lakh, B: Rs {b} lakh")

3 partitions

dt=2026-10-19: 2880 -> 22 files
dt=2026-10-20: 2880 -> 24 files
dt=2026-10-21: 1500 -> 2 files
objects saved: 14424

import math
parts = {"dt=2026-10-19": (5400, 2880), "dt=2026-10-20": (6100, 2880), "dt=2026-10-21": (300, 1500)}
saved = 0
for p, (mb, files) in parts.items():
    out = max(1, math.ceil(mb / 256))
    saved += (files - out) * 2
    print(f"{p}: {files} -> {out} files")
print("objects saved:", saved)

Real-world scenario questions (asked in company interviews)

The small files problem is one of the most asked Hadoop interview topics, from freshers to architects.