Compression codecs and splittability

1 The story

One gzip file, one lonely map task

Kabir wanted to save space, so he gzipped a whole year of canteen orders into one file before loading it into HDFS. The file shrank to a quarter of its size. Then the nightly job, which used to start hundreds of map tasks, started exactly one. That single task read the whole file from start to end while every other worker sat idle. Riya asks Arjun Sir whether compression was a mistake.

Arjun Sir

Compression was not the mistake. The choice of codec and where you used it was. Every codec trades size against speed, and only some can be split, which means a map task can start reading in the middle of the file.

Riya

So which one should we use?

Arjun Sir

It depends on the place: input files, the shuffle between map and reduce, the final output, or inside a container format like Parquet. Today we measure six codecs on 194 MB of orders, run the same job on each compressed file, and compress a shuffle from 60 MB down to 13 KB.

2 Definition

Compression codecs and splittability

Formal definition (write this in exams)

A compression codec in Hadoop is an implementation of org.apache.hadoop.io.compress.CompressionCodec that creates compressing output streams and decompressing input streams for one algorithm, such as DEFLATE, gzip, bzip2, LZ4, Snappy or Zstandard. CompressionCodecFactory selects the codec for an input file from its extension (.gz, .bz2, .zst…), so jobs read compressed text transparently. A codec is splittable if a reader can start decompressing at an arbitrary position in the stream, which Hadoop expresses with SplittableCompressionCodec; among the built-in codecs only bzip2 is. A file compressed with a non-splittable codec becomes a single input split, however many HDFS blocks it spans. Compression can be applied to job input, to intermediate map output (mapreduce.map.output.compress), to job output (mapreduce.output.fileoutputformat.compress), and inside container formats that compress block by block and therefore remain splittable.

In simple words

A codec makes files smaller, at some CPU cost. Some codecs (gzip) must be read from the very first byte, so a big gzip file gets only one map task. Fast codecs suit the shuffle; strong ones suit cold storage; containers like Parquet and ORC give you compression and splitting together.

Words you'll meet in this lesson

CodecCompressor + decompressor for one algorithm.

Compression ratioCompressed size ÷ original size; lower is better.

SplittableA reader can start in the middle of the file (bzip2).

CompressionCodecFactoryChooses the codec from the file name extension.

Map output compressionCompressing the shuffle data written by maps.

Output compressionCompressing the job's part files.

Native librarylibhadoop.so plus zlib, zstd, bzip2: fast C code. Check with hadoop checknative.

Block compressionContainers compress chunks independently, so they stay splittable.

From our lab (Hadoop 3.5.0, 194,186,990 bytes of CSV, one thread)

Smallest	bzip2: 31,984,098 bytes (16.5%), but 24.2 s to compress and 6.7 s to decompress
Balanced	Zstandard: 50,199,622 bytes (25.9%) in 1.4 s, decompressed in 0.3 s; gzip: 23.5% but 14.5 s to compress
Fastest	LZ4: 37.8%, 0.7 s / 0.1 s; Snappy: 42.0%, 0.7 s / 0.3 s
Splits (8 MB blocks)	plain 24 blocks → 24 splits; .gz 6 → 1; .zst 6 → 1; .snappy 10 → 1; .bz2 4 → 4
Shuffle	map output 60,298,184 bytes → Snappy 2,846,994 → Zstandard 13,385

Compression at three levels

Pick your level. New to this? Start at Beginner and climb.

Beginner Packing a suitcase

Squeezing clothes tightly (bzip2) fits the most, but takes long to pack and unpack. Folding loosely (Snappy, LZ4) is quick but takes more space.
gzip is like vacuum-sealing the whole suitcase as one bag: to get one shirt, you open everything from the start.
bzip2 and container formats pack in separate pouches: anyone can open pouch 3 without touching pouches 1 and 2. That is splittability.

Intermediate What happens in a job

The input format asks CompressionCodecFactory for a codec by extension. If the codec is not splittable, isSplitable() returns false and the whole file is one split (our .gz, .zst, .snappy runs).
bzip2 marks every compressed block with a 48-bit pattern, so a reader can scan forward from a split's start to the next block. That gave our .bz2 file 4 maps.
Map output is compressed when spilled and decompressed by the reducer after the shuffle. It never leaves Hadoop, so pick the fastest codec: Snappy, LZ4 or Zstandard.
Job output compression produces part-r-00000.gz. Choose it with the next reader in mind: a later job reading one big gzip part gets one map again.

Advanced How teams choose

Data that jobs read often: store it in Parquet or ORC (Snappy or ZSTD inside). Compression is per page or stripe, so files stay splittable (Lessons 10.1, 10.2).
Raw text that must stay text: many files around the block size each (one per map), or bzip2 if it must be one big file.
Shuffle: mapreduce.map.output.compress=true with Snappy, LZ4 or ZSTD; almost always a win on real networks.
Cold archives: stronger levels of gzip, bzip2 or ZSTD; CPU is spent once and storage is saved for years.
Native libraries matter: without them Hadoop falls back to slower Java code, or fails for codecs with no Java version. Check with hadoop checknative.

Exam tip: For “Which compression codecs are splittable in Hadoop and why does it matter?”, answer: among the built-ins only bzip2 (it implements SplittableCompressionCodec); gzip, Snappy, LZ4 and Zstandard files are read by a single map task, which removes parallelism and data locality. Mention that container formats (SequenceFile, Avro, Parquet, ORC) compress internally per block and stay splittable.

3 Types and ways to use it

Hadoop's codecs compared

Codec (class)	Extension	Our ratio	Compress / decompress	Splittable	Typical use
DEFLATE (DefaultCodec)	.deflate	23.5%	18.7 s / 1.2 s	No	Default inside SequenceFile
gzip (GzipCodec)	.gz	23.5%	14.5 s / 1.0 s	No	Exchanging files with other tools
bzip2 (BZip2Codec)	.bz2	16.5%	24.2 s / 6.7 s	Yes	Big text files that must stay splittable; archives
LZ4 (Lz4Codec)	.lz4	37.8%	0.7 s / 0.1 s	No	Shuffle, hot temporary data
Snappy (SnappyCodec)	.snappy	42.0%	0.7 s / 0.3 s	No	Shuffle; default inside Parquet
Zstandard (ZStandardCodec)	.zst	25.9%	1.4 s / 0.3 s	No	Good all-rounder; Parquet/ORC; shuffle
LZO (hadoop-lzo, separate GPL project)	.lzo	not tested	fast	Only after building an index	Older clusters

Default compression levels, one thread, on our 194 MB file. Timings change with machine load (a quieter run gave about 12 s for gzip); the ratios don't.

Where you can compress

Input files

Hadoop decompresses by extension automatically. Non-splittable codecs mean one map per file, so keep each file near the block size.

read side

Map output

mapreduce.map.output.compress=true + …map.output.compress.codec. Less spill I/O and network. Our shuffle: 60 MB → 13 KB with ZSTD.

shuffle

Job output

mapreduce.output.fileoutputformat.compress=true + …compress.codec. Gives part-r-00000.gz.

write side

Inside containers

SequenceFile BLOCK, Avro blocks, Parquet pages, ORC stripes: compressed in pieces, still splittable.

best of both

hdfs dfs -text decompresses any file Hadoop has a codec for, and also decodes SequenceFiles and Avro. Use it instead of -cat to peek at compressed data.

4 Step by step

Measuring codecs on a real cluster

We use the Module 8 lab, orders_x10.csv (10 copies of Module 9's student_orders.csv, 194 MB), CodecBench.java and codecs_lab.sh from “Run it on your PC”.

Check the native libraries
```
hadoop checknative
```
hadoop: true /opt/hadoop/lib/native/libhadoop.so.1.0.0 zlib: true /lib/x86_64-linux-gnu/libz.so.1 zstd : true /lib/x86_64-linux-gnu/libzstd.so.1 bzip2: true /lib/x86_64-linux-gnu/libbz2.so.1 openssl: false Cannot load libcrypto.so … ISA-L: false Loading ISA-L failed …
zlib (gzip, deflate), zstd and bzip2 run as fast native code. Snappy and LZ4 come as Java libraries (snappy-java, lz4-java). OpenSSL and ISA-L are only needed for encryption and erasure coding.
Compress one file with every codec
```
java -Djava.library.path=/opt/hadoop/lib/native -cp mr10.jar:$(hadoop classpath):$(ls /opt/hadoop/share/hadoop/tools/lib/lz4-java-*.jar) \
  CodecBench orders_x10.csv
```
codec bytes ratio compress decompress splittable (original) 194,186,990 100% .deflate 45,696,977 23.5% 18.7 s 1.2 s no .gz 45,696,989 23.5% 14.5 s 1.0 s no .bz2 31,984,098 16.5% 24.2 s 6.7 s yes .snappy 81,629,934 42.0% 0.7 s 0.3 s no .lz4 73,417,559 37.8% 0.7 s 0.1 s no .zst 50,199,622 25.9% 1.4 s 0.3 s no
gzip and deflate are the same algorithm (gzip adds a 12-byte header and trailer). bzip2 squeezes hardest but is 17 times slower to compress than Zstandard; Snappy and LZ4 are 20 times faster than gzip but files are nearly twice as big.
Store each one in HDFS with 8 MB blocks
```
for ext in "" .gz .bz2 .zst .snappy; do
  hdfs dfs -D dfs.blocksize=8388608 -put -f orders_x10.csv$ext /user/hadoop/codec/
done
```
Small blocks make the effect visible on a small file: 24 blocks for the plain CSV, 6 for gzip, 4 for bzip2. On a real cluster the same happens with 128 MB blocks and files of tens of GB.
Run the same job on each file
```
hadoop jar mr10.jar ItemCountAny -libjars $LIBJARS /user/hadoop/codec/orders_x10.csv.gz /out/c_.gz
```
orders_x10.csv 24 blocks number of splits:24 99324 ms chai=1496690 orders_x10.csv.gz 6 blocks number of splits:1 29959 ms chai=1496690 orders_x10.csv.bz2 4 blocks number of splits:4 39627 ms chai=1496690 orders_x10.csv.zst 6 blocks number of splits:1 28175 ms chai=1496690 orders_x10.csv.snappy 10 blocks number of splits:1 28541 ms chai=1496690
The same answer every time; nothing in the job changed except the file name. Only bzip2 was split. Honest note: on our 3-worker laptop cluster one map was faster than 24 tiny ones, because task start-up dominates. Scale it up and the picture flips: a 50 GB gzip file is one map reading 50 GB, and only its first block is local to that map.
Compress the shuffle
```
hadoop jar /tmp/mr9/mr9.jar ItemPartition -D part.reducers=1 \
  -D mapreduce.map.output.compress=true \
  -D mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.ZStandardCodec \
  /user/hadoop/codec/orders_x10.csv /out/mo_ZStandardCodec
```
none 97182 ms Map output bytes 50298040 materialized 60298184 shuffle 60298184 SnappyCodec 97027 ms Map output bytes 50298040 materialized 2846994 shuffle 2846994 ZStandardCodec 100432 ms Map output bytes 50298040 materialized 13385 shuffle 13385
This job has no combiner, so 5 million (item, amount) pairs are shuffled. Map output is sorted, so it is extremely repetitive (“chai 15”, “chai 15”…): Snappy cut it 21 times, Zstandard 4,500 times. On one machine the time barely moved; on a real network 60 MB less traffic per job adds up quickly.

Compress the job's output

hadoop jar mr10.jar ItemCountAny -libjars $LIBJARS \
  -D mapreduce.output.fileoutputformat.compress=true \
  -D mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec \
  /user/hadoop/data/student_orders.csv /out/gzout
hdfs dfs -ls /out/gzout ; hdfs dfs -text /out/gzout/part-r-00000.gz | head -3

102 /out/gzout/part-r-00000.gz chai 149669 coffee 40364 dosa 49927

The part file gets the codec's extension, so any later job (or -text) knows how to read it.

A codec that works on your laptop can fail inside tasks. In Lesson 10.2, Parquet with ZSTD failed with ClassNotFoundException: com.github.luben.zstd.RecyclingBufferPool until the zstd-jni jar was added with -libjars. Test every codec on the cluster, not just in the client.

5 See it happen

gzip vs bzip2 on the same data

Six steps: how a gzip file and a bzip2 file become splits and map tasks.

Splits from compressed files 6 steps

What you see

Behind the screen

6 Real-world use

Compression at work

1. Cluster-wide shuffle compression

Platform teams set mapreduce.map.output.compress=true with Snappy, LZ4 or ZSTD in mapred-site.xml for every job; Spark does the same for its shuffle by default (LZ4).

Map output→fast codec→Less disk and network

2. Data lakes in Parquet with ZSTD or Snappy

Tables are stored as Parquet or ORC with Snappy or ZSTD inside. Many teams moved from Snappy to ZSTD for 20–40% smaller files at similar read speed.

Raw CSV/JSON→Parquet + ZSTD→Smaller, splittable, columnar

3. Landing zones with many gzip files

Logs and exports often arrive as gzip. Ingestion keeps them as many files of a few hundred MB each (one map each), then converts them to a columnar format.

Hourly .gz files→one map per file→Parquet tables

4. Cold archives

Data kept for compliance (years of transactions) is compressed hard once: bzip2 when it must stay splittable text, or high-level ZSTD inside containers.

Old partitions→strong codec→Cheaper storage

Simplified; organisations differ in tools and policies.

7 Run the code

Compare codecs on a CSV sample

Python ships three of the algorithms Hadoop uses: zlib (DEFLATE, the algorithm inside gzip), bz2 (bzip2) and lzma (xz, not a Hadoop codec but a useful “maximum squeeze” reference). Snappy, LZ4 and Zstandard need extra packages, so the lab numbers stand in for them.

import zlib, bz2, lzma, random, time
random.seed(1)
items = {"chai": 15, "coffee": 20, "samosa": 18, "vada pav": 20, "poha": 25, "idli": 30, "dosa": 45, "thali": 80}
rows = []
for oid in range(1, 20001):
    item = random.choice(list(items))
    rows.append(f"{oid},S{random.randint(1, 2000):04d},2026-09-{random.randint(1, 30):02d} {random.randint(8, 20):02d}:{random.randint(0, 59):02d},{item},{items[item]}")
data = ("\n".join(rows) + "\n").encode()
print(f"original {len(data):,} bytes")

for name, compress, decompress in [("zlib (gzip)", zlib.compress, zlib.decompress),
                                   ("bz2", bz2.compress, bz2.decompress),
                                   ("lzma (xz)", lzma.compress, lzma.decompress)]:
    t0 = time.perf_counter(); c = compress(data); t1 = time.perf_counter()
    assert decompress(c) == data
    t2 = time.perf_counter()
    print(f"{name:12} {len(c):>8,} bytes  {100 * len(c) / len(data):5.1f}%  compress {1000 * (t1 - t0):6.1f} ms  decompress {1000 * (t2 - t1):5.1f} ms")

original 751,116 bytes zlib (gzip) 185,729 bytes 24.7% compress 57.0 ms decompress 4.4 ms bz2 126,647 bytes 16.9% compress 72.8 ms decompress 27.3 ms lzma (xz) 143,888 bytes 19.2% compress 498.5 ms decompress 10.5 ms

assert decompress(c) == dataCompression is lossless: decompressing gives back exactly the original bytes.
for name, compress, decompress in [("zlib (gzip)", zlib.compress, zlib.decompress),gzip and Hadoop’s DefaultCodec both use DEFLATE (zlib); gzip only adds a small header and trailer.

Run it on your PC: six codecs, splits, shuffle and output

Use the Module 8 lab and the Module 10 setup from Lesson 10.1 (get_jars.sh and formats_env.sh, which put the extra jars on the classpath). codecs_lab.sh builds orders_x10.csv from Module 9's student_orders.csv (copied to /tmp/mr9), runs CodecBench, loads the compressed files with 8 MB blocks and runs the jobs. It takes about 12 minutes; start it in the background if your terminal times out.

Compress one file with every Hadoop codec

// CodecBench.java: compress and decompress one local file with each Hadoop codec, single thread.
// Usage: hadoop CodecBench <file>
import java.io.*;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.io.compress.*;
import org.apache.hadoop.util.ReflectionUtils;

public class CodecBench {
  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    File in = new File(args[0]);
    Class<?>[] codecs = {DefaultCodec.class, GzipCodec.class, BZip2Codec.class, SnappyCodec.class, Lz4Codec.class, ZStandardCodec.class};
    System.out.printf("%-14s %12s %7s %10s %10s %11s%n", "codec", "bytes", "ratio", "compress", "decompress", "splittable");
    System.out.printf("%-14s %,12d %7s%n", "(original)", in.length(), "100%");
    for (Class<?> c : codecs) {
      CompressionCodec codec = (CompressionCodec) ReflectionUtils.newInstance(c, conf);
      File out = new File(in.getPath() + codec.getDefaultExtension());
      long t0 = System.nanoTime();
      try (InputStream is = new FileInputStream(in); OutputStream os = codec.createOutputStream(new FileOutputStream(out))) {
        IOUtils.copyBytes(is, os, 1 << 16);
      }
      long t1 = System.nanoTime();
      try (InputStream is = codec.createInputStream(new FileInputStream(out)); OutputStream os = OutputStream.nullOutputStream()) {
        IOUtils.copyBytes(is, os, 1 << 16);
      }
      long t2 = System.nanoTime();
      System.out.printf("%-14s %,12d %6.1f%% %8.1f s %8.1f s %11s%n", codec.getDefaultExtension(), out.length(),
          100.0 * out.length() / in.length(), (t1 - t0) / 1e9, (t2 - t1) / 1e9,
          codec instanceof SplittableCompressionCodec ? "yes" : "no");
    }
  }
}

// Inside the namenode container, in /tmp/mr10 (with the other Module 10 classes):
//   javac -cp $(hadoop classpath):$HADOOP_CLASSPATH -d classes *.java && jar cf mr10.jar -C classes .
//   java -Djava.library.path=/opt/hadoop/lib/native \
//        -cp mr10.jar:$(hadoop classpath):$(ls /opt/hadoop/share/hadoop/tools/lib/lz4-java-*.jar) \
//        CodecBench orders_x10.csv

codec bytes ratio compress decompress splittable (original) 194,186,990 100% .deflate 45,696,977 23.5% 18.7 s 1.2 s no .gz 45,696,989 23.5% 14.5 s 1.0 s no .bz2 31,984,098 16.5% 24.2 s 6.7 s yes .snappy 81,629,934 42.0% 0.7 s 0.3 s no .lz4 73,417,559 37.8% 0.7 s 0.1 s no .zst 50,199,622 25.9% 1.4 s 0.3 s no

Real output, default compression levels, one thread. Times vary with load (a quieter run took 11.9 s for deflate and 14.4 s for bzip2); sizes do not. lz4-java sits in share/hadoop/tools/lib, which is not on the default classpath: without it, LZ4 fails with NoClassDefFoundError: net/jpountz/lz4/LZ4Factory.

Splits, shuffle compression and output compression

# codecs_lab.sh: Hadoop's codecs compared, which compressed files can be split, and compressing map output
cd /tmp/mr10; source formats_env.sh
for i in $(seq 10); do cat /tmp/mr9/student_orders.csv; done > orders_x10.csv
echo "== 1. one file, every codec (single thread, 194 MB of CSV)"
# lz4-java lives in share/hadoop/tools/lib, so add it; native zlib, bzip2 and zstd come from lib/native
java -Djava.library.path=/opt/hadoop/lib/native -cp mr10.jar:$(hadoop classpath):$(ls /opt/hadoop/share/hadoop/tools/lib/lz4-java-*.jar) \
  CodecBench orders_x10.csv 2>&1 | grep -vE "INFO|WARN"
echo "== 2. put them in HDFS with 8 MB blocks and run the same job on each"
hdfs dfs -mkdir -p /user/hadoop/codec
for ext in "" .gz .bz2 .zst .snappy; do
  hdfs dfs -D dfs.blocksize=8388608 -put -f orders_x10.csv$ext /user/hadoop/codec/ 2>/dev/null
done
for ext in "" .gz .bz2 .zst .snappy; do
  f=orders_x10.csv$ext
  hdfs dfs -rm -r -f -skipTrash /out/c_$ext > /dev/null 2>&1
  start=$(date +%s%N)
  hadoop jar mr10.jar ItemCountAny -libjars $LIBJARS /user/hadoop/codec/$f /out/c_$ext > /tmp/c_$ext.log 2>&1
  printf "%-20s %5s blocks  %-18s %6d ms  chai=%s\n" $f $(hdfs fsck /user/hadoop/codec/$f 2>/dev/null | grep "Total blocks" | awk '{print $4}') \
    "$(grep -o 'number of splits:[0-9]*' /tmp/c_$ext.log)" $(( ($(date +%s%N) - start) / 1000000 )) \
    $(hdfs dfs -cat /out/c_$ext/part-r-00000 | grep chai | cut -f2)
done
echo "== 3. compress the map output (shuffle) of a job without a combiner"
for codec in none SnappyCodec ZStandardCodec; do
  opts="-D part.reducers=1"
  [ $codec != none ] && opts="$opts -D mapreduce.map.output.compress=true -D mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.$codec"
  hdfs dfs -rm -r -f -skipTrash /out/mo_$codec > /dev/null 2>&1
  start=$(date +%s%N)
  hadoop jar /tmp/mr9/mr9.jar ItemPartition $opts /user/hadoop/codec/orders_x10.csv /out/mo_$codec > /tmp/mo_$codec.log 2>&1
  printf "%-15s %6d ms  Map output bytes %s  materialized %s  shuffle %s\n" $codec $(( ($(date +%s%N) - start) / 1000000 )) \
    $(grep "Map output bytes=" /tmp/mo_$codec.log | cut -d= -f2) $(grep "Map output materialized bytes=" /tmp/mo_$codec.log | cut -d= -f2) \
    $(grep "Reduce shuffle bytes=" /tmp/mo_$codec.log | cut -d= -f2)
done
echo "== 4. compress the job's final output"
hdfs dfs -rm -r -f -skipTrash /out/gzout > /dev/null 2>&1
hadoop jar mr10.jar ItemCountAny -libjars $LIBJARS -D mapreduce.output.fileoutputformat.compress=true \
  -D mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec \
  /user/hadoop/data/student_orders.csv /out/gzout > /dev/null 2>&1
hdfs dfs -ls /out/gzout | grep part | awk '{print $5, $8}'
hdfs dfs -text /out/gzout/part-r-00000.gz 2>/dev/null | grep -v INFO | head -3

# Run inside the namenode container:  bash /tmp/mr10/codecs_lab.sh

== 1. one file, every codec (single thread, 194 MB of CSV) codec bytes ratio compress decompress splittable (original) 194,186,990 100% .deflate 45,696,977 23.5% 18.7 s 1.2 s no .gz 45,696,989 23.5% 14.5 s 1.0 s no .bz2 31,984,098 16.5% 24.2 s 6.7 s yes .snappy 81,629,934 42.0% 0.7 s 0.3 s no .lz4 73,417,559 37.8% 0.7 s 0.1 s no .zst 50,199,622 25.9% 1.4 s 0.3 s no == 2. put them in HDFS with 8 MB blocks and run the same job on each orders_x10.csv 24 blocks number of splits:24 99324 ms chai=1496690 orders_x10.csv.gz 6 blocks number of splits:1 29959 ms chai=1496690 orders_x10.csv.bz2 4 blocks number of splits:4 39627 ms chai=1496690 orders_x10.csv.zst 6 blocks number of splits:1 28175 ms chai=1496690 orders_x10.csv.snappy 10 blocks number of splits:1 28541 ms chai=1496690 == 3. compress the map output (shuffle) of a job without a combiner none 97182 ms Map output bytes 50298040 materialized 60298184 shuffle 60298184 SnappyCodec 97027 ms Map output bytes 50298040 materialized 2846994 shuffle 2846994 ZStandardCodec 100432 ms Map output bytes 50298040 materialized 13385 shuffle 13385 == 4. compress the job's final output 102 /out/gzout/part-r-00000.gz chai 149669 coffee 40364 dosa 49927

Real output. All five input files give the same answer (chai=1496690); only bzip2 was split. The Snappy and Zstandard shuffles moved 21× and 4,500× fewer bytes.

8 Line by line

Why gzip can't be split

Compress a short file and try to decompress from the start and from the middle. Watch which one works.

Worked examples: real problems, solved and traced in memory

Three compression decisions: storage saved, shuffle saved, and choosing a codec by reading pattern.

Example 1BeginnerStorage · savings

How much disk does a codec save?

Problem statement

The canteen keeps 2 TB of raw CSV per year in HDFS with replication 3. Using our measured ratios (gzip 23.5%, Zstandard 25.9%, Snappy 42%), how much raw disk does each codec need?

Input

2 TB × 3 replicas

Expected output

none     6.00 TB of disk
gzip     1.41 TB of disk
zstd     1.55 TB of disk
snappy   2.52 TB of disk

Approach

Raw disk = data × replication.
Compressed = raw × ratio.
Compare.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 2IntermediateShuffle · network

What does map output compression save per day?

Problem statement

A daily job without a combiner shuffles 600 GB of map output across a 1 Gbit/s (125 MB/s) network shared by 20 nodes. If Snappy reduces it to 5% (our lab gave 4.7%), how much network time per node is saved?

Input

600 GB · 20 nodes · 125 MB/s each · 5%

Expected output

plain 4.1 min, snappy 0.2 min per node

Approach

Bytes per node = total / nodes.
Time = bytes / bandwidth.
Compare plain and compressed.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 3AdvancedDesign · choice

Pick a codec for three datasets

Problem statement

Choose a format and codec for: (a) a 40 GB daily log file that jobs read many times, (b) map output of a heavy join job, (c) 5 years of transactions kept only for audits.

Input

3 use cases

Expected output

daily log, read often  -> convert to Parquet/ORC with Snappy or ZSTD (splittable, columnar)
join map output        -> mapreduce.map.output.compress=true with Snappy, LZ4 or ZSTD
5-year audit archive   -> ORC/Parquet with ZSTD at a high level, or bzip2 if it must stay text

Approach

Read often and in parallel: splittable, fast to read.
Shuffle: fastest codec.
Cold: smallest size, CPU spent once.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

9 Quiz and revision

Check yourself

Quick revision

Lesson 10.3 in one screen

Compression trades CPU for less disk, network and I/O. Ratio and speed vary hugely between codecs.
Lab, 194 MB CSV: bzip2 16.5% (24 s), gzip 23.5% (14.5 s), ZSTD 25.9% (1.4 s), LZ4 37.8% (0.7 s), Snappy 42% (0.7 s).
Only bzip2 is splittable among built-ins. Lab with 8 MB blocks: plain 24 splits, .gz/.zst/.snappy 1, .bz2 4.
CompressionCodecFactory picks the codec by extension; hdfs dfs -text decompresses for you.
Map output: mapreduce.map.output.compress=true + Snappy/LZ4/ZSTD. Lab shuffle 60 MB → 2.8 MB → 13 KB.
Job output: mapreduce.output.fileoutputformat.compress=true + codec → part-r-00000.gz.
Best for data read often: Parquet/ORC (or SequenceFile/Avro) with compression inside: compact AND splittable.

Interview and viva questions

Which compression formats are splittable in Hadoop?

Among the codecs shipped with Hadoop only bzip2 (BZip2Codec) is splittable. gzip, DEFLATE, Snappy, LZ4 and Zstandard files are each processed by a single map task. LZO becomes splittable only after building an index with the separate hadoop-lzo project. Container formats such as SequenceFile, Avro, Parquet and ORC remain splittable whatever codec they use internally, because they compress block by block.

Why is gzip not splittable?

A gzip file is one continuous DEFLATE stream: decoding any part requires the state built from everything before it, and there are no markers where a reader could safely begin. A map task assigned the middle of the file could not decompress it, so Hadoop makes the whole file one split.

How do I enable compression in a MapReduce job?

For intermediate data set mapreduce.map.output.compress=true and mapreduce.map.output.compress.codec (for example org.apache.hadoop.io.compress.SnappyCodec). For the final output set mapreduce.output.fileoutputformat.compress=true and mapreduce.output.fileoutputformat.compress.codec, or call FileOutputFormat.setCompressOutput and setOutputCompressorClass in the driver. Compressed input is detected automatically from the file extension.

Which codec should I use in Hadoop?

For the shuffle, a fast codec: Snappy, LZ4 or Zstandard. For data that jobs read often, a columnar container (Parquet or ORC) with Snappy or Zstandard. For plain text that must be one large splittable file, bzip2. For archives, a high compression level of Zstandard, gzip or bzip2. Always test on your own data, because ratios depend heavily on content.

What are native libraries in Hadoop?

libhadoop.so and system libraries (zlib, libzstd, libbz2) that implement compression and other functions in C for speed. hadoop checknative lists which are loaded. Without them Hadoop uses slower Java implementations where they exist, and codecs without a Java version fail.

Practice problems

Compression exercises using the real numbers from the lab.

P1Easy

Compression ratios

Compute each codec's compressed size as a percentage of 194,186,990 bytes.

Input

CodecBench sizes

Expected output

gzip    23.5%
bzip2   16.5%
snappy  42.0%
lz4     37.8%
zstd    25.9%

Show solution and explanation

orig = 194_186_990
for name, size in [("gzip", 45_696_989), ("bzip2", 31_984_098), ("snappy", 81_629_934), ("lz4", 73_417_559), ("zstd", 50_199_622)]:
    print(f"{name:7} {100 * size / orig:.1f}%")

gzip 23.5% bzip2 16.5% snappy 42.0% lz4 37.8% zstd 25.9%

How it works: 23.5, 16.5, 42.0, 37.8 and 25.9%, the same numbers CodecBench printed.

P2Easy

gzip vs deflate

gzip and deflate use the same algorithm. How many bytes larger is the .gz file?

Input

45,696,989 vs 45,696,977

Expected output

12 bytes

Show solution and explanation

print(45_696_989 - 45_696_977, "bytes")

12 bytes

How it works: 12 bytes: only the gzip header and trailer differ.

P3Easy

Shuffle saved

How many times smaller was the shuffle with Snappy and with Zstandard?

Input

60,298,184 · 2,846,994 · 13,385

Expected output

21.2 4505

Show solution and explanation

print(round(60_298_184 / 2_846_994, 1), round(60_298_184 / 13_385))

21.2 4505

How it works: About 21× and 4,505×. Sorted, repetitive map output compresses extremely well.

P4Medium

Throughput

Compute compression speed in MB/s for gzip, ZSTD and LZ4 from the lab times.

Input

194 MB · 14.5 s · 1.4 s · 0.7 s

Expected output

gzip      13 MB/s
zstd     132 MB/s
lz4      265 MB/s

Show solution and explanation

mb = 194_186_990 / 1024 / 1024
for name, s in [("gzip", 14.5), ("zstd", 1.4), ("lz4", 0.7)]:
    print(f"{name:5} {mb / s:6.0f} MB/s")

gzip 13 MB/s zstd 132 MB/s lz4 265 MB/s

How it works: About 13, 132 and 265 MB/s on one thread: a slow codec can become the bottleneck before the disk does.

P5Medium

Blocks of a compressed file

How many 8 MB blocks does each compressed file need? Compare with the fsck counts (6, 4, 6, 10).

Input

sizes from CodecBench

Expected output

gz 6
bz2 4
zst 6
snappy 10

Show solution and explanation

import math
for name, size in [("gz", 45_696_989), ("bz2", 31_984_098), ("zst", 50_199_622), ("snappy", 81_629_934)]:
    print(name, math.ceil(size / (8 * 1024 * 1024)))

gz 6 bz2 4 zst 6 snappy 10

How it works: 6, 4, 6 and 10, matching fsck. Only for bz2 did blocks become splits.

P6Medium

Many gzip files instead of one

A 40 GB log is written as many gzip files holding 250 MB of text each, instead of one big file. How many map tasks now?

Input

40 GB · 250 MB per file

Expected output

164 map tasks (one per file)

Show solution and explanation

import math
print(math.ceil(40 * 1024 / 250), "map tasks (one per file)")

164 map tasks (one per file)

How it works: 164 map tasks: non-splittable codecs are fine when each file is small enough to be one task.

P7Business problem

Storage bill

Cloud storage costs ₹1.8 per GB per month. 50 TB of CSV is converted to Parquet with ZSTD at about 15% of the size (Lesson 10.2 got 14.3%). What is saved per year?

Input

50 TB · ₹1.8/GB-month · 15%

Expected output

Rs 940,032 per year

Show solution and explanation

gb = 50 * 1024
saved = gb * (1 - 0.15) * 1.8 * 12
print(f"Rs {saved:,.0f} per year")

Rs 940,032 per year

How it works: About ₹9.4 lakh a year (illustrative prices), and scans also read 85% fewer bytes.

P8Advanced

Split planner with locality

Write a function that, for a file size, block size and codec, returns the number of map tasks and how many blocks must be read remotely (assuming each splittable map is data-local and a single map is local only to block 1).

Input

file, block, codec

Expected output

none     400 maps,    0 blocks read over the network
bzip2    400 maps,    0 blocks read over the network
gzip       1 maps,  399 blocks read over the network
snappy     1 maps,  399 blocks read over the network
zstd       1 maps,  399 blocks read over the network

Show solution and explanation

import math
SPLITTABLE = {"none": True, "bzip2": True, "gzip": False, "snappy": False, "zstd": False}

def plan(size_mb, block_mb, codec):
    blocks = math.ceil(size_mb / block_mb)
    if SPLITTABLE[codec]:
        return blocks, 0
    return 1, blocks - 1

for codec in SPLITTABLE:
    maps, remote = plan(50 * 1024, 128, codec)
    print(f"{codec:7} {maps:4} maps, {remote:4} blocks read over the network")

none 400 maps, 0 blocks read over the network bzip2 400 maps, 0 blocks read over the network gzip 1 maps, 399 blocks read over the network snappy 1 maps, 399 blocks read over the network zstd 1 maps, 399 blocks read over the network

How it works: For 50 GB: 400 maps and no remote reads when splittable, versus 1 map pulling 399 blocks over the network for gzip, Snappy or ZSTD.

Real-world scenario questions (asked in company interviews)

Compression questions are common for data engineers: splittability, codec choice and where to compress.

Concepts · splittabilityFresher

“Why does my job on a 30 GB .gz file run only one mapper?”

A classic first question about compression.

The interviewer is checking: splittable vs non-splittable codecs.

Show a strong answer

gzip is one DEFLATE stream with no restart points, so the input format makes one split for the whole file.
Fixes: recompress with bzip2, split the data into many gzip files near the block size, or convert to Parquet/ORC.
Also mention lost data locality: one map reads most blocks over the network.

Performance · shuffleExperienced

A join job's shuffle saturates the network

Reducers spend most of their time in the copy phase.

The interviewer is checking: map output compression and other shuffle reducers.

Show a strong answer

Enable mapreduce.map.output.compress with Snappy, LZ4 or ZSTD; cheap CPU for much less network.
Add a combiner if the reduce is an aggregation, and emit compact types.
If one side is small, use a map-side join and skip the shuffle (Lesson 9.6).

Design · storageExperienced

Choose a storage format and codec for a new data lake

Analysts query a few columns; data arrives as gzipped JSON.

The interviewer is checking: container formats vs raw compressed files.

Show a strong answer

Land the gzip files as they come, then convert to Parquet or ORC with Snappy or ZSTD.
Columnar + compression per page/stripe gives small, splittable files and column pruning.
Keep file sizes around 128 MB–1 GB and compact small files.

Operations · debuggingExperienced

A job with Zstandard works locally but fails on the cluster

Tasks fail with ClassNotFoundException or “native zstd library not available”.

The interviewer is checking: native libraries and task classpaths.

Show a strong answer

Run hadoop checknative on the worker nodes; install libzstd or the Hadoop native build if missing.
Ship missing jars (for example zstd-jni for Parquet) with -libjars or install them cluster-wide.
Read the failed attempt's diagnostics in the JobHistory UI or yarn logs, not just the client output.

Compression codecs and splittability

One gzip file, one lonely map task

Compression codecs and splittability

Words you'll meet in this lesson

From our lab (Hadoop 3.5.0, 194,186,990 bytes of CSV, one thread)

Compression at three levels

Beginner Packing a suitcase

Intermediate What happens in a job

Advanced How teams choose

Hadoop's codecs compared

Where you can compress

Input files

Map output

Job output

Inside containers

Measuring codecs on a real cluster

Check the native libraries

Compress one file with every codec

Store each one in HDFS with 8 MB blocks

Run the same job on each file

Compress the shuffle

Compress the job's output

gzip vs bzip2 on the same data

Compression at work

1. Cluster-wide shuffle compression

2. Data lakes in Parquet with ZSTD or Snappy

3. Landing zones with many gzip files

4. Cold archives

Compare codecs on a CSV sample

Run it on your PC: six codecs, splits, shuffle and output

Compress one file with every Hadoop codec

Splits, shuffle compression and output compression

Why gzip can't be split

Worked examples: real problems, solved and traced in memory

How much disk does a codec save?

What does map output compression save per day?

Pick a codec for three datasets

Check yourself

Quick revision

Lesson 10.3 in one screen

Interview and viva questions

Practice problems

Compression ratios

gzip vs deflate

Shuffle saved

Throughput

Blocks of a compressed file

Many gzip files instead of one

Storage bill

Split planner with locality

Real-world scenario questions (asked in company interviews)

“Why does my job on a 30 GB .gz file run only one mapper?”

A join job's shuffle saturates the network

Choose a storage format and codec for a new data lake

A job with Zstandard works locally but fails on the cluster