Text, SequenceFile and Avro

1 The story

The day someone added a column

The canteen's 5 lakh orders live in HDFS as student_orders.csv. On Monday the billing team added a payment column at the end of each new line. On Tuesday Kabir's monthly report crashed: his job expected five fields and found six. Riya's Python script didn't crash, but it read "cash" as the amount. Nothing in a CSV file says what its columns are called or what type they have. Arjun Sir pulls up a different version of the same data.

Arjun Sir

A file format is a contract between whoever writes the data and everyone who will ever read it. CSV's contract is “some text with commas”. That is easy to read with your eyes, and that is all it promises.

Riya

So binary formats are smaller, and they also remember the column names?

Arjun Sir

Some of them. A SequenceFile stores binary keys and values and writes their Java classes in its header. An Avro file stores a full schema, field names and types, in its header, so a reader in Java, Python or anything else knows exactly what each record contains. And Avro has rules for changing that schema without breaking old files.

Kabir

Today we convert the same orders into all of these, compare the sizes, read them back with one job, and then add Monday's payment column the safe way.

2 Definition

Row-oriented file formats in Hadoop

Formal definition (write this in exams)

A file format defines how records are encoded into bytes in a file. In Hadoop it is read by an InputFormat and written by an OutputFormat. Text files (TextInputFormat) store one record per line as characters with no schema. A SequenceFile is Hadoop's flat binary file of key-value pairs: a header names the key and value classes (Writables) and the compression codec, records may be stored uncompressed, record-compressed or block-compressed, and periodic sync markers let a reader start in the middle of the file, making it splittable. Apache Avro is a language-neutral data serialization system: an Avro data file (object container file) stores its schema as JSON in the header, followed by blocks of compactly encoded binary records separated by sync markers, each block optionally compressed. Avro supports schema evolution: data written with one schema (the writer's schema) can be read with a different but compatible schema (the reader's schema). All three are row-oriented: the fields of one record are stored together.

In simple words

Text is human-readable but has no types. SequenceFile is Hadoop's own compact key-value box. Avro is a compact box that carries its own label (the schema), works in any language, and lets the label change over time without breaking old boxes.

Words you'll meet in this lesson

SerializationTurning an object or record into bytes, and back (deserialization).

SchemaThe names and types of a record's fields.

Sync markerA 16-byte random marker repeated in the file so a reader can find a record boundary.

SplittableA reader can start in the middle of the file, so many map tasks can share it.

Block compressionCompressing many records together, which finds far more repetition.

Writer's schemaThe schema the data was written with (stored in the Avro file).

Reader's schemaThe schema your program expects; Avro resolves the two.

Row-orientedAll fields of a record stored together (vs columnar, Lesson 10.2).

From our lab (Hadoop 3.5.0, 5 lakh orders, one replica)

Text (CSV)	19,418,699 bytes, about 39 bytes per order
SequenceFile	NONE 24,034,722 · RECORD (deflate) 27,878,984 · BLOCK (deflate) 4,683,378
Avro	no codec 16,558,165 · snappy 7,791,915 · deflate 4,793,471
Schema evolution	Old file read with a new schema: `"payment":"cash"` filled from the default; without a default: “AvroTypeException: … missing required field payment”
Same job, three formats	chai = 149,669 every time; HDFS bytes read 19,427,239 (text) vs 4,683,705 (SequenceFile) vs 4,818,410 (Avro); about 25 s each

File formats at three levels

Pick your level. New to this? Start at Beginner and climb.

Beginner Three ways to keep a register

Text is a notebook where each order is written as one line. Anyone can read it, but nothing says which number is the amount and which is the roll number.
SequenceFile is a register with two columns, key and value, printed in Hadoop's own shorthand. Compact, but you need Hadoop (or Java) to read it.
Avro is a register with the column headings printed on the first page, in a shorthand every language understands. When a new column is added, old pages still make sense.

Intermediate What the bytes look like

Text: numbers as characters (“45” is 2 bytes, “1000001” is 7), commas, a newline. Our CSV: about 39 bytes per order.
SequenceFile NONE: every record stores its record length and key length (4 + 4 bytes), the key as an 8-byte LongWritable, then the value as Text, plus a 20-byte sync entry about every 100 KB. That's why it was bigger than the CSV: 24.0 MB vs 19.4 MB.
SequenceFile RECORD: each value compressed on its own. A 30-byte value can't shrink, and each one pays the deflate overhead: 27.9 MB, the worst of all. BLOCK compresses many records together: 4.7 MB.
Avro: no field names or separators per record (the schema says what comes next), numbers as variable-length zig-zag integers, strings as length + bytes. Without compression already 16.6 MB, smaller than the CSV.

Advanced Splitting, compression and evolution

Both binary formats put a 16-byte sync marker between blocks. A map task given bytes 128 MB–256 MB scans forward to the next marker and starts there, like LineRecordReader does with newlines (Lesson 8.2). So they stay splittable even when compressed, unlike a gzip'd CSV (Lesson 10.3).
Avro schema resolution matches fields by name: a field in the reader's schema but not in the file needs a default; a field in the file but not the reader's schema is skipped; renames use aliases; types can be promoted (int → long, float → double).
Compatibility is directional: backward = new readers can read old data (add fields with defaults); forward = old readers can read new data (remove only fields that had defaults). Kafka's Schema Registry enforces exactly these rules.
SequenceFile's types are Java class names, so other languages struggle with it. It is best for Hadoop-internal data: intermediate job output, or packing small files (Lesson 7.5).

Exam tip: For “Compare Text, SequenceFile and Avro”, use four headings: schema (none / Java classes in header / JSON schema in header), size (largest / compact with BLOCK compression / compact), splittable when compressed (no with gzip / yes / yes), languages (any / mainly Java / any). Add one line on Avro schema evolution: new fields need defaults.

3 Types and ways to use it

Text vs SequenceFile vs Avro

	Text / CSV	SequenceFile	Avro
Human-readable	Yes	No (use `hdfs dfs -text`)	No (use `hdfs dfs -text` or avro-tools)
Schema	None	Key and value class names	Full JSON schema in the header
Our size	19.4 MB	4.7 MB (BLOCK, deflate)	4.8 MB (deflate)
Splittable compressed	Only with bzip2 (Lesson 10.3)	Yes (sync markers)	Yes (sync markers)
Schema evolution	By convention only	No	Yes, with defaults and aliases
Languages	Any	Java mostly	Java, Python, C, C++, C#, Rust, …
Typical use	Landing raw data, exports	Intermediate MR data, packing small files	Ingestion, Kafka messages, data exchange

The InputFormat and OutputFormat for each

Text

TextInputFormat gives (byte offset, line); TextOutputFormat writes key TAB value lines.

default

SequenceFile

SequenceFileInputFormat / SequenceFileOutputFormat; the job's key and value classes must match the file's.

hadoop-common

Avro

AvroKeyInputFormat / AvroKeyOutputFormat from avro-mapred; set schemas with AvroJob.

extra jar

Choosing

Exchanging data or expecting schema changes: Avro. Hadoop-only intermediate data: SequenceFile. Analytics on a few columns: Parquet or ORC (next lesson).

rule of thumb

Row formats are fast to write one record at a time, which is why streaming pipelines (Kafka → HDFS) usually land data as Avro. Analytical tables are then rewritten into columnar formats, which are faster to read when a query needs only a few columns. Lesson 10.2 measures that difference.

4 Step by step

Converting the orders and reading them back

We use the Module 8 lab cluster and Module 9's student_orders.csv. Hadoop ships the Avro core library but not avro-mapred (the Avro InputFormat and OutputFormat), so get_jars.sh downloads it from Maven Central, and formats_env.sh passes it to the job with -libjars.

Write SequenceFiles three ways
```
hadoop jar mr10.jar ToSequenceFile -libjars $LIBJARS -D seq.compress=NONE   $D /out/f_seq_none
hadoop jar mr10.jar ToSequenceFile -libjars $LIBJARS -D seq.compress=RECORD $D /out/f_seq_record
hadoop jar mr10.jar ToSequenceFile -libjars $LIBJARS -D seq.compress=BLOCK  $D /out/f_seq_block
```
text 19418699 seq_none 24034722 seq_record 27878984 seq_block 4683378
A map-only job turns each line into (order_id as LongWritable, rest of the line as Text). Uncompressed it grows by about 9 bytes per order (lengths, the 8-byte key). Compressing each tiny value separately makes it even bigger; compressing blocks of records shrinks it to a quarter of the CSV.
Look at the first bytes
```
hdfs dfs -cat /out/f_seq_block/part-m-00000 | head -c 150
```
SEQ.!org.apache.hadoop.io.LongWritable.org.apache.hadoop.io.Text..*org.apache.hadoop.io.compress.DefaultCodec....P...........8`......P...........8`...
The header: magic bytes SEQ, a version byte, the key class, the value class, the compression flags and codec, then a 16-byte sync marker. Any reader learns the types from the file itself.
Write Avro three ways
```
hadoop jar mr10.jar ToAvro -libjars $LIBJARS -D avro.codec=null    $D /out/f_avro_null
hadoop jar mr10.jar ToAvro -libjars $LIBJARS -D avro.codec=deflate $D /out/f_avro_deflate
hadoop jar mr10.jar ToAvro -libjars $LIBJARS -D avro.codec=snappy  $D /out/f_avro_snappy
```
avro_null 16558165 avro_deflate 4793471 avro_snappy 7791915
Even with no codec, Avro is smaller than the CSV: no commas, no field names per record, and numbers stored as variable-length integers. Deflate compresses better; snappy is faster but larger (Lesson 10.3).
Read the binary files as text
```
hdfs dfs -text /out/f_seq_block/part-m-00000 | head -2
hdfs dfs -text /out/f_avro_deflate/part-m-00000.avro | head -2
java -jar lib/avro-tools-1.11.5.jar getmeta orders.avro
```
1 S1413,2026-09-14 15:30,dosa,45 2 S0973,2026-09-26 20:11,samosa,18 {"order_id":1,"student":"S1413","time":"2026-09-14 15:30","item":"dosa","amount":45} {"order_id":2,"student":"S0973","time":"2026-09-26 20:11","item":"samosa","amount":18} avro.schema {"type":"record","name":"Order","namespace":"in.myphatshala","fields":[{"name":"order_id","type":"long"},… avro.codec deflate
-text (unlike -cat) recognises SequenceFiles and Avro files and decodes them. The Avro file's metadata holds the whole schema and the codec name.
Add Monday's column the safe way
```
{"name":"payment","type":"string","default":"cash"}
```
{"order_id":432369,"student":"S0454","time":"2026-09-20 12:05","item":"chai","amount":15,"payment":"cash"} {"order_id":432370,"student":"S1436","time":"2026-09-18 18:50","item":"chai","amount":15,"payment":"cash"} -- and a new field WITHOUT a default: AvroTypeException: Found in.myphatshala.Order, expecting in.myphatshala.Order, missing required field payment
The old file was written without payment. Reading it with the new schema, Avro fills the default. Without a default there is no value to fill, so Avro refuses instead of guessing, a loud failure rather than Riya's silent one.
Read all three with the same job
```
hadoop jar mr10.jar ItemCountAny -libjars $LIBJARS -D in.format=text $D /out/r_text
hadoop jar mr10.jar ItemCountAny -libjars $LIBJARS -D in.format=seq  /out/f_seq_block /out/r_seq
hadoop jar mr10.jar ItemCountAny -libjars $LIBJARS -D in.format=avro /out/f_avro_deflate /out/r_avro
```
text 25221 ms HDFS bytes read 19427239 map input records 500000 chai=149669 seq_block 25073 ms HDFS bytes read 4683705 map input records 500000 chai=149669 avro_deflate 25813 ms HDFS bytes read 4818410 map input records 500000 chai=149669
Same answer, a quarter of the bytes read from HDFS. The times are equal here because 19 MB is tiny and job start-up dominates; on terabytes, reading 4× fewer bytes is the difference that matters, paid for with CPU to decompress.

Don't use RECORD compression for small records: our SequenceFile grew from 24.0 MB to 27.9 MB. Use BLOCK compression (or Avro with a codec), which compresses many records together.

5 See it happen

One order, three encodings

Six steps: the same order as text, as a SequenceFile record, compressed record by record and in blocks, as Avro, and read with a newer schema.

Inside the files 6 steps

What you see

Behind the screen

6 Real-world use

Row formats at work

1. Event streams with a schema registry

Payment, order and click events are published to Kafka as Avro, with schemas stored in a schema registry that rejects incompatible changes. Connectors then land them in HDFS or a data lake as Avro files.

App events→Kafka + Avro schema→HDFS landing zone

2. Packing small files

Images, PDFs or tiny logs are packed into SequenceFiles with the file name as key and the bytes as value, so the NameNode tracks one file instead of thousands (Lesson 7.5).

Small files→(name, bytes) records→One splittable file

3. Data between chained jobs

When one MapReduce job feeds another, the middle data is written as block-compressed SequenceFiles: binary Writables need no parsing, and Hive, Pig and Mahout used them this way.

Job 1→SequenceFile (BLOCK)→Job 2

4. Raw zone, then curated tables

Data lakes keep the raw, append-only zone in Avro (fast writes, full records, schema history), then nightly jobs rewrite it into Parquet or ORC tables for analysts (Lesson 10.2).

Raw (Avro)→nightly rewrite→Curated (Parquet/ORC)

Simplified; organisations differ in tools and policies.

7 Run the code

Encode one order three ways

This model builds the bytes of one order as a CSV line, as an uncompressed SequenceFile record, and as an Avro record (zig-zag variable-length integers, strings as length + bytes). The layouts follow the real formats; the file headers are left out.

import struct

order = (1, "S1413", "2026-09-14 15:30", "dosa", 45)

def zigzag_varint(n):                       # Avro's int/long encoding
    n = (n << 1) ^ (n >> 63)                # zig-zag: 0,-1,1,-2 -> 0,1,2,3
    out = b""
    while n > 0x7F:
        out += bytes([(n & 0x7F) | 0x80])   # 7 bits per byte, high bit = "more follows"
        n >>= 7
    return out + bytes([n])

csv = (",".join(map(str, order)) + "\n").encode()

value = ",".join(map(str, order[1:])).encode()               # Text value
seq = struct.pack(">ii", 8 + 1 + len(value), 8) + struct.pack(">q", order[0]) + bytes([len(value)]) + value

avro = zigzag_varint(order[0])
for s in order[1:4]:
    avro += zigzag_varint(len(s)) + s.encode()
avro += zigzag_varint(order[4])

for name, b in [("CSV", csv), ("SequenceFile", seq), ("Avro", avro)]:
    print(f"{name:13} {len(b):3} bytes  {b[:24]!r}")

CSV 33 bytes b'1,S1413,2026-09-14 15:30' SequenceFile 47 bytes b"\x00\x00\x00'\x00\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x01\x1eS1413,2" Avro 30 bytes b'\x02\nS1413 2026-09-14 15:30'

n = (n << 1) ^ (n >> 63) # zig-zag: 0,-1,1,-2 -> 0,1,2,3Zig-zag maps small negative numbers to small positive ones, so -1 also fits in one byte.
seq = struct.pack(">ii", 8 + 1 + len(value), 8) + struct.pack(">q", order[0]) + bytes([len(value)]) + valueRecord length, key length, an 8-byte LongWritable key, then the Text value with its length. About 9 bytes more than the CSV line per order.
avro += zigzag_varint(order[4])No field names and no separators: the schema in the file header says what comes next.

Run it on your PC: one dataset, seven encodings

Use the Module 8 lab cluster (docker compose -p lab8 up -d) and Module 9's student_orders.csv, already in HDFS. Hadoop ships the Avro core library but not avro-mapred, which holds Avro's InputFormat and OutputFormat, so get_jars.sh (in the last item) downloads it and the other format jars from Maven Central into /tmp/mr10/lib, and formats_env.sh puts them on the client classpath and ships them to the tasks with -libjars. Then compile the Java files into mr10.jar and run the lab script; it takes about 6 minutes.

CSV to SequenceFile

// ToSequenceFile.java: map-only job. CSV orders -> SequenceFile of (order_id, "student,time,item,amount").
//   -D seq.compress=NONE|RECORD|BLOCK   (codec: -D seq.codec=default|snappy|zstd|gzip)
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.SequenceFile.CompressionType;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.compress.*;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class ToSequenceFile {
  public static class M extends Mapper<LongWritable, Text, LongWritable, Text> {
    private final LongWritable id = new LongWritable();
    private final Text rest = new Text();
    @Override
    protected void map(LongWritable k, Text line, Context ctx) throws IOException, InterruptedException {
      String s = line.toString();
      int comma = s.indexOf(',');
      id.set(Long.parseLong(s.substring(0, comma)));
      rest.set(s.substring(comma + 1));
      ctx.write(id, rest);
    }
  }

  static Class<? extends CompressionCodec> codec(String name) {
    switch (name) {
      case "snappy": return SnappyCodec.class;
      case "zstd": return ZStandardCodec.class;
      case "gzip": return GzipCodec.class;
      default: return DefaultCodec.class;           // zlib/deflate
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    String[] rest = new GenericOptionsParser(conf, args).getRemainingArgs();
    CompressionType type = CompressionType.valueOf(conf.get("seq.compress", "NONE"));
    Job job = Job.getInstance(conf, "to sequencefile " + type);
    job.setJarByClass(ToSequenceFile.class);
    job.setMapperClass(M.class);
    job.setNumReduceTasks(0);
    job.setOutputKeyClass(LongWritable.class);
    job.setOutputValueClass(Text.class);
    job.setOutputFormatClass(SequenceFileOutputFormat.class);
    if (type != CompressionType.NONE) {
      FileOutputFormat.setCompressOutput(job, true);
      FileOutputFormat.setOutputCompressorClass(job, codec(conf.get("seq.codec", "default")));
      SequenceFileOutputFormat.setOutputCompressionType(job, type);
    }
    FileInputFormat.addInputPath(job, new Path(rest[0]));
    FileOutputFormat.setOutputPath(job, new Path(rest[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

// Inside the namenode container, in /tmp/mr10 (after get_jars.sh):
//   source formats_env.sh
//   mkdir -p classes && javac -cp $(hadoop classpath):$HADOOP_CLASSPATH -d classes *.java && jar cf mr10.jar -C classes .
//   hadoop jar mr10.jar ToSequenceFile -libjars $LIBJARS -D seq.compress=BLOCK /user/hadoop/data/student_orders.csv /out/f_seq_block

== size on HDFS (one replica) text 19418699 seq_none 24034722 seq_record 27878984 seq_block 4683378 text 25221 ms HDFS bytes read 19427239 map input records 500000 chai=149669 seq_block 25073 ms HDFS bytes read 4683705 map input records 500000 chai=149669 == the first bytes of a SequenceFile (a header that names its own types) SEQ.!org.apache.hadoop.io.LongWritable.org.apache.hadoop.io.Text..*org.apache.hadoop.io.compress.DefaultCodec....P...........8`......P...........8`... == hdfs dfs -text decodes both binary formats 1 S1413,2026-09-14 15:30,dosa,45 2 S0973,2026-09-26 20:11,samosa,18

Real output from rowformats_lab.sh (below): the sizes, the first 150 bytes of the block-compressed file (non-printable bytes shown as dots), and hdfs dfs -text decoding it.

CSV to Avro, then a new schema

// ToAvro.java: map-only job. CSV orders -> Avro data files with an explicit schema.
//   -D avro.codec=null|deflate|snappy|zstandard
import java.io.IOException;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.mapred.AvroKey;
import org.apache.avro.mapreduce.AvroJob;
import org.apache.avro.mapreduce.AvroKeyOutputFormat;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class ToAvro {
  static final String SCHEMA = "{\"type\":\"record\",\"name\":\"Order\",\"namespace\":\"in.myphatshala\",\"fields\":["
      + "{\"name\":\"order_id\",\"type\":\"long\"},"
      + "{\"name\":\"student\",\"type\":\"string\"},"
      + "{\"name\":\"time\",\"type\":\"string\"},"
      + "{\"name\":\"item\",\"type\":\"string\"},"
      + "{\"name\":\"amount\",\"type\":\"int\"}]}";

  public static class M extends Mapper<LongWritable, Text, AvroKey<GenericRecord>, NullWritable> {
    private Schema schema;
    private GenericRecord rec;
    private final AvroKey<GenericRecord> out = new AvroKey<>();
    @Override
    protected void setup(Context ctx) { schema = new Schema.Parser().parse(SCHEMA); rec = new GenericData.Record(schema); }
    @Override
    protected void map(LongWritable k, Text line, Context ctx) throws IOException, InterruptedException {
      String[] f = line.toString().split(",");
      rec.put("order_id", Long.parseLong(f[0]));
      rec.put("student", f[1]);
      rec.put("time", f[2]);
      rec.put("item", f[3]);
      rec.put("amount", Integer.parseInt(f[4]));
      out.datum(rec);
      ctx.write(out, NullWritable.get());
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    String[] rest = new GenericOptionsParser(conf, args).getRemainingArgs();
    Job job = Job.getInstance(conf, "to avro " + conf.get("avro.codec", "deflate"));
    job.setJarByClass(ToAvro.class);
    job.setMapperClass(M.class);
    job.setNumReduceTasks(0);
    AvroJob.setOutputKeySchema(job, new Schema.Parser().parse(SCHEMA));
    job.setOutputFormatClass(AvroKeyOutputFormat.class);
    String codec = conf.get("avro.codec", "deflate");
    if (!codec.equals("null")) { FileOutputFormat.setCompressOutput(job, true); job.getConfiguration().set("avro.output.codec", codec); }
    FileInputFormat.addInputPath(job, new Path(rest[0]));
    FileOutputFormat.setOutputPath(job, new Path(rest[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

// Inside the namenode container, in /tmp/mr10:
//   hadoop jar mr10.jar ToAvro -libjars $LIBJARS -D avro.codec=deflate /user/hadoop/data/student_orders.csv /out/f_avro_deflate

== size on HDFS (one replica) avro_null 16558165 avro_deflate 4793471 avro_snappy 7791915 avro_deflate 25813 ms HDFS bytes read 4818410 map input records 500000 chai=149669 == hdfs dfs -text on the Avro file {"order_id":1,"student":"S1413","time":"2026-09-14 15:30","item":"dosa","amount":45} {"order_id":2,"student":"S0973","time":"2026-09-26 20:11","item":"samosa","amount":18} == an Avro file carries its schema avro.schema {"type":"record","name":"Order","namespace":"in.myphatshala","fields":[{"name":"order_id","type":"long"},{"name":"student","type":"string"},{"name":"time","type":"string"},{"name":"item"," avro.codec deflate == schema evolution: read the old file with a NEW schema that has an extra field with a default {"order_id":432369,"student":"S0454","time":"2026-09-20 12:05","item":"chai","amount":15,"payment":"cash"} {"order_id":432370,"student":"S1436","time":"2026-09-18 18:50","item":"chai","amount":15,"payment":"cash"} -- and a new field WITHOUT a default: AvroTypeException: Found in.myphatshala.Order, expecting in.myphatshala.Order, missing required field payment

Real output from rowformats_lab.sh. The schema-evolution rows come from part-m-00002.avro, written by the third map task, which is why they start at order 432369.

One job that reads any format

// ItemCountAny.java: orders per item, reading any of our formats.  -D in.format=text|seq|avro|parquet|orc
// For parquet and orc only the "item" column is requested (column projection); -D proj=false reads all columns.
import java.io.IOException;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.mapred.AvroKey;
import org.apache.avro.mapreduce.AvroJob;
import org.apache.avro.mapreduce.AvroKeyInputFormat;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.reduce.LongSumReducer;
import org.apache.hadoop.util.GenericOptionsParser;
import org.apache.orc.OrcConf;
import org.apache.orc.mapred.OrcStruct;
import org.apache.orc.mapreduce.OrcInputFormat;
import org.apache.parquet.example.data.Group;
import org.apache.parquet.hadoop.example.ExampleInputFormat;

public class ItemCountAny {
  static final LongWritable ONE = new LongWritable(1);

  public static class TextM extends Mapper<LongWritable, Text, Text, LongWritable> {
    private final Text item = new Text();
    @Override protected void map(LongWritable k, Text v, Context c) throws IOException, InterruptedException {
      item.set(v.toString().split(",")[3]); c.write(item, ONE);
    }
  }
  public static class SeqM extends Mapper<LongWritable, Text, Text, LongWritable> {   // value = student,time,item,amount
    private final Text item = new Text();
    @Override protected void map(LongWritable k, Text v, Context c) throws IOException, InterruptedException {
      item.set(v.toString().split(",")[2]); c.write(item, ONE);
    }
  }
  public static class AvroM extends Mapper<AvroKey<GenericRecord>, NullWritable, Text, LongWritable> {
    private final Text item = new Text();
    @Override protected void map(AvroKey<GenericRecord> k, NullWritable v, Context c) throws IOException, InterruptedException {
      item.set(k.datum().get("item").toString()); c.write(item, ONE);
    }
  }
  public static class ParquetM extends Mapper<Void, Group, Text, LongWritable> {
    private final Text item = new Text();
    @Override protected void map(Void k, Group g, Context c) throws IOException, InterruptedException {
      item.set(g.getString("item", 0)); c.write(item, ONE);
    }
  }
  public static class OrcM extends Mapper<NullWritable, OrcStruct, Text, LongWritable> {
    @Override protected void map(NullWritable k, OrcStruct row, Context c) throws IOException, InterruptedException {
      c.write((Text) row.getFieldValue("item"), ONE);
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    String[] rest = new GenericOptionsParser(conf, args).getRemainingArgs();
    String fmt = conf.get("in.format", "text");
    Job job = Job.getInstance(conf, "item count from " + fmt);
    job.setJarByClass(ItemCountAny.class);
    switch (fmt) {
      case "seq": job.setInputFormatClass(SequenceFileInputFormat.class); job.setMapperClass(SeqM.class); break;
      case "avro":
        job.setInputFormatClass(AvroKeyInputFormat.class);
        AvroJob.setInputKeySchema(job, new Schema.Parser().parse(ToAvro.SCHEMA));
        job.setMapperClass(AvroM.class); break;
      case "parquet":
        job.setInputFormatClass(ExampleInputFormat.class);
        if (conf.getBoolean("proj", true))
          job.getConfiguration().set("parquet.read.schema", "message order { required binary item (UTF8); }");   // projection
        job.setMapperClass(ParquetM.class); break;
      case "orc":
        job.setInputFormatClass(OrcInputFormat.class);
        if (conf.getBoolean("proj", true))
          OrcConf.INCLUDE_COLUMNS.setString(job.getConfiguration(), "3");                                       // only column 3: item
        job.setMapperClass(OrcM.class); break;
      default: job.setMapperClass(TextM.class);
    }
    job.setCombinerClass(LongSumReducer.class);
    job.setReducerClass(LongSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(LongWritable.class);
    FileInputFormat.addInputPath(job, new Path(rest[0]));
    FileOutputFormat.setOutputPath(job, new Path(rest[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

// In.format=text|seq|avro here; parquet and orc are used in Lesson 10.2.
//   hadoop jar mr10.jar ItemCountAny -libjars $LIBJARS -D in.format=avro /out/f_avro_deflate /out/r_avro

== read each format with the same item-count job text 25221 ms HDFS bytes read 19427239 map input records 500000 chai=149669 seq_block 25073 ms HDFS bytes read 4683705 map input records 500000 chai=149669 avro_deflate 25813 ms HDFS bytes read 4818410 map input records 500000 chai=149669

Real output. Only the InputFormat and the mapper's first line change between formats; the combiner and reducer are Hadoop's built-in LongSumReducer.

The whole lab: jars, environment, conversions, reads

# ---- get_jars.sh (run once in /tmp/mr10) ----
# get_jars.sh: download the Avro-MapReduce, Parquet and ORC jars (Hadoop ships only avro core)
mkdir -p lib && cd lib
M=https://repo1.maven.org/maven2
for a in org/apache/avro/avro-mapred/1.11.5/avro-mapred-1.11.5.jar \
  org/apache/avro/avro-tools/1.11.5/avro-tools-1.11.5.jar \
  org/apache/parquet/parquet-hadoop/1.15.2/parquet-hadoop-1.15.2.jar \
  org/apache/parquet/parquet-column/1.15.2/parquet-column-1.15.2.jar \
  org/apache/parquet/parquet-common/1.15.2/parquet-common-1.15.2.jar \
  org/apache/parquet/parquet-encoding/1.15.2/parquet-encoding-1.15.2.jar \
  org/apache/parquet/parquet-format-structures/1.15.2/parquet-format-structures-1.15.2.jar \
  org/apache/parquet/parquet-jackson/1.15.2/parquet-jackson-1.15.2.jar \
  org/apache/orc/orc-core/1.9.5/orc-core-1.9.5.jar \
  org/apache/orc/orc-mapreduce/1.9.5/orc-mapreduce-1.9.5.jar \
  org/apache/orc/orc-shims/1.9.5/orc-shims-1.9.5.jar \
  org/apache/orc/orc-tools/1.9.5/orc-tools-1.9.5-uber.jar \
  org/apache/hive/hive-storage-api/2.8.1/hive-storage-api-2.8.1.jar \
  io/airlift/aircompressor/0.27/aircompressor-0.27.jar \
  com/google/protobuf/protobuf-java/3.21.12/protobuf-java-3.21.12.jar \
  org/threeten/threeten-extra/1.7.1/threeten-extra-1.7.1.jar \
  commons-pool/commons-pool/1.6/commons-pool-1.6.jar; do
  curl -sfO $M/$a || echo "download failed: $a"
done
ls | wc -l

# ---- formats_env.sh ----
# formats_env.sh: put the Avro-MapReduce, Parquet and ORC jars on the client and task classpaths
L=/tmp/mr10/lib
export LIBJARS=$(ls $L/*.jar | grep -vE "tools|cli" | paste -sd,)
export HADOOP_CLASSPATH=$(echo $LIBJARS | tr , :)
export LIBJARS=$LIBJARS,$(ls /opt/hadoop/share/hadoop/tools/lib/zstd-jni-*.jar)   # Parquet ZSTD needs zstd-jni (in tools/lib, not on the task classpath)
export HADOOP_CLASSPATH=$(echo $LIBJARS | tr , :)

# ---- rowformats_lab.sh ----
# rowformats_lab.sh: the same 5 lakh orders as Text, SequenceFile and Avro: size, what's inside, read speed
cd /tmp/mr10; source formats_env.sh
D=/user/hadoop/data/student_orders.csv
conv() {  # name, class, options
  name=$1; cls=$2; shift 2
  hdfs dfs -rm -r -f -skipTrash /out/f_$name > /dev/null 2>&1
  hadoop jar mr10.jar $cls -libjars $LIBJARS "$@" $D /out/f_$name > /tmp/conv_$name.log 2>&1 || echo "$name FAILED"
}
conv seq_none   ToSequenceFile -D seq.compress=NONE
conv seq_record ToSequenceFile -D seq.compress=RECORD
conv seq_block  ToSequenceFile -D seq.compress=BLOCK
conv avro_null   ToAvro -D avro.codec=null
conv avro_deflate ToAvro -D avro.codec=deflate
conv avro_snappy ToAvro -D avro.codec=snappy
echo "== size on HDFS (one replica)"
printf "%-14s %12s\n" text $(hdfs dfs -du -s $D | awk '{print $1}')
for n in seq_none seq_record seq_block avro_null avro_deflate avro_snappy; do
  printf "%-14s %12s\n" $n $(hdfs dfs -du -s /out/f_$n | awk '{print $1}')
done
echo "== the first bytes of a SequenceFile (a header that names its own types)"
hdfs dfs -cat /out/f_seq_block/part-m-00000 2>/dev/null | head -c 150 | tr -c '[:print:]' '.'; echo
echo "== hdfs dfs -text decodes both binary formats"
hdfs dfs -text /out/f_seq_block/part-m-00000 2>/dev/null | grep -v INFO | head -2
hdfs dfs -text /out/f_avro_deflate/part-m-00000.avro 2>/dev/null | grep -v INFO | head -2
echo "== an Avro file carries its schema"
hdfs dfs -get -f /out/f_avro_deflate/part-m-00002.avro /tmp/orders.avro 2>/dev/null
java -jar lib/avro-tools-1.11.5.jar getmeta /tmp/orders.avro 2>/dev/null | cut -c1-200
echo "== schema evolution: read the old file with a NEW schema that has an extra field with a default"
cat > /tmp/order_v2.avsc <<'J'
{"type":"record","name":"Order","namespace":"in.myphatshala","fields":[
 {"name":"order_id","type":"long"},{"name":"student","type":"string"},{"name":"time","type":"string"},
 {"name":"item","type":"string"},{"name":"amount","type":"int"},
 {"name":"payment","type":"string","default":"cash"}]}
J
java -jar lib/avro-tools-1.11.5.jar tojson --head=2 --reader-schema-file /tmp/order_v2.avsc /tmp/orders.avro 2>/dev/null
echo "-- and a new field WITHOUT a default:"
sed 's/,"default":"cash"//' /tmp/order_v2.avsc > /tmp/order_bad.avsc
java -jar lib/avro-tools-1.11.5.jar tojson --head=2 --reader-schema-file /tmp/order_bad.avsc /tmp/orders.avro 2>&1 | grep -m1 -oE "AvroTypeException.*|Found .*" | cut -c1-170
echo "== read each format with the same item-count job"
read_job() {  # name, input path, format
  hdfs dfs -rm -r -f -skipTrash /out/r_$1 > /dev/null 2>&1
  start=$(date +%s%N)
  hadoop jar mr10.jar ItemCountAny -libjars $LIBJARS -D in.format=$3 $2 /out/r_$1 > /tmp/read_$1.log 2>&1
  ms=$(( ($(date +%s%N) - start) / 1000000 ))
  printf "%-14s %6d ms  HDFS bytes read %10s  map input records %s  chai=%s\n" $1 $ms \
    $(grep "HDFS: Number of bytes read=" /tmp/read_$1.log | head -1 | cut -d= -f2) \
    $(grep "Map input records" /tmp/read_$1.log | cut -d= -f2) \
    $(hdfs dfs -cat /out/r_$1/part-r-00000 | grep chai | cut -f2)
}
read_job text $D text
read_job seq_block /out/f_seq_block seq
read_job avro_deflate /out/f_avro_deflate avro

# Inside the namenode container:  bash /tmp/mr10/rowformats_lab.sh

== size on HDFS (one replica) text 19418699 seq_none 24034722 seq_record 27878984 seq_block 4683378 avro_null 16558165 avro_deflate 4793471 avro_snappy 7791915 == the first bytes of a SequenceFile (a header that names its own types) SEQ.!org.apache.hadoop.io.LongWritable.org.apache.hadoop.io.Text..*org.apache.hadoop.io.compress.DefaultCodec....P...........8`......P...........8`... == hdfs dfs -text decodes both binary formats 1 S1413,2026-09-14 15:30,dosa,45 2 S0973,2026-09-26 20:11,samosa,18 {"order_id":1,"student":"S1413","time":"2026-09-14 15:30","item":"dosa","amount":45} {"order_id":2,"student":"S0973","time":"2026-09-26 20:11","item":"samosa","amount":18} == an Avro file carries its schema avro.schema {"type":"record","name":"Order","namespace":"in.myphatshala","fields":[{"name":"order_id","type":"long"},{"name":"student","type":"string"},{"name":"time","type":"string"},{"name":"item"," avro.codec deflate == schema evolution: read the old file with a NEW schema that has an extra field with a default {"order_id":432369,"student":"S0454","time":"2026-09-20 12:05","item":"chai","amount":15,"payment":"cash"} {"order_id":432370,"student":"S1436","time":"2026-09-18 18:50","item":"chai","amount":15,"payment":"cash"} -- and a new field WITHOUT a default: AvroTypeException: Found in.myphatshala.Order, expecting in.myphatshala.Order, missing required field payment == read each format with the same item-count job text 25221 ms HDFS bytes read 19427239 map input records 500000 chai=149669 seq_block 25073 ms HDFS bytes read 4683705 map input records 500000 chai=149669 avro_deflate 25813 ms HDFS bytes read 4818410 map input records 500000 chai=149669

Real output, complete. Sizes are for one replica (hdfs dfs -du); HDFS stores three. Your timings will differ; the byte counts will not, because the dataset and the writers are deterministic.

8 Line by line

Encoding 300 as a zig-zag varint

Avro writes every int and long this way. Follow how 300 becomes two bytes.

Worked examples: real problems, solved and traced in memory

Three format decisions with numbers: storage cost, finding a record in the middle of a file, and whether a schema change is safe.

Example 1BeginnerSizing · storage

What does the format save over three years?

Problem statement

The college's orders grow to 50 lakh a month and are kept for 3 years with replication 3. Using our lab's sizes for 5 lakh orders (CSV 19,418,699 bytes, Avro deflate 4,793,471 bytes), how much raw HDFS disk does each format need?

Input

50 lakh/month · 36 months · ×3 replication

Expected output

CSV           38.8 B/order ->   19.5 GB raw disk
Avro deflate   9.6 B/order ->    4.8 GB raw disk

Approach

Bytes per order from the lab.
Orders in 3 years × bytes × 3 replicas.
Compare.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 2IntermediateInternals · sync markers

Where does a map task start reading?

Problem statement

A file of records has a 4-byte sync marker #SY# written before every few records. A map task is given the split starting at byte 14. Find where it really starts reading, and why it can't simply start at byte 14.

Input

one byte string, split start 14

Expected output

byte 14 is inside: b'r2:dosa|'
first sync marker after 14 is at 28 -> start reading at 32 : b'r4:idli'

Approach

Byte 14 is in the middle of a record.
Scan forward for the next sync marker.
Start right after it; the previous task finishes the record it is in.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 3AdvancedDesign · evolution

Which schema changes are safe?

Problem statement

Version 1 of the order schema is in use. The team proposes four changes. For each, can new readers read old files (backward compatible)?

Input

4 proposed changes

Expected output

add payment, default 'cash'  backward compatible: True
add payment, no default      backward compatible: False
remove time                  backward compatible: True
amount int -> string         backward compatible: False

Approach

Adding a field: old files lack it, so the reader needs a default.
Removing a field: the reader just skips it in old files.
Changing a type: only promotions like int → long are allowed.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

9 Quiz and revision

Check yourself

Quick revision

Lesson 10.1 in one screen

Text/CSV: readable, no schema, splittable only uncompressed (or bzip2). Ours: 19.4 MB.
SequenceFile: binary key-value Writables, class names + codec in the header, sync markers (~every 100 KB). NONE 24.0 MB, RECORD 27.9 MB, BLOCK 4.7 MB.
Avro: JSON schema in the header, compact binary (zig-zag varints), sync markers between blocks, many languages. null 16.6 MB, snappy 7.8 MB, deflate 4.8 MB.
Schema evolution: fields matched by name; new fields need a default; renames use aliases; int → long promotion allowed.
Without a default: “AvroTypeException: … missing required field payment”.
hdfs dfs -text decodes SequenceFile and Avro; avro-tools getmeta / tojson inspect Avro.
Hadoop ships avro core, not avro-mapred: add it with -libjars and HADOOP_CLASSPATH.

Interview and viva questions

What is a SequenceFile in Hadoop?

A flat binary file of key-value pairs, where the keys and values are Writables. Its header names the key and value classes and the compression codec. Records can be stored uncompressed, record-compressed or block-compressed, and sync markers written every 100 KB or so let readers start in the middle, so the file stays splittable even when compressed. It is commonly used for intermediate MapReduce data and for packing many small files into one.

What is Apache Avro and why is it used with Hadoop?

Avro is a language-neutral data serialization system. An Avro data file stores its schema as JSON in the header, followed by compact binary records in blocks separated by sync markers, optionally compressed with deflate, snappy, zstandard or others. It is splittable, readable from many languages, and supports schema evolution, which is why it is popular for data ingestion, Kafka messages and the raw zone of data lakes.

What is schema evolution in Avro?

Reading data written with one schema (the writer's schema, stored in the file) using a different, compatible schema (the reader's schema). Avro matches fields by name: fields only in the reader's schema are filled from their default values, fields only in the writer's schema are ignored, aliases handle renames and some types can be promoted (int to long, float to double). Adding a field without a default breaks backward compatibility.

What is the difference between SequenceFile and Avro?

Both are binary, row-oriented, compressible and splittable. A SequenceFile stores Java Writable key-value pairs and records only the class names, so it is mainly used inside Hadoop. Avro stores a full language-neutral schema with named, typed fields, can be read from many languages and supports schema evolution, so it suits data exchanged between systems and teams.

Which compression type should I use for SequenceFiles?

BLOCK compression, which compresses many records together. RECORD compression compresses each value separately, which barely helps (or even hurts) for small values: in our lab a 5 lakh order SequenceFile was 24.0 MB uncompressed, 27.9 MB with RECORD and 4.7 MB with BLOCK compression.

Practice problems

Format exercises using the byte counts from our lab.

P1Easy

Bytes per order

Compute the average bytes per order for each format from the lab sizes.

Input

7 sizes for 500,000 orders

Expected output

text           38.8 B/order
seq_none       48.1 B/order
seq_record     55.8 B/order
seq_block       9.4 B/order
avro_null      33.1 B/order
avro_deflate    9.6 B/order
avro_snappy    15.6 B/order

Show solution and explanation

sizes = {"text": 19418699, "seq_none": 24034722, "seq_record": 27878984, "seq_block": 4683378,
         "avro_null": 16558165, "avro_deflate": 4793471, "avro_snappy": 7791915}
for k, v in sizes.items():
    print(f"{k:13} {v / 500_000:5.1f} B/order")

text 38.8 B/order seq_none 48.1 B/order seq_record 55.8 B/order seq_block 9.4 B/order avro_null 33.1 B/order avro_deflate 9.6 B/order avro_snappy 15.6 B/order

How it works: From 55.8 bytes (RECORD) down to 9.4 (BLOCK): a 6× spread for the same data.

P2Easy

SequenceFile overhead

How many extra bytes per order did the uncompressed SequenceFile add over the CSV?

Input

24,034,722 vs 19,418,699

Expected output

9.232046 bytes per order

Show solution and explanation

print((24034722 - 19418699) / 500_000, "bytes per order")

9.232046 bytes per order

How it works: About 9.2: record length 4 + key length 4 + 8-byte key + value length 1, minus the order_id text, comma and newline.

P3Easy

Compression ratio

What percentage of the CSV size is each compressed format?

Input

seq_block, avro_deflate, avro_snappy

Expected output

seq_block      24.1% of the CSV
avro_deflate   24.7% of the CSV
avro_snappy    40.1% of the CSV

Show solution and explanation

csv = 19418699
for name, s in [("seq_block", 4683378), ("avro_deflate", 4793471), ("avro_snappy", 7791915)]:
    print(f"{name:13} {100 * s / csv:5.1f}% of the CSV")

seq_block 24.1% of the CSV avro_deflate 24.7% of the CSV avro_snappy 40.1% of the CSV

How it works: About 24%, 25% and 40%: deflate compresses more, snappy is faster (Lesson 10.3).

P4Medium

Decode a varint

Decode the Avro zig-zag varint bytes d8 04 back to a number.

Input

bytes 0xd8 0x04

Expected output

zigzag value 600 -> number 300

Show solution and explanation

data = bytes([0xD8, 0x04])
n, shift = 0, 0
for b in data:
    n |= (b & 0x7F) << shift
    shift += 7
    if b < 0x80:
        break
print("zigzag value", n, "-> number", (n >> 1) ^ -(n & 1))

zigzag value 600 -> number 300

How it works: 600 after the 7-bit groups are joined, then un-zig-zag: 300.

P5Medium

Schema resolver

Write a resolver that also skips fields present in the file but not in the reader's schema.

Input

writer has an extra "counter" field

Expected output

{'order_id': 7, 'item': 'chai', 'amount': 15, 'payment': 'cash'}

Show solution and explanation

row = {"order_id": 7, "item": "chai", "amount": 15, "counter": "A"}
reader = [("order_id", None), ("item", None), ("amount", None), ("payment", "cash")]
out = {n: row.get(n, d) for n, d in reader if n in row or d is not None}
print(out)

{'order_id': 7, 'item': 'chai', 'amount': 15, 'payment': 'cash'}

How it works: "counter" is ignored, "payment" filled from its default: both directions of evolution in one line.

P6Medium

Small values don't compress

Compress one 30-byte value with zlib. Is the result smaller?

Input

"S1413,2026-09-14 15:30,dosa,45"

Expected output

30 -> 38

Show solution and explanation

import zlib
v = b"S1413,2026-09-14 15:30,dosa,45"
print(len(v), "->", len(zlib.compress(v)))

30 -> 38

How it works: 30 → 38 bytes: compression needs repetition, and one short value has none. That is RECORD compression's problem.

P7Business problem

Which format for the landing zone?

Partners send 20 crore events a day; three teams (Java, Python, Spark SQL) read them, and fields get added every quarter. Compare yearly raw storage (×3 replicas) as CSV vs Avro deflate using our bytes per order, and pick a format.

Input

38.8 vs 9.6 B/record

Expected output

CSV            7.7 TB per year
Avro deflate   1.9 TB per year

Show solution and explanation

per_year = 200_000_000 * 365
for name, b in [("CSV", 38.8), ("Avro deflate", 9.6)]:
    print(f"{name:12} {per_year * b * 3 / 1024**4:5.1f} TB per year")

CSV 7.7 TB per year Avro deflate 1.9 TB per year

How it works: About 7.7 TB vs 1.9 TB a year, and Avro's schema with defaults lets the quarterly new fields arrive without breaking the Python and Java readers.

P8Advanced

A tiny container file

Write records into blocks of 3 with a sync marker before each block, then read only the block that starts after byte 20.

Input

7 records

Expected output

b'\xff\xfer1,r2,r3;\xff\xfer4,r5,r6;\xff\xfer7;'
split from byte 20 starts at 22 -> b'r7'

Show solution and explanation

SYNC = b"\xff\xfe"
recs = [f"r{i}".encode() for i in range(1, 8)]
f = b""
for i in range(0, len(recs), 3):
    f += SYNC + b",".join(recs[i:i + 3]) + b";"
start = f.find(SYNC, 20)
print(f)
print("split from byte 20 starts at", start, "->", f[start + 2:].split(b";")[0])

b'\xff\xfer1,r2,r3;\xff\xfer4,r5,r6;\xff\xfer7;' split from byte 20 starts at 22 -> b'r7'

How it works: The reader skips the partial block and starts at the next marker; the reader of the previous split finishes that block. This is how Avro and SequenceFile splits work.

Real-world scenario questions (asked in company interviews)

File format questions are common for data engineers: expect comparisons, schema evolution and “which format would you choose”.

Formats · basicsFresher

“Compare Text, SequenceFile and Avro”

A standard question in Hadoop interviews.

The interviewer is checking: schema, size, splittability, languages.

Show a strong answer

Text: readable, no schema, large; splittable only uncompressed or bzip2.
SequenceFile: binary Writable key-value pairs, class names in the header, splittable with sync markers, BLOCK compression; Java-centric.
Avro: JSON schema in the file, compact binary, splittable, any language, schema evolution. In my lab: CSV 19.4 MB, SequenceFile BLOCK 4.7 MB, Avro deflate 4.8 MB.

Avro · evolutionExperienced

A producer added a field and consumers started failing

After a release, Spark jobs reading old Avro files fail with “missing required field”.

The interviewer is checking: reader vs writer schema and defaults.

Show a strong answer

The new reader schema has a field that old files lack and it has no default, so resolution fails.
Fix: add a default (or make it a union with null, default null) and redeploy; old and new files then both read.
Prevent it: a schema registry with BACKWARD compatibility checks in CI before producers can publish.

Performance · storageExperienced

“We compressed our SequenceFiles and they got bigger”

A team switched on compression for an intermediate dataset of short records.

The interviewer is checking: RECORD vs BLOCK compression.

Show a strong answer

They probably used RECORD compression: each small value is compressed alone, with no repetition to exploit and per-value overhead.
Switch to BLOCK compression (mapreduce.output.fileoutputformat.compress.type=BLOCK), which compresses many records together.
Our measurement: 24.0 MB uncompressed, 27.9 MB RECORD, 4.7 MB BLOCK.

Design · choiceExperienced

Choose formats for a new data lake

Events arrive from Kafka; analysts run SQL over a few columns of billions of rows.

The interviewer is checking: row vs columnar by workload.

Show a strong answer

Raw/landing zone: Avro, which is fast to append, keeps whole records and schema history, and matches the Kafka schemas.
Curated tables: Parquet or ORC, since columnar storage reads only the needed columns and compresses better.
A scheduled job converts and compacts raw Avro into partitioned columnar tables.

Text, SequenceFile and Avro

The day someone added a column

Row-oriented file formats in Hadoop

Words you'll meet in this lesson

From our lab (Hadoop 3.5.0, 5 lakh orders, one replica)

File formats at three levels

Beginner Three ways to keep a register

Intermediate What the bytes look like

Advanced Splitting, compression and evolution

Text vs SequenceFile vs Avro

The InputFormat and OutputFormat for each

Text

SequenceFile

Avro

Choosing

Converting the orders and reading them back

Write SequenceFiles three ways

Look at the first bytes

Write Avro three ways

Read the binary files as text

Add Monday's column the safe way

Read all three with the same job

One order, three encodings

Row formats at work

1. Event streams with a schema registry

2. Packing small files

3. Data between chained jobs

4. Raw zone, then curated tables

Encode one order three ways

Run it on your PC: one dataset, seven encodings

CSV to SequenceFile

CSV to Avro, then a new schema

One job that reads any format

The whole lab: jars, environment, conversions, reads

Encoding 300 as a zig-zag varint

Worked examples: real problems, solved and traced in memory

What does the format save over three years?

Where does a map task start reading?

Which schema changes are safe?

Check yourself

Quick revision

Lesson 10.1 in one screen

Interview and viva questions

Practice problems

Bytes per order

SequenceFile overhead

Compression ratio

Decode a varint

Schema resolver

Small values don't compress

Which format for the landing zone?

A tiny container file

Real-world scenario questions (asked in company interviews)

“Compare Text, SequenceFile and Avro”

A producer added a field and consumers started failing

“We compressed our SequenceFiles and they got bigger”

Choose formats for a new data lake