Who's who: Hadoop glossary

Not either/or. Hadoop is storage (HDFS) + resources (YARN) + an engine (MapReduce). Spark is an engine that usually runs on YARN and reads HDFS. In our lab five cached Spark passes took 35 s; five MapReduce jobs took 199 s.

A data lake is files in cheap storage. A lakehouse adds a table format (Iceberg, Delta, Hudi) on top, giving transactions, updates, snapshots and time travel.

Authentication proves who you are (Kerberos tickets). Authorization decides what you may do (HDFS permissions, ACLs, Ranger policies). Without authentication, authorization can be bypassed by claiming another name, as our unsecured lab showed.

A backup is a copy you can restore from, maybe slowly. DR is a ready second site with recent data and a rehearsed plan to switch over within the RTO.

capacity is what a queue is guaranteed; maximum-capacity is how far it may grow when others are idle; user-limit-factor caps one user inside the queue. In our lab maximum-capacity 100% did nothing until user-limit-factor was raised.

The container (mapreduce.map.memory.mb) is what YARN reserves on the node; the heap (-Xmx) is what Java may use for objects, about 80% of it. Too small a heap gives "Java heap space"; heap + overhead above the container gets the container killed.

Row formats (CSV, SequenceFile, Avro) keep each record together: good for writing and reading whole records. Columnar (Parquet, ORC) keep each column together: good for analytics that read a few columns of many rows.

A .gz file is one stream: not splittable, one map. A container (SequenceFile BLOCK, Avro, Parquet, ORC) compresses inside blocks, so the file stays splittable whatever the codec.

A reducer runs once per key on all values, after the shuffle. A combiner runs on one map task's output, zero or more times, before the shuffle; it only works for operations like sum, count, max and min.

Reduce-side: both inputs are shuffled; works for any sizes (24.4 MB shuffled, 24.8 s in our lab). Map-side: the small side is copied to every map task; no shuffle (18.9 s), but the small side must fit in memory.

The partitioner decides which reducer gets a key. The grouping comparator decides which keys, already on that reducer, share one reduce() call.

Failed: the attempt crashed (exception, exit code 137) and counts toward the retry limit. Killed: the framework stopped it on purpose, usually the losing copy in speculative execution. Our job showed both and still succeeded.

Map output is temporary, sorted, on the worker's local disk, served to reducers and deleted after the job. Job output (part-r files) is written to HDFS with replication.

Trash catches a normal -rm for a while, but -skipTrash or expiry bypasses it. A snapshot keeps the old state even after a -skipTrash delete, until you delete the snapshot.

HA gives one namespace two NameNodes (active + standby) for availability. Federation splits the namespace across several active NameNodes for scale. They can be combined.

Replication keeps 3 full copies (200% overhead), fast to read and repair. Erasure coding keeps data + parity (e.g. 50% overhead), cheaper for cold data but costlier CPU and network on repair.

A block is how HDFS physically stores bytes (128 MB pieces). An input split is the logical piece one map task reads; it normally matches a block, but a record that crosses a block boundary is read whole by one split.

Both are masters, of different things. The NameNode manages storage (files and blocks). The ResourceManager manages compute (memory and CPU for jobs).

Both run on a worker. The DataNode stores blocks; the NodeManager runs task containers. Running both on one machine is what makes data locality possible.

The Secondary only merges the edit log into the fsimage. It cannot replace a failed NameNode. The Standby (HA) is a hot spare that takes over automatically.

The ResourceManager serves the whole cluster and hands out containers. An ApplicationMaster serves one job and decides which of its tasks run in those containers.

Replication keeps live copies to survive hardware failure; deletes and bad writes reach every copy. A backup or snapshot is a separate point-in-time copy you can restore.

Node-local: the task runs on a machine holding its block. Rack-local: on another machine in the same rack, so data crosses only the rack switch.

A dead node has been silent past the timeout (≈10.5 min) and its replicas are re-created elsewhere. A stale node has missed heartbeats for a shorter time (30 s by default) and can be avoided for reads and writes without being declared dead.

Standalone runs a job inside one Java process on local files; no daemons. Pseudo-distributed runs every daemon (NameNode, DataNode, ResourceManager…) as a separate process on one machine, with real HDFS and YARN.

core-default.xml holds Hadoop's built-in defaults inside the jars; never edit it. core-site.xml in etc/hadoop is where you override the values you need.

Decommissioning copies every block away first, then marks the node Decommissioned: no moment with fewer replicas. Stopping it is treated as a failure: HDFS waits until the node is declared dead (≈10.5 min) before re-replicating.

-mv only renames in the NameNode, so it is instant for any size. -cp reads and rewrites every byte through the client (5.6 s vs 1.9 s for 403 MB in our lab).

stop pauses the containers and keeps their disks, so HDFS data survives. down deletes the containers, and without volumes the data goes with them.