The Hadoop ecosystem: Hive, HBase, ZooKeeper and more

Twenty names in one meeting

The Hadoop ecosystem

Words you'll meet in this lesson

From our lab (Spark 4.0.4 on our YARN cluster)

Architecture at a glance

The ecosystem in layers, bottom to top. Everything runs on HDFS (or cloud object storage) and most of it runs inside YARN containers. Grey boxes are retired or dormant projects you will still meet in older clusters and job descriptions.

The ecosystem at three levels

Who does what, and who is still alive

Replacements you will see in job descriptions

Animated example: one day of data at a food-delivery company

SQL on Hadoop, on our own cluster

We added Spark 4.0.4 (downloaded from dlcdn.apache.org) to the Module 8 lab's namenode container and ran spark-sql on YARN with three 512 MB executors. The SQL file is in “Run it on your PC”.

1. Define a table over an existing HDFS file

   CREATE TABLE canteen.orders (order_id BIGINT, student STRING, ts STRING, item STRING, amount INT)
     USING csv OPTIONS (path 'hdfs://namenode/user/hadoop/data/student_orders.csv');

   Time taken: 0.753 seconds
   Nothing was copied. The metastore stored the columns and the file location: under a second.

2. Look at what the metastore knows

   DESCRIBE TABLE EXTENDED canteen.orders;

   Type EXTERNAL Provider csv Location hdfs://namenode/user/hadoop/data/student_orders.csv Serde Library org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
   EXTERNAL means dropping the table leaves the file alone. The Hive class names show where the metastore idea came from.

3. Ask a business question

   SELECT item, count(*) AS orders, sum(amount) AS revenue
   FROM canteen.orders GROUP BY item ORDER BY revenue DESC;

   thali 30292 2423360 dosa 49927 2246715 chai 149669 2245035 vada pav 69754 1395080 … coffee 40364 807280 Time taken: 11.084 seconds, Fetched 8 row(s)
   The same numbers our Java jobs produced in Module 9, written as one SQL statement. Thali earns most with the fewest orders.

4. A second question, no new code

   SELECT substr(ts, 12, 2) AS hour, count(*) AS orders
   FROM canteen.orders GROUP BY 1 ORDER BY orders DESC LIMIT 3;

   14 62213 13 56026 08 55852 Time taken: 2.838 seconds, Fetched 3 row(s)
   Lunch (13–14 h) and breakfast (08 h) are the rush hours. With Java MapReduce this would be a new Mapper, Reducer and jar.

5. See MapReduce inside the plan

   EXPLAIN SELECT item, count(*) FROM canteen.orders GROUP BY item;

   +- HashAggregate(keys=[item#14], functions=[count(1)]) +- Exchange hashpartitioning(item#14, 200) +- HashAggregate(keys=[item#14], functions=[partial_count(1)]) +- FileScan csv spark_catalog.canteen.orders[item#14] … ReadSchema: struct<item:string>
   Read from the bottom: scan only the item column, count per item inside each task (a combiner, Lesson 9.1), shuffle by hash of item (Lesson 9.2), add the partial counts (the reduce).

One SQL query through the stack

Six steps of our real spark-sql run: what the terminal showed, and which ecosystem layer did the work.

The ecosystem at work

Simplified; organisations differ in tools and policies.

Sources (checked 22 Oct 2026): Apache Attic: retired projects · Apache Flume README (dormant notice) · Apache Hive downloads · endoflife.date: Apache Spark · Amazon EMR 7.x application versions · Cloudera Base 7.3.2 summary

Pick the right tool, then run a GROUP BY by hand

Part one is a tiny rule engine: describe a requirement and it suggests ecosystem projects. Part two does what our SQL engine did for GROUP BY item: partial counts per task (map + combine), a shuffle by key, and a final sum (reduce).

RULES = [  # (keyword in the requirement, suggestion)
    ("sql", "Hive / Spark SQL / Trino over metastore tables"),
    ("single row", "HBase (get by row key)"),
    ("events", "Kafka, then Spark or Flink"),
    ("update", "Iceberg / Delta / Hudi table format"),
    ("schedule", "Airflow (Oozie is retired)"),
    ("leader", "ZooKeeper"),
    ("access", "Ranger policies + Kerberos"),
]
def pick(req):
    return [tool for key, tool in RULES if key in req.lower()] or ["plain HDFS files + Spark"]

for r in ["Analysts need SQL on 3 years of bills",
          "App must fetch a single row per customer in 10 ms",
          "Clickstream events every second, update totals"]:
    print(f"{r[:44]:44} -> {pick(r)}")

# GROUP BY item, the way the plan executed it
tasks = [["chai", "dosa", "chai"], ["thali", "chai"], ["dosa", "chai", "thali"]]
partial = []
for t in tasks:                               # HashAggregate partial_count, per task
    counts = {}
    for item in t:
        counts[item] = counts.get(item, 0) + 1
    partial.append(counts)
print("partial counts:", partial)
final = {}
for counts in partial:                        # Exchange (shuffle by item) + final count
    for item, n in counts.items():
        final[item] = final.get(item, 0) + n
print("final        :", dict(sorted(final.items())))

• return [tool for key, tool in RULES if key in req.lower()] or ["plain HDFS files + Spark"]A requirement can match several rules: real platforms combine tools.
• for t in tasks: # HashAggregate partial_count, per taskThis loop is the combiner (Lesson 9.1): each task counts its own rows first.
• for counts in partial: # Exchange (shuffle by item) + final countOnly small partial counts cross the network, then the reducer adds them.

Run it on your PC: SQL on Hadoop with Spark

Uses the Module 8 lab. Download Spark 4.0.4 (spark-4.0.4-bin-hadoop3.tgz from dlcdn.apache.org), copy it into the namenode container and unpack it. Spark needs to know where Hadoop's configuration is so it can find HDFS and YARN. Three executors of 512 MB fit our small cluster.

-- sql_on_hadoop.sql: SQL over files in HDFS, with a Hive-style metastore (Spark SQL, run on YARN)
CREATE DATABASE IF NOT EXISTS canteen;
DROP TABLE IF EXISTS canteen.orders;
CREATE TABLE canteen.orders (order_id BIGINT, student STRING, ts STRING, item STRING, amount INT)
  USING csv OPTIONS (path 'hdfs://namenode/user/hadoop/data/student_orders.csv');
DESCRIBE TABLE EXTENDED canteen.orders;
SELECT item, count(*) AS orders, sum(amount) AS revenue FROM canteen.orders GROUP BY item ORDER BY revenue DESC;
SELECT substr(ts, 12, 2) AS hour, count(*) AS orders FROM canteen.orders GROUP BY 1 ORDER BY orders DESC LIMIT 3;
EXPLAIN SELECT item, count(*) FROM canteen.orders GROUP BY item;

-- In the namenode container (after unpacking Spark into /tmp/opt):
--   export SPARK_HOME=/tmp/opt/spark-4.0.4-bin-hadoop3
--   export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
--   $SPARK_HOME/bin/spark-sql --master yarn --num-executors 3 --executor-memory 512m \
--       --executor-cores 1 --driver-memory 512m -f sql_on_hadoop.sql

How a metastore answers “where is this table?”

An engine never scans HDFS looking for a table. It asks the metastore for the location and format, then plans the read.

Worked examples: real problems, solved and traced in memory

Four “which tool?” decisions from different industries, each reasoned from the requirement.

CSV in HDFS · SQL users

Define an external table over 900 GB; copy needed: False
Engine: Hive or Spark SQL for batch, Trino for quick questions

40 crore rows · 1 row per request · 10 ms

full scan on one disk: 8 minutes; HBase get by key: a few milliseconds

120 workflows · 1.5 days each · 2 engineers

90 working days, about 4.3 months

2 TB · 5,000 files · rows of 1 customer spread over ~40 files

rewrite everything: 2048 GB; table format rewrites ~16.4 GB in 40 files

Check yourself

Quick revision

Interview and viva questions

Practice problems

Ecosystem exercises: place projects, read our real results, and choose tools.

thali 30292 2423360 · dosa 49927 2246715 · chai 149669 2245035

thali 80.0
dosa 45.0
chai 15.0

for item, n, rev in [("thali", 30292, 2423360), ("dosa", 49927, 2246715), ("chai", 149669, 2245035)]:
    print(item, rev / n)

13 → 56,026 · 14 → 62,213

23.6%

print(f"{(56026 + 62213) / 500000:.1%}")

status table

4

gone = {"Sqoop", "Oozie", "Flume", "Sentry"}
print(len(gone & {"Sqoop", "Hive", "Oozie", "Flume", "HBase", "Sentry"}))

8 projects

HDFS       storage
Iceberg    tables
YARN       resources
Spark      engine
Trino      SQL
Kafka      ingestion
ZooKeeper  coordination
Ranger     security

layer = {"HDFS": "storage", "Iceberg": "tables", "YARN": "resources", "Spark": "engine",
         "Trino": "SQL", "Kafka": "ingestion", "ZooKeeper": "coordination", "Ranger": "security"}
for p, l in layer.items():
    print(f"{p:10} {l}")

500,000 rows · 8 keys · 3 tasks

without: 500000 with partial aggregation: 24

rows, keys, tasks = 500_000, 8, 3
print("without:", rows, "with partial aggregation:", keys * tasks)

3 workloads

nightly 2 TB batch       -> Spark (or Hive on Tez)
ad-hoc SQL in seconds    -> Trino / Impala
per-second event stream  -> Flink or Spark Structured Streaming via Kafka

m = {"nightly 2 TB batch": "Spark (or Hive on Tez)", "ad-hoc SQL in seconds": "Trino / Impala",
     "per-second event stream": "Flink or Spark Structured Streaming via Kafka"}
for k, v in m.items():
    print(f"{k:24} -> {v}")

₹50 lakh × 5% vs ₹12 lakh

4.8 years

risk_per_year = 50 * 0.05
print(f"{12 / risk_per_year:.1f} years")

2 tables

tables: {} files left: {'/data/orders.csv'}

files = {"/data/orders.csv", "/warehouse/menu"}
ms = {"orders": ("EXTERNAL", "/data/orders.csv"), "menu": ("MANAGED", "/warehouse/menu")}
def drop(t):
    kind, loc = ms.pop(t)
    if kind == "MANAGED":
        files.discard(loc)
drop("orders"); drop("menu")
print("tables:", ms, "files left:", files)

Real-world scenario questions (asked in company interviews)

Ecosystem questions test whether you know what each tool is for, and what is current.