The big data ecosystem map: every tool in its place

Kabir's first day: 40 tool names and one diagram

What is the big data ecosystem?

Words you'll meet in this lesson

The six layers at a glance

The same map at three levels

Three ways to build the same platform

One layer, three names: open source vs the big clouds

Cloud product names change often; the layers don't. Learn the layer first, then the local name.

Which tool for which question?

Follow one TiffinGo order through all six layers

1. Source: the app creates an event

   {"order_id": 90017, "user": "kabir", "kitchen": "K12", "city": "Pune",
    "item": "veg thali", "amount": 120, "ts": "2026-10-19T12:31:08"}

   The moment Kabir taps Place order, the app sends this event. At lunch rush thousands of phones send events like this every minute.

2. Ingest: Kafka receives it

   kafka-console-producer --topic orders --bootstrap-server broker1:9092

   The event is appended to the orders topic. Kafka doesn't process it; it keeps events in order, durably, and lets many systems read them: the live dashboard, the fraud checker and the lake loader.

3. Store: raw copy in the lake

   s3://tiffingo-lake/raw/orders/date=2026-10-19/hour=12/part-0003.json

   A loader writes Kafka events into the lake every few minutes, in folders by date and hour. Raw data is never changed, so it can always be reprocessed if a bug is found later.

4. Process: Spark cleans and joins

   orders = spark.read.json("s3://tiffingo-lake/raw/orders/date=2026-10-19/")
   kitchens = spark.read.parquet("s3://tiffingo-lake/clean/kitchens/")
   daily = (orders.dropDuplicates(["order_id"])
                  .join(kitchens, "kitchen")
                  .groupBy("city", "kitchen_name").sum("amount"))
   daily.write.mode("overwrite").parquet("s3://tiffingo-lake/gold/daily_sales/date=2026-10-19/")

   This is real PySpark (you'll write it in the Spark course). Spark splits the day's files across many executors, removes duplicate events, joins kitchen names and totals sales, exactly the partition → process → combine pattern from Lesson 1.3.

5. Query: SQL on the result

   SELECT city, SUM(amount) AS revenue
   FROM gold.daily_sales
   WHERE date = '2026-10-19'
   GROUP BY city ORDER BY revenue DESC;

   Trino, Hive or a warehouse runs this SQL directly over the Parquet files or loaded tables. Analysts never need to know where the files live.

6. Visualise and orchestrate

   The Power BI dashboard refreshes, and the city manager sees yesterday's sales at 6 am.

   Behind all of it, an Airflow DAG ran at 2 am: check raw data arrived → run Spark job → check row counts → load warehouse → refresh dashboard. If Spark fails, Airflow retries it and alerts the on-call engineer.

One order: what Kabir sees, and what the platform does

Step through the six frames. Left: Kabir's phone and his manager's dashboard. Right: the layer that is working at that moment.

The ecosystem in real time: three live scenarios

The same layers behave very differently at lunch rush, at 2 am and when a company moves to the cloud. Watch all three.

Lunch rush: the streaming path saves a kitchen

12:00 to 13:00 on a weekday. Orders flow through Kafka into a stream processor (Flink) that counts orders per kitchen every minute. Watch what happens when one kitchen is overloaded.

2 am: the orchestrator handles a failure while everyone sleeps

The nightly daily_sales DAG starts at 02:00. Halfway through, a cloud machine running the Spark job disappears. Watch Airflow deal with it.

Moving to the cloud: same layers, new names

TiffinGo's investors want it on the cloud. The team maps every layer of their open-source stack to a managed service. Watch the map change, and notice what stays the same.

Where you'll see the ecosystem at work

Simplified to show the layers. Real platforms vary by company.

Build a tiny data platform in Python

Each function below plays the role of one layer. ingest accepts every event like Kafka, store keeps raw copies like a data lake, process cleans like Spark, serve answers a question like a SQL engine, and the last three lines are the orchestrator. Notice how the corrupted event and the wrong data type are handled in process, never by changing the raw copy.

import json

raw_events = [
    '{"order": 1, "city": "Pune", "item": "thali", "amount": 120}',
    '{"order": 2, "city": "Mumbai", "item": "biryani", "amount": 250}',
    '{"order": 3, "city": "Pune", "item": "thali", "amount": 120}',
    '{"order": 4, "city": "Pune", "item": "dosa", "amount": "90"}',
    'CORRUPTED###',
]

def ingest(events):                 # like Kafka: accept everything, in order
    print("INGEST :", len(events), "events received")
    return list(events)

def store(events, lake):            # like a data lake: keep the raw copy
    lake["raw/orders"] = events
    print("STORE  :", len(events), "raw events saved")

def process(lake):                  # like Spark: parse, clean, fix types
    clean = []
    for e in lake["raw/orders"]:
        try:
            r = json.loads(e)
        except json.JSONDecodeError:
            continue                # skip broken events (real systems log them)
        r["amount"] = int(r["amount"])
        clean.append(r)
    lake["clean/orders"] = clean
    print("PROCESS:", len(clean), "clean rows")

def serve(lake):                    # like Hive / a warehouse: answer a question
    by_city = {}
    for r in lake["clean/orders"]:
        by_city[r["city"]] = by_city.get(r["city"], 0) + r["amount"]
    print("SERVE  : revenue by city ->", by_city)

# ORCHESTRATE: like Airflow, run each layer in the right order
lake = {}
store(ingest(raw_events), lake)
process(lake)
serve(lake)

• lake["raw/orders"] = eventsThe raw copy is saved before cleaning. If the cleaning code has a bug, you fix it and reprocess from raw. That is why lakes keep raw data.
• except json.JSONDecodeError:One broken event must not stop the platform. The processing layer skips it; real pipelines also write it to an error table for checking.
• r["amount"] = int(r["amount"])Order 4 sent the amount as text "90". Fixing types is a typical processing job.
• store(ingest(raw_events), lake)The last three lines are the orchestrator: they decide the order. Swap process and serve and see what breaks.

One pipeline, one line at a time

A smaller pipeline: three raw events, a process layer and a serve layer. Watch the lake dictionary gain a "clean" copy while "raw" stays untouched, and watch each layer get its own frame on the stack.

Worked examples: real problems, solved and traced in memory

Four problems, from first-week intern to platform architect. Each runs live and is traced line by line.

stack: Spark, Kafka, Power BI, S3, Airflow, Hive

Ingest       ['Kafka']
Store        ['S3']
Process      ['Spark']
Query        ['Hive']
Visualise    ['Power BI']
Orchestrate  ['Airflow']

two lists of (stage, seconds)

Streaming path: 1.7 seconds
Batch path    : 7.2 hours

needs: clean←ingest, features←clean, report←clean

['ingest', 'clean', 'features', 'report']

stack per layer + chosen cloud

Ingest      Kafka    -> Pub/Sub
Store       HDFS     -> Cloud Storage
Process     Spark    -> Dataproc
Query       Hive     -> BigQuery
Orchestrate Airflow  -> Cloud Composer

Check yourself

Quick revision

Interview and viva questions

Practice problems

From reading a platform diagram to reasoning like an architect. Problems are marked Easy, Medium, Advanced and Business.

Kafka:Ingest, NiFi:Ingest, S3:Store, Spark:Process, Flink:Process, Hive:Query

{'Ingest': 2, 'Store': 1, 'Process': 2, 'Query': 1}

tools = [("Kafka", "Ingest"), ("NiFi", "Ingest"), ("S3", "Store"),
         ("Spark", "Process"), ("Flink", "Process"), ("Hive", "Query")]
count = {}
for name, layer in tools:
    count[layer] = count.get(layer, 0) + 1
print(count)

[2, 45, 300, 30]

6.3 minutes

print(round(sum([2, 45, 300, 30]) / 60, 1), "minutes")

have: Ingest, Store, Visualise

['Orchestrate', 'Process', 'Query']

all_layers = {"Ingest", "Store", "Process", "Query", "Visualise", "Orchestrate"}
have = {"Ingest", "Store", "Visualise"}
print(sorted(all_layers - have))

ingest 5, store 20, spark 900, load 60, refresh 15 (seconds)

spark takes 90 % of the time

stages = {"ingest": 5, "store": 20, "spark": 900, "load": 60, "refresh": 15}
slowest = max(stages, key=stages.get)
share = stages[slowest] / sum(stages.values()) * 100
print(slowest, "takes", round(share), "% of the time")

500 GB/day, ratio 1/5

Raw JSON    : 178.2 TB per year
Parquet     : 35.6 TB per year

raw_tb = 500 * 365 / 1024
print("Raw JSON    :", round(raw_tb, 1), "TB per year")
print("Parquet     :", round(raw_tb / 5, 1), "TB per year")

clean→features, clean→report, features→model, report→email

Skipped: ['features', 'report', 'model', 'email']

downstream = {"clean": ["features", "report"], "features": ["model"],
              "report": ["email"], "model": [], "email": []}
to_skip, queue = [], ["clean"]
while queue:
    task = queue.pop(0)
    for nxt in downstream[task]:
        if nxt not in to_skip:
            to_skip.append(nxt)
            queue.append(nxt)
print("Skipped:", to_skip)

100 GB/day, hot 30 days, cold 365 days

Hot : 3000 GB
Cold: 36500 GB
Monthly cost: Rs 16950

hot_gb = 100 * 30
cold_gb = 100 * 365
cost = hot_gb * 2 + cold_gb * 0.3
print("Hot :", hot_gb, "GB")
print("Cold:", cold_gb, "GB")
print("Monthly cost: Rs", round(cost))

sales: 5h old, limit 24 · fraud: 2h old, limit 1 · stock: 30h old, limit 24

sales  OK
fraud  STALE
stock  STALE

dashboards = {"sales": (5, 24), "fraud": (2, 1), "stock": (30, 24)}
for name, (age, limit) in dashboards.items():
    print(f"{name:6}", "OK" if age <= limit else "STALE")

as given

Self-hosted: Rs 4.0 lakh
Managed    : Rs 3.95 lakh

self_hosted = 2 * 1.5 + 1.0
managed = 3.2 + 0.5 * 1.5
print("Self-hosted: Rs", self_hosted, "lakh")
print("Managed    : Rs", managed, "lakh")

Real-world scenario questions (asked in company interviews)

Questions that test whether you understand the ecosystem, not just its names. Decide your answer first, then open the model answer.