Big data on the cloud: AWS, Azure and GCP

A ₹50 lakh surprise

Big data on the cloud

Words you’ll meet in this lesson

India regions (2026)

Region lists and service names change. Always check the provider’s current documentation before designing.

Big data on the cloud at three levels

Service models and the AWS, Azure and GCP map

The same job, three clouds

Databricks, Snowflake and Confluent run on all three clouds. Names are as of 2026 and may change.

Building a small cloud data platform

1. Pick a region and create the lake bucket

   Kabir creates a bucket in the Mumbai region with folders by zone: bronze/, silver/, gold/, versioning on, encryption on, public access blocked.

   The region decides latency, price and legal residency. Security settings are your job under the shared-responsibility model.

2. Stream app events into the lake

   App events flow into a managed stream (Kinesis, MSK, Event Hubs or Pub/Sub) and a delivery service writes them to bronze/ every few minutes as compressed files.

   No Kafka brokers to patch. The trade-off: you pay per shard, partition or GB, and you accept the provider’s limits.

3. Catalog the tables

   A crawler or a table format registers the files as tables (schema, partitions, location) in a catalog so SQL engines and Spark can find them by name.

   The catalog is what turns a bucket of files into a lake people can query, and where access permissions are managed.

4. Transform with managed Spark

   nodes, rate = 20, 40       # node-hours at illustrative Rs 40
   print("always on :", nodes * 24 * 30 * rate, "rupees/month")
   print("3 h a day :", nodes * 3 * 30 * rate, "rupees/month")

   always on : 576000 rupees/month 3 h a day : 72000 rupees/month
   A managed Spark cluster (EMR, Databricks, Dataproc) or a serverless Spark job builds silver and gold tables. Starting it only for the job instead of leaving it on is the easiest big saving.

5. Query with serverless SQL, carefully

   lake_tb, scan_fraction = 40, 0.01
   price_per_tb = 450       # illustrative rupees per TB scanned
   print("SELECT * (whole table):", lake_tb * price_per_tb, "rupees per query")
   print("one day, 2 columns    :", lake_tb * scan_fraction * price_per_tb, "rupees per query")

   SELECT * (whole table): 18000 rupees per query one day, 2 columns : 180.0 rupees per query
   Per-TB-scanned engines reward Parquet, partitions and selecting only needed columns. The same question can cost a hundred times less with a good layout.

6. Watch the bill like a metric

   Every resource is tagged with a team and project; a budget alert fires at 80% of the monthly limit; a weekly report shows the most expensive queries and idle resources.

   In the cloud, cost is an engineering metric, just like latency and errors. Teams that watch it daily rarely get bill shocks.

One serverless query: what Kabir sees, and what the cloud does

Step through the six frames. Left: Kabir running a query. Right: how a serverless engine plans, scales out, bills and disappears.

Real-time scenarios: cost, capacity and compliance

Three live scenarios: a monthly bill cut by 95% (real numbers from the program below), a spot worker reclaimed mid-job, and an advanced look at data residency, disaster recovery, egress and lock-in.

From ₹50 lakh to ₹2.4 lakh a month

These figures come from the program in “Run it on your PC” below, with illustrative prices. Watch each fix shrink TiffinGo’s monthly bill.

A spot worker disappears in the middle of the job

Spot (or preemptible) machines are spare cloud capacity at a deep discount, with one catch: the provider can take them back at short notice. Watch why they suit Spark batch jobs.

Where may the data live? Residency, DR, egress and lock-in

Choosing a cloud region is not only about price and speed. Watch four decisions an Indian fintech data team must get right before writing any pipeline.

How Indian companies use the cloud for data

Translate a design between clouds

A small dictionary maps each job in a data platform to a service on each cloud. The program prints the same architecture on AWS, Azure and Google Cloud. Then try the challenges: compare scan costs, always-on vs on-demand clusters, spot savings, egress and storage tiers.

services = {
    "lake storage":   {"AWS": "S3",         "Azure": "ADLS Gen2",       "GCP": "Cloud Storage"},
    "managed Spark":  {"AWS": "EMR / Glue", "Azure": "Databricks",      "GCP": "Dataproc"},
    "warehouse":      {"AWS": "Redshift",   "Azure": "Synapse/Fabric",  "GCP": "BigQuery"},
    "event stream":   {"AWS": "Kinesis/MSK","Azure": "Event Hubs",      "GCP": "Pub/Sub"},
    "orchestration":  {"AWS": "MWAA",       "Azure": "Data Factory",    "GCP": "Cloud Composer"},
}
design = ["event stream", "lake storage", "managed Spark", "warehouse", "orchestration"]

for cloud in ["AWS", "Azure", "GCP"]:
    print(f"{cloud:5}: " + " -> ".join(services[job][cloud] for job in design))

• "lake storage": {"AWS": "S3", "Azure": "ADLS Gen2", "GCP": "Cloud Storage"},Every cloud has the same categories; only the product names differ.
• design = ["event stream", "lake storage", "managed Spark", "warehouse", "orchestration"]Design in categories first, then pick the product on the chosen cloud.

Run it on your PC: estimate and shrink a cloud bill

This program estimates the monthly bill of a small cloud data platform (storage, SQL scans, a Spark cluster and data transfer) with illustrative prices, then applies five fixes one by one: columnar files and partitions, auto-shutdown, spot workers and keeping data in region. Change the assumptions at the top and run it again. Save it as cloud_bill.py.

"""Estimate a monthly cloud bill for a small data platform, and see what
partitioning, columnar files, spot instances and auto-shutdown save.
All prices are ILLUSTRATIVE round numbers in rupees, not a real price list."""

PRICE = {
    "storage_per_tb_month": 1_900,     # object storage, standard tier
    "scan_per_tb": 450,                # serverless SQL, charged per TB scanned
    "node_hour_on_demand": 40,         # one Spark worker node
    "node_hour_spot": 12,              # same node as spot / preemptible capacity
    "egress_per_tb": 7_000,            # data sent out of the cloud region
}
LAKE_TB = 60
QUERIES_PER_DAY = 400
NIGHTLY_NODES, NIGHTLY_HOURS = 20, 3


def bill(scan_tb_per_query, spot, cluster_always_on, egress_tb):
    storage = LAKE_TB * PRICE["storage_per_tb_month"]
    scans = scan_tb_per_query * QUERIES_PER_DAY * 30 * PRICE["scan_per_tb"]
    rate = PRICE["node_hour_spot"] if spot else PRICE["node_hour_on_demand"]
    hours = 24 if cluster_always_on else NIGHTLY_HOURS
    compute = NIGHTLY_NODES * hours * 30 * rate
    egress = egress_tb * PRICE["egress_per_tb"]
    return {"storage": storage, "queries": scans, "compute": compute, "egress": egress}


def show(name, b):
    total = sum(b.values())
    parts = ", ".join(f"{k} {v / 1e5:.1f}L" for k, v in b.items())
    print(f"{name:37} Rs {total / 1e5:6.1f} lakh/month  ({parts})")
    return total


before = show("1. Naive: CSV, SELECT *, always on", bill(0.8, False, True, 5))
step2 = show("2. + Parquet, partitions, columns", bill(0.02, False, True, 5))
step3 = show("3. + cluster only for the 3-hour job", bill(0.02, False, False, 5))
step4 = show("4. + spot workers", bill(0.02, True, False, 5))
after = show("5. + keep data in-region (no egress)", bill(0.02, True, False, 0))
print(f"\nSaving: {(1 - after / before) * 100:.0f}% of the original bill")

# In a terminal:  python cloud_bill.py

Adding up a cloud bill, one line at a time

Three meters are added: storage, compute and scans. Watch the total grow, and notice which meter dominates.

Worked examples: real problems, solved and traced in memory

Four cloud decisions, from mapping services to estimating a spot-instance plan. Each runs live and is traced.

S3, EMR, Redshift, Kinesis, MWAA

S3         -> Cloud Storage
EMR        -> Dataproc
Redshift   -> BigQuery
Kinesis    -> Pub/Sub
MWAA       -> Cloud Composer
QuickSight -> no direct match: check docs

6 charts, every 15 min × 12 h, 25 GB each, 22 days

every 15 min, raw table  :   154.7 TB scanned, Rs     69,609 per month
hourly, aggregate table  :     0.8 TB scanned, Rs        348 per month

₹450/TB vs ₹3 lakh/month · 150 TB, 1,200 TB

break-even: 667 TB per month
marketing   : serverless Rs 67,500 vs fixed Rs 300,000 -> serverless
data science: serverless Rs 540,000 vs fixed Rs 300,000 -> provisioned

160 node-hours, ₹40 vs ₹12, 15% rework, 3 on-demand nodes × 4 h

all on-demand: Rs 6400
mixed plan   : Rs 2522 (170 spot node-hours incl. rework)
saving       : 61%

Check yourself

Quick revision

Interview and viva questions

Practice problems

Cloud cost and design exercises with illustrative prices. The last two are business tasks.

25 TB

47500

print(25 * 1900)

300 GB

131.84

print(round(300 / 1024 * 450, 2))

event streaming

Event Hubs

azure = {"storage": "ADLS Gen2", "spark": "Azure Databricks", "event streaming": "Event Hubs"}
print(azure["event streaming"])

12 nodes, 19 idle h/day

273600

print(12 * 40 * 19 * 30)

36 TB, 7/360 days, 3/18 columns

0.117 TB

print(round(36 * 7 / 360 * 3 / 18, 3), "TB")

300 nh

on-demand: 12000 | spot: 4320

print("on-demand:", 300 * 40, "| spot:", round(300 * 1.2 * 12))

Rs 40/h vs Rs 2.1 lakh/year

5250 hours per year ( 60 % of the year )

hours = 2_10_000 / 40
print(round(hours), "hours per year (", round(hours / 8760 * 100), "% of the year )")

as given

Rs 3.02 lakh per month

total = 50 * 1900 + 10 * 30 * 450 + 15 * 4 * 30 * 40
print(f"Rs {total / 1e5:.2f} lakh per month")

Rs 2.4 lakh, 30 crore orders

Rs 0.80 per 1,000 orders

print(f"Rs {2.4e5 / 30e7 * 1000:.2f} per 1,000 orders")

Real-world scenario questions (asked in company interviews)

Cloud questions from real projects and interviews. Decide your answer first, then open the model answer.