Hadoop in the cloud, and Cloudera

A cluster that exists for three hours a night

Hadoop in the cloud

Words you'll meet in this lesson

What the platforms ship (checked 22 Oct 2026)

Architecture at a glance

Left: the classic cluster, where every worker stores blocks and runs containers. Right: the cloud pattern, where storage is a service and clusters are temporary.

The cloud at three levels

Ways to run Hadoop workloads today

Migration patterns

Animated example: one night on a transient cluster

Deciding with numbers

We did not create cloud resources for this lesson, so there is no cloud lab output here. Instead we feed numbers we measured on our own cluster into a small calculator, cloud_vs_onprem.py, with prices clearly marked as illustrative.

1. Measure your own throughput first

   DistCp, 5 files, 162 MB, prod → DR          25.5 s   (Lesson 12.2)
   TestDFSIO write, 1 task                     113 MB/s (Lesson 12.3)
   TestDFSIO write, 6 tasks, aggregate         ~210 MB/s on one shared laptop disk
   Spark on YARN, item revenue, wall time      ~46–50 s incl. executor start (Lesson 13.2)

   Cloud sizing starts with how fast your jobs and copies really are. Our laptop numbers are small, but the method is the same on a 200-node cluster.

2. Compare compute: always-on vs per-night

   always_on = workers * 24 * 30 * NODE_HOUR
   transient = workers * (job_hours + 0.25) * 30 * NODE_HOUR

   always-on 20 workers : Rs 576,000 transient, 3h15m a night : Rs 78,000 (14% of always-on) transient, 4 core + 16 spot: Rs 40,560 (7%)
   The ratio is what matters: 3 h 15 min of 24 h is 14%. Spot workers cut it further. Prices are placeholders: check the provider's price page.

3. Compare storage: three copies on disks vs object storage

   hdfs = data_tb * 3 / 0.7 * DISK_TB_MONTH   # x3 copies, disks at most 70% full
   obj  = data_tb * OBJECT_TB_MONTH

   HDFS: 429 TB of raw disk : Rs 385,714 object storage, 100 TB : Rs 170,000
   100 TB in HDFS needs about 429 TB of raw disk (three copies, 30% headroom, Lesson 12.3). Erasure coding (Lesson 7.3) or Parquet compression (Lesson 10.2) shrink both sides.

4. Estimate the move

   mb_s = gbps * 1000 / 8 * 0.7      # 70% of the link usable

   at our lab's DistCp speed (6.4 MB/s): 191 days over a 1 Gbps link at 70% use (88 MB/s): 13.9 days over a 10 Gbps link at 70% use (875 MB/s): 1.4 days
   Migration is often limited by the network link, not by Hadoop. Large moves use dedicated links or physical transfer appliances, and DistCp with -update catches up the changes made during the copy.

5. Change the paths, not the logic

   spark-submit job.py hdfs://namenode/user/hadoop/data/student_orders.csv   # our lab
   spark-submit job.py s3a://canteen-data/orders/                              # AWS
   spark-submit job.py gs://canteen-data/orders/                               # Google Cloud

   The Hadoop FileSystem API (Lesson 6.5) hides the store behind a URI scheme. What changes is configuration: credentials come from cloud roles, and output commit uses a cloud-aware committer.

The same job, two lifecycles

Six steps of one nightly job: on a long-running on-premises cluster (left in each picture) and on a transient cloud cluster.

Cloud and Cloudera at work

Simplified; organisations differ in tools, prices, contracts and regulations. This is not legal or financial advice.

Sources (checked 22 Oct 2026): Amazon EMR 7.x application versions · Dataproc 2.3.x release versions · HDInsight roadmap (Directions on Microsoft) · HDInsight 4.0 retirement · HDInsight on AKS retirement · Cloudera Base on premises 7.3.2 · Apache Hadoop releases

Always-on vs transient: a cost and time calculator

Prices are illustrative variables, not quotes: change them to the numbers on your provider's price page. The structure of the comparison is what matters.

NODE_HOUR = 40.0            # illustrative Rs per worker-hour
workers, job_hours, nights = 20, 3, 30
setup = 0.25                # 15 minutes to create and terminate

always_on = workers * 24 * nights * NODE_HOUR
transient = workers * (job_hours + setup) * nights * NODE_HOUR
print(f"always-on : Rs {always_on:>9,.0f} a month")
print(f"transient : Rs {transient:>9,.0f} a month ({transient / always_on:.0%})")
busy = job_hours / 24
print(f"the always-on cluster is busy {busy:.0%} of the time")

• transient = workers * (job_hours + setup) * nights * NODE_HOURYou pay only while the cluster exists, including a few minutes to boot and shut down.
• busy = job_hours / 24Utilisation decides the winner. A cluster busy 20 hours a day is cheaper to keep running.

Run it on your PC: compare the options with your own numbers

No cloud account is needed: this lesson did not create cloud resources. cloud_vs_onprem.py takes the throughput we measured in Lessons 12.2 and 12.3 and illustrative prices, and prints the three comparisons from Part 4. Put your provider's real prices into the variables at the top before using the numbers for a decision.

# cloud_vs_onprem.py: always-on cluster vs a transient (per-job) cloud cluster, and the time to
# move data to the cloud. ALL PRICES ARE ILLUSTRATIVE - check the provider's price page.
# Measured inputs come from our lab: DistCp 162 MB in 25.5 s (12.2), TestDFSIO aggregate write
# ~210 MB/s on one shared laptop disk (12.3).

# --- 1. compute: a nightly 3-hour job on 20 workers -----------------------------------------
NODE_HOUR = 40.0          # illustrative Rs per worker-hour (on-demand)
SPOT_DISCOUNT = 0.6       # illustrative: spot/preemptible workers 60% cheaper
workers, job_hours = 20, 3
always_on = workers * 24 * 30 * NODE_HOUR
transient = workers * (job_hours + 0.25) * 30 * NODE_HOUR          # +15 min to create and terminate
spot = 4 * (job_hours + 0.25) * 30 * NODE_HOUR + 16 * (job_hours + 0.25) * 30 * NODE_HOUR * (1 - SPOT_DISCOUNT)
print("1. Compute for a month (illustrative prices)")
print(f"   always-on 20 workers      : Rs {always_on:>10,.0f}")
print(f"   transient, 3h15m a night  : Rs {transient:>10,.0f}  ({transient / always_on:.0%} of always-on)")
print(f"   transient, 4 core + 16 spot: Rs {spot:>10,.0f}  ({spot / always_on:.0%})")

# --- 2. storage: 100 TB of data, HDFS x3 replication vs object storage ---------------------
data_tb = 100
DISK_TB_MONTH = 900.0     # illustrative Rs per raw TB-month of cluster disk (server + power + ops)
OBJECT_TB_MONTH = 1700.0  # illustrative Rs per TB-month of object storage (replication included)
hdfs = data_tb * 3 / 0.7 * DISK_TB_MONTH                            # x3 copies, keep disks <= 70% full
obj = data_tb * OBJECT_TB_MONTH
print("2. Storage for 100 TB a month (illustrative prices)")
print(f"   HDFS: {data_tb * 3 / 0.7:.0f} TB of raw disk  : Rs {hdfs:>10,.0f}")
print(f"   object storage, 100 TB  : Rs {obj:>10,.0f}")

# --- 3. moving data: our measured DistCp speed, and a real network link --------------------
lab_mb_s = 162 / 25.5
print("3. Moving 100 TB")
print(f"   at our lab's DistCp speed ({lab_mb_s:.1f} MB/s): {100 * 1024 * 1024 / lab_mb_s / 86400:,.0f} days")
for gbps in [1, 10]:
    mb_s = gbps * 1000 / 8 * 0.7                                      # assume 70% of the link is usable
    print(f"   over a {gbps:>2} Gbps link at 70% use ({mb_s:,.0f} MB/s): {100 * 1024 * 1024 / mb_s / 86400:5.1f} days")

# Run:  python3 cloud_vs_onprem.py

Which URI scheme talks to which store?

Hadoop picks a FileSystem class from the scheme of the path. Follow four paths through the lookup.

Worked examples: real problems, solved and traced in memory

Four cloud decisions from different industries, each with the numbers written out. All prices are illustrative.

30 workers · 4 h a night · ₹50/hour · 30 nights

always-on Rs 1,080,000, transient Rs 191,250 (18%)

40 TB · ×3 / 0.7 · ₹900 vs ₹1,700

HDFS 171 TB raw: Rs 154,286 | object store: Rs 68,000

2 PB · 10 Gbps · 70%

28 days at 875 MB/s

3 options · 4 criteria

A on-prem  total 9  (residency, steady, burst, ops) = [3, 3, 1, 2]
B cloud    total 9  (residency, steady, burst, ops) = [2, 1, 3, 3]
C hybrid   total 10  (residency, steady, burst, ops) = [3, 3, 3, 1]

Check yourself

Quick revision

Interview and viva questions

Practice problems

Cloud and Cloudera exercises using our lab measurements and illustrative prices.

6 / 24

25% busy

print(f"{6 / 24:.0%} busy")

one path, two stores

s3a://shop/data/orders
gs://shop/data/orders

path = "hdfs://namenode/data/orders"
rest = path.split("/", 3)[3]
print("s3a://shop/" + rest)
print("gs://shop/" + rest)

25 × 3 / 0.7

107 TB

print(f"{25 * 3 / 0.7:.0f} TB")

hours + 0.25 vs 24

6 transient cheaper
12 transient cheaper
18 transient cheaper
23 transient cheaper
23.75 always-on cheaper or equal
24 always-on cheaper or equal

for h in [6, 12, 18, 23, 23.75, 24]:
    print(h, "transient cheaper" if h + 0.25 < 24 else "always-on cheaper or equal")

10 workers · 5 h · 30 nights

Rs 60,000 -> Rs 31,200, saving Rs 28,800

full = 10 * 5 * 30 * 40
spot = 2 * 5 * 30 * 40 + 8 * 5 * 30 * 40 * 0.4
print(f"Rs {full:,.0f} -> Rs {spot:,.0f}, saving Rs {full - spot:,.0f}")

100 GB · 6.35 MB/s

4.5 hours

mb_s = 162 / 25.5
print(f"{100 * 1024 / mb_s / 3600:.1f} hours")

50 × 72 × ₹40

Rs 144,000

print(f"Rs {50 * 72 * 40:,}")

3 inputs

(True, 0.8, 'high') -> hybrid: regulated data on-prem, masked data to the cloud for peaks
(False, 0.15, 'high') -> cloud with transient clusters
(False, 0.85, 'low') -> compare carefully: steady load can be cheaper on owned hardware
(True, 0.5, 'low') -> on-prem (e.g. Cloudera Base)

def suggest(on_prem_rule, utilisation, burst):
    if on_prem_rule and burst == "high":
        return "hybrid: regulated data on-prem, masked data to the cloud for peaks"
    if on_prem_rule:
        return "on-prem (e.g. Cloudera Base)"
    if utilisation > 0.7 and burst == "low":
        return "compare carefully: steady load can be cheaper on owned hardware"
    return "cloud with transient clusters"
for case in [(True, 0.8, "high"), (False, 0.15, "high"), (False, 0.85, "low"), (True, 0.5, "low")]:
    print(case, "->", suggest(*case))

Real-world scenario questions (asked in company interviews)

Cloud-migration and platform questions are common for senior data engineers and architects.