Lambda and Kappa architectures explained

Two numbers for the same sale

What are the Lambda and Kappa architectures?

Words you'll meet in this lesson

The two architectures at a glance

Lambda and Kappa at three levels

Lambda vs Kappa side by side

Typical tools in each layer

A query during the sale, the Lambda way

1. Every event is appended, twice over

   {"offset": 50213, "order": "O90017", "amount": 499, "type": "sale", "ts": "19:42:05"}

   The event goes into Kafka. From there, one consumer writes it to the master dataset in the lake (batch layer), and another feeds the speed layer.

2. The batch view from last night

   batch_view = {"revenue_till_yesterday": 18_40_00_000}   # computed at 02:00 by Spark

   A nightly Spark job recomputed revenue from all raw events up to midnight: duplicates removed, refunds subtracted, late events included. Exact, but it knows nothing about today.

3. The speed layer covers today

   speed_view = {"revenue_today": 0}
   for e in todays_events:                      # updated within seconds of each event
       speed_view["revenue_today"] += e["amount"] if e["type"] == "sale" else -e["amount"]

   The stream processor keeps a running total for events after midnight only. Quick, but approximate: a duplicate or a late refund may slip through until tomorrow's batch.

4. The serving layer merges

   batch_view = 18_40_00_000
   speed_view = 2_31_00_000
   print("Revenue (this season):", batch_view + speed_view)

   Revenue (this season): 207100000
   The dashboard asks the serving layer; it adds the exact past to the fresh present. The CEO sees a number that is both complete and up to the minute.

5. Next night, batch catches up

   At 02:00 the batch job recomputes everything including the sale day: ₹2.24 crore for the day, not ₹2.31 crore.

   The new batch view replaces the speed layer's estimate for that day, and the speed layer starts again from zero for the new day. Errors in the fast path never last longer than one batch cycle. That is Lambda's safety net.

Sale night, the morning after, and the week Kabir switched to Kappa

Step through six frames. Left: what the CEO and finance see. Right: which layer is doing the work.

Real-time scenarios with both architectures

Watch the live estimate and the batch correction on one chart, replay a Kafka log after a bug, and (advanced) walk the decision an architect makes when choosing.

Sale night on one chart: speed layer vs batch correction

Revenue per hour during the sale. The green line is what the speed layer showed live; the blue line is what the batch layer computed afterwards from complete data.

Kappa in action: replaying a month after a bug

TiffinGo moved to Kappa. On 19 October, a bug is found in how coupons were applied since 20 September. The log keeps 30 days. Watch the team fix a month of numbers without any batch system.

The architect's decision: Lambda, Kappa or a unified engine?

Three teams at TiffinGo ask for real-time plus historical data. Watch the architect walk each one through the same questions and reach different answers.

Where these patterns appear

Simplified patterns; real systems often blend Lambda and Kappa ideas.

Lambda in miniature: batch view + speed view

An append-only list of events plays the master dataset. Last night's batch job computed an exact view up to a cut-off; the speed layer keeps a running total for everything after it. The query adds the two. Then, in the challenges, replay the log Kappa-style, watch two codebases drift, and size a log for replay.

# The master dataset: every event ever, append-only (sale = +, refund = -)
master = [("O1", 120, "sale"), ("O2", 250, "sale"), ("O3", 90, "sale"),
          ("O2", 250, "refund"), ("O4", 300, "sale"),          # up to last night
          ("O5", 140, "sale"), ("O6", 60, "sale")]             # arrived today

def revenue(events):
    total = 0
    for order, amount, kind in events:
        total += -amount if kind == "refund" else amount
    return total

CUTOFF = 5                                 # last night's batch job processed events 0-4

batch_view = revenue(master[:CUTOFF])      # BATCH LAYER: exact, recomputed nightly
speed_view = revenue(master[CUTOFF:])      # SPEED LAYER: only events since the batch run

print("Batch view (till last night):", batch_view)
print("Speed view (today, live)    :", speed_view)
print("Query answer = batch + speed:", batch_view + speed_view)

# A new event arrives: only the speed view changes, instantly
master.append(("O4", 300, "refund"))
speed_view = revenue(master[CUTOFF:])
print("After a refund arrives      :", batch_view + speed_view)

• master = [...]The master dataset is only ever appended to. Corrections (like a refund) are new events, not edits.
• batch_view = revenue(master[:CUTOFF])The batch layer recomputes from all raw events up to the cut-off. Accurate, but only as fresh as its last run.
• speed_view = revenue(master[CUTOFF:])The speed layer handles only the recent tail, so it can update within seconds.
• batch_view + speed_viewThe serving layer's job: merge the two views into one answer.

Run it on your PC: a real log file, replayed

This program writes an append-only log file (orders.log), one JSON event per line with an offset, just like a Kafka partition. It then shows Kappa (a buggy v1 job, then a fixed v2 replayed from offset 0) and Lambda (batch view up to offset 4 plus speed view after it). Save it as replay_log.py, run it, and open orders.log to see the events.

import json, os

# ---------- An append-only event log, like a Kafka topic ----------
LOG = "orders.log"
raw = [("O1", 120, "sale"), ("O2", 250, "sale"), ("O3", 90, "sale"), ("O2", 250, "refund"),
       ("O4", 300, "sale"), ("O5", 140, "sale"), ("O6", 60, "sale"), ("O4", 300, "refund")]
with open(LOG, "w") as f:
    for offset, (oid, amount, kind) in enumerate(raw):
        f.write(json.dumps({"offset": offset, "order": oid, "amount": amount, "type": kind}) + "\n")
print(f"Log written: {len(raw)} events (offsets 0-{len(raw) - 1}), never edited afterwards\n")


def consume(logic, start=0, end=None):
    """Read the log from an offset, like a consumer reading a Kafka partition."""
    total = 0
    with open(LOG) as f:
        for line in f:
            e = json.loads(line)
            if e["offset"] < start or (end is not None and e["offset"] >= end):
                continue
            total = logic(total, e)
    return total


def v1(total, e):                      # BUG: treats refunds as sales
    return total + e["amount"]


def v2(total, e):                      # FIXED: refunds reduce revenue
    return total - e["amount"] if e["type"] == "refund" else total + e["amount"]


# ---------- KAPPA: one streaming code path; fix a bug by replaying the log ----------
print("KAPPA")
print(f"  {'v1 job (live since morning) says revenue':44}: Rs {consume(v1)}")
print("  bug found -> deploy v2, replay from offset 0")
print(f"  {'v2 job after replay says revenue':44}: Rs {consume(v2)}")

# ---------- LAMBDA: batch view (complete but late) + speed view (recent, fast) ----------
print("\nLAMBDA")
batch_upto = 5                         # last night's batch job processed offsets 0-4
batch_view = consume(v2, 0, batch_upto)
speed_view = consume(v2, batch_upto)   # events that arrived since the batch ran
print(f"  {'batch view (offsets 0-4, computed at 02:00)':44}: Rs {batch_view}")
print(f"  {'speed view (offsets 5-7, live since then)':44}: Rs {speed_view}")
print(f"  {'query answer = batch view + speed view':44}: Rs {batch_view + speed_view}")

# In a terminal:  python replay_log.py

A serving-layer query, one line at a time

The batch view was computed overnight; the speed layer adds today's events one by one. Watch speed change with each event while batch stays fixed, and the query merge them at the end.

Worked examples: real problems, solved and traced in memory

Four problems on merging views, catching up, code drift and replay planning. Each runs live and is traced.

batch 18.40 cr · today: 1.12, +0.80, +0.39 (cr)

dashboard: 19.52 cr
dashboard: 20.32 cr
dashboard: 20.71 cr

offsets 0–9, cut-off 6 then 10

cutoff 6 | batch: [0, 1, 2, 3, 4, 5] | speed: [6, 7, 8, 9]
cutoff 10 | batch: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] | speed: []

events with minutes-until-cancel (None = not cancelled)

batch: 510 | stream: 820 | drift: 310

3,000,000/day × 90 days · replay 80,000/s · live 40/s · budget 8 h

Backlog: 270,000,000 events
Catch-up time: 0.94 hours
Fits in 8 hours? True

Check yourself

Quick revision

Interview and viva questions

Practice problems

Exercises on views, replays and planning, from easy to advanced, ending with two business decisions.

9450 + 312

Dashboard: 9762

batch, speed = 9450, 312
print("Dashboard:", batch + speed)

“seconds-old count”, “exact last month”, “combined answer”

seconds-old count  -> speed layer
exact last month   -> batch layer
combined answer    -> serving layer

needs = {"seconds-old count": "speed layer", "exact last month": "batch layer",
         "combined answer": "serving layer"}
for need, layer in needs.items():
    print(f"{need:18} -> {layer}")

500000 × 500 B × 30

6.98 GB

print(round(500_000 * 500 * 30 / 1024**3, 2), "GB")

(20,100), (23,50), (24,70), (26,30)

Speed view: 100

events = [(20, 100), (23, 50), (24, 70), (26, 30)]
cutoff = 24
print("Speed view:", sum(a for t, a in events if t >= cutoff))

2.31 vs 2.24

3.1 % higher than exact

live, exact = 2.31, 2.24
print(round((live - exact) / exact * 100, 1), "% higher than exact")

sale 500, refund 200, sale 300, refund 100

Buggy: 1100 | Fixed after replay: 500

log = [(500, "sale"), (200, "refund"), (300, "sale"), (100, "refund")]
buggy = sum(a for a, k in log)
fixed = sum(-a if k == "refund" else a for a, k in log)
print("Buggy:", buggy, "| Fixed after replay:", fixed)

backlog 20,000,000; 40,000/s vs 500/s

8.4 minutes until v2 catches up

backlog = 20_000_000
minutes = backlog / (40_000 - 500) / 60
print(round(minutes, 1), "minutes until v2 catches up")

12 metrics

Lambda (two codebases): 84 days
Unified engine        : 60 days

metrics = 12
print("Lambda (two codebases):", metrics * (3 + 4), "days")
print("Unified engine        :", metrics * 5, "days")

6.5 h vs 8 h window

Fits: True | buffer: 1.5 hours

replay_h, window_h = 6.5, 8
print("Fits:", replay_h <= window_h, "| buffer:", window_h - replay_h, "hours")

Real-world scenario questions (asked in company interviews)

Architecture questions asked in data engineering interviews. Decide your answer first, then open the model answer.