Batch vs stream processing: now or later?

Final over: the live score and tomorrow's newspaper

What are batch and stream processing?

Words you'll meet in this lesson

Batch and stream at a glance

Batch vs stream at three levels

Three processing styles, three window types

Window types

Delivery guarantees

A UPI payment's fraud check, in under a second

1. The payment event arrives

   {"txn": "T88214", "payer": "aman@okbank", "amount": 4900,
    "device": "new-android-7f", "city": "Kolkata", "event_time": "10:02:11"}

   The bank's app publishes the event to a Kafka topic before the money moves. The stream processor reads it within milliseconds.

2. Look up state for this payer

   usual_city = "Pune"; known_devices = {"iphone-12a"}
   recent_payments = [("10:00:40", 4900), ("10:01:05", 4900), ("10:01:31", 4900), ("10:01:50", 4900)]

   The processor keeps state per payer: usual city, known devices and the last few minutes of payments. This lives in the processor's state store, not in a slow database call.

3. Apply a sliding-window rule

   device, city = "new-android-7f", "Kolkata"     # from the new event
   in_last_2_min = [p for p in recent_payments if p[0] >= "10:00:11"]
   suspicious = len(in_last_2_min) >= 4 and device not in known_devices and city != usual_city
   print(suspicious)

   True
   Five payments of ₹4,900 within two minutes, from a new phone, in a city 1,600 km from usual. A sliding window of 2 minutes catches it.

4. Decide and respond

   The payment is put on hold and the customer gets a call: “Did you make these payments?”

   The whole check took about 80 milliseconds. A batch fraud report tomorrow morning would have found the same pattern, after the money was gone.

5. Batch still has a job

   The same events are also written to the lake.

   Every night a batch job uses months of history to retrain the fraud model and tune the rules. Stream makes the decision; batch makes the decision-maker smarter. Most real systems use both.

The same overs, processed as a stream and as a batch

Step through the six frames. Left: Riya's phone, then the newspaper. Right: the stream processor during the match, and the batch job after it.

Real-time scenarios where the choice matters

Three live scenarios: one where only streaming saves the day, one that shows the three window types on a real timeline, and an advanced one about events that arrive late.

A stolen phone: stream stops it, batch reports it

Aman's phone is stolen in a crowded market. The thief starts sending UPI payments of ₹4,900 every 20–30 seconds. Watch the stream processor's 2-minute sliding window, and compare with a nightly batch check.

One timeline, three windows

The same food orders from one office building between 13:00 and 13:40. Watch how tumbling, sliding and session windows group them differently, and why each answers a different business question.

The order from the tunnel: late data and watermarks

A passenger on a local train orders food at 18:58, just as the train enters a tunnel. The phone stays offline for 4 minutes and the event reaches the system at 19:02. Which minute should it count in, and how long should the system wait?

Which one do real companies use?

Examples are simplified; real systems combine both styles. Rules and numbers shown are illustrative.

The same orders, processed two ways

Six orders arrive during the day. The stream job updates its answer with every event; the batch job waits for the complete list and answers once. Compare when each answer is available. Then try windows, a fraud rule, late events and duplicates in the challenges.

orders = [  # (minute, kitchen, amount)
    (1, "K12", 120), (1, "K11", 90), (2, "K12", 250),
    (3, "K12", 180), (3, "K13", 60), (4, "K12", 140),
]

# STREAM: update the answer as each event arrives
def stream_job(events):
    total = 0
    for minute, kitchen, amount in events:
        total = total + amount
        print(f"[stream  @ minute {minute}] +{amount:>3} from {kitchen} -> running total Rs {total}")

# BATCH: wait until all data is collected, then process it in one go
def batch_job(all_orders):
    total = sum(amount for _, _, amount in all_orders)
    busiest = max(set(k for _, k, _ in all_orders),
                  key=lambda k: sum(a for _, kk, a in all_orders if kk == k))
    print(f"[batch   @ end of day] {len(all_orders)} orders, Rs {total}, busiest kitchen {busiest}")

stream_job(orders)
batch_job(orders)

• for minute, kitchen, amount in events:The stream job never sees “all” the data. It handles one event, updates its state (total) and moves on.
• print(f"[stream @ minute {minute}] ...An answer is available after every event: low latency.
• total = sum(amount for _, _, amount in all_orders)The batch job has the complete, bounded data, so it can answer harder questions in one pass, like “which kitchen was busiest overall?”

Run it on your PC: a live score stream in your terminal

This program generates a “live” ball every 0.3 seconds and processes it as a stream: the score updates immediately, and a 6-ball tumbling window closes at the end of every over. After the last ball it prints a batch summary computed from the complete data. Save it as live_score.py and watch it scroll.

import random, time

random.seed(27)
OUTCOMES = [0, 0, 1, 1, 1, 2, 4, 6, "W"]


def ball_stream(overs=3):
    """An endless-looking stream of balls, one every 0.3 s (sped up)."""
    for over in range(1, overs + 1):
        for ball in range(1, 7):
            time.sleep(0.3)
            yield over, ball, random.choice(OUTCOMES)


# STREAM processing: update the answer the moment each ball arrives
total, wickets, over_runs, per_over = 0, 0, 0, []
start = time.time()
for over, ball, result in ball_stream():
    if result == "W":
        wickets += 1
    else:
        total += result
        over_runs += result
    print(f"{time.time() - start:4.1f}s  ball {over - 1}.{ball}  {str(result):>2}   score {total}/{wickets}")
    if ball == 6:                       # the "over" window closes
        print(f"        --- over {over}: {over_runs} runs ---")
        per_over.append(over_runs)
        over_runs = 0

# BATCH processing: after the match, compute from the complete data
print("Batch summary after the match:", per_over, "| run rate", round(total / len(per_over), 2))

# In a terminal:  python live_score.py

A tumbling window, one event at a time

Orders arrive with the minute they happened. The stream job keeps a running total for the current minute; when an event from a new minute arrives, the old window closes and its result is printed. Watch window and total in the frame.

Worked examples: real problems, solved and traced in memory

Four problems from first-week analyst to streaming engineer. Each runs live and is traced line by line.

fraud check 1 s · live score 2 s · salary 86400 s · monthly bill 604800 s · surge price 30 s · sales report 21600 s

fraud check   STREAM
live score    STREAM
salary        batch
monthly bill  batch
surge price   STREAM
sales report  batch

12 balls: 6,1,1,1,0,0, 1,W,2,1,2,4

Runs per over: [9, 10]
Best over: 2

times (s): 0, 10, 20, 30, 45, 200

0 1 ok
10 2 ok
20 3 ok
30 4 FLAG
45 5 FLAG
200 1 ok

event times (minute): 1, 4, 12, 8, 15, 3, 22

Windows: {0: 3, 10: 2, 20: 1}
Late: [3]

Check yourself

Quick revision

Interview and viva questions

Practice problems

Window and latency exercises, from easy to advanced. The last two are business decisions about cost and service levels.

[2, 3600, 0.5]

2 -> stream
3600 -> batch
0.5 -> stream

for need in [2, 3600, 0.5]:
    print(need, "->", "stream" if need < 10 else "batch")

900000 events, 300 s

3000.0 events per second

print(900000 / 300, "events per second")

2500 events/s

9,000,000 events per hourly batch

print(f"{2500 * 3600:,} events per hourly batch")

[1, 3, 9, 12, 15, 27, 28, 29]

{0: 3, 10: 2, 20: 3}

minutes = [1, 3, 9, 12, 15, 27, 28, 29]
counts = {}
for m in minutes:
    w = (m // 10) * 10
    counts[w] = counts.get(w, 0) + 1
print(counts)

[30, 32, 34, 40, 41]

30 -> avg of last 3: 30.0
32 -> avg of last 3: 31.0
34 -> avg of last 3: 32.0
40 -> avg of last 3: 35.3
41 -> avg of last 3: 38.3

temps = [30, 32, 34, 40, 41]
for i in range(len(temps)):
    window = temps[max(0, i - 2):i + 1]
    print(temps[i], "-> avg of last 3:", round(sum(window) / len(window), 1))

[0, 5, 8, 40, 42, 90]

Sessions: 3

clicks = [0, 5, 8, 40, 42, 90]
sessions = 1
for prev, cur in zip(clicks, clicks[1:]):
    if cur - prev > 20:
        sessions = sessions + 1
print("Sessions:", sessions)

[e1, e2, e2, e3, e1, e4]

Unique: 4 | duplicates dropped: 2

events = ["e1", "e2", "e2", "e3", "e1", "e4"]
unique = []
for e in events:
    if e not in unique:
        unique.append(e)
print("Unique:", len(unique), "| duplicates dropped:", len(events) - len(unique))

Rs 30/node-hour

Stream: Rs 86400
Batch : Rs 36000

stream = 4 * 24 * 30 * 30
batch = 20 * 2 * 30 * 30
print("Stream: Rs", stream)
print("Batch : Rs", batch)

[80, 95, 120, 1500, 70, 90, 2100, 85]

75 % within 1 second

times_ms = [80, 95, 120, 1500, 70, 90, 2100, 85]
ok = sum(1 for t in times_ms if t <= 1000)
print(round(ok / len(times_ms) * 100), "% within 1 second")

Real-world scenario questions (asked in company interviews)

Situations where choosing batch or stream, or handling time correctly, is the real test. Decide your answer first, then open the model answer.