Replication: leaders, followers and copies

2:13 am: “orders-db primary down”

What is replication?

Words you’ll meet in this lesson

Replication in tools you will use

Replication at three levels

Leaders, followers and how they sync

Synchronous vs asynchronous

One write, three copies, one failure

1. The leader writes and acknowledges

   leader, followers = [], {"F1": [], "F2": []}
   def write(entry):
       leader.append(entry)            # 1. leader appends to its log
       return "OK"                     # 2. async: acknowledge immediately
   print(write("order O1"), write("order O2"))
   print("leader:", leader, "| F1:", followers["F1"])

   OK OK leader: ['order O1', 'order O2'] | F1: []
   Every write goes to the leader, which appends it to its replication log. With asynchronous replication it says “OK” right away, before any follower has the change.

2. Followers replay the log

   leader = ["order O1", "order O2"]
   followers = {"F1": [], "F2": []}
   for f in followers:                 # 3. followers copy the log in order
       followers[f].extend(leader)
   print(followers)

   {'F1': ['order O1', 'order O2'], 'F2': ['order O1', 'order O2']}
   Followers receive the log entries in the same order and apply them, so they end up with identical data. Order matters: “add ₹100” then “set to ₹0” gives a different result in the other order.

3. Reads from a follower can be stale

   leader = ["O1", "O2", "O3", "O4"]
   f2 = ["O1", "O2"]                   # F2 is two entries behind
   print("read from leader:", len(leader), "orders | read from F2:", len(f2), "orders")
   print("lag:", len(leader) - len(f2), "entries")

   read from leader: 4 orders | read from F2: 2 orders lag: 2 entries
   Followers are good for spreading read load, but they can be behind. A user who just changed their address might read the old one from a lagging follower.

4. The leader fails: promote the best follower

   logs = {"F1": ["O1", "O2", "O3"], "F2": ["O1", "O2"]}
   acked = ["O1", "O2", "O3", "O4"]    # the leader had said OK to O4
   new_leader = max(logs, key=lambda f: len(logs[f]))
   print("promote:", new_leader, "| lost:", [o for o in acked if o not in logs[new_leader]])

   promote: F1 | lost: ['O4']
   The most up-to-date follower becomes leader. O4 had been acknowledged by the old leader but never reached a follower, so it is lost: the cost of asynchronous replication.

5. Redirect clients and rejoin

   Clients learn the new leader’s address (through a DNS change, a proxy or a coordination service). When the old leader comes back, it must rejoin as a follower and discard writes the new leader never had.

   If the old leader came back still believing it was the leader, there would be two leaders: split brain (scenario 3).

6. Choose the safety level per data

   Payments: synchronous or semi-synchronous replication within a region. Analytics replicas and cross-region copies: asynchronous. Kafka: acks=all with min.insync.replicas=2 for important topics.

   Replication is a dial between speed and safety. Choose it per dataset, based on how much loss (RPO) and downtime (RTO) the business can accept.

Riya changes her address: which copy answers?

Step through the six frames. Left: Riya updating and re-checking her address. Right: the leader and followers, and which copy serves each request.

Real-time scenarios: copies under failure

Three live scenarios: an automatic failover at 2 am, what Kafka’s acks setting really guarantees, and an advanced look at split brain and how fencing prevents it.

2:13 am: the leader dies

Three database nodes: A leads, B and C follow. At 2:13 am A's disk fails. Watch an automatic failover step by step, and where data can be lost.

Kafka acks: when is a message really safe?

A payments service writes events to a Kafka topic with replication factor 3. The producer setting acks decides when the producer is told “done”. Watch what each setting means when a broker crashes.

Split brain: two leaders at once

The worst replication failure is not a crash but a network split, where the old leader keeps running and a new one is elected. Watch how majorities and fencing tokens keep only one leader in charge.

Replication at work

A leader and two followers

Writes go to the leader's log. Each follower copies the log after its own delay. The program shows what each copy contains at a moment in time, which reads would be stale, and what a failover would lose. Then try the challenges: synchronous writes, choosing the new leader, quorums, Kafka acks and fencing tokens.

writes = [(0, "O1"), (2, "O2"), (4, "O3"), (6, "O4")]   # (time, order)
lag = {"F1": 1, "F2": 4}                                 # seconds behind the leader
now = 7

leader = [o for t, o in writes if t <= now]
followers = {f: [o for t, o in writes if t + d <= now] for f, d in lag.items()}

print("leader:", leader)
for f, data in followers.items():
    stale = len(leader) - len(data)
    print(f"{f}: {data}  ({stale} behind)")

# The leader crashes now: promote the follower with the most data
new_leader = max(followers, key=lambda f: len(followers[f]))
lost = [o for o in leader if o not in followers[new_leader]]
print("promote", new_leader, "| acknowledged but lost:", lost)

• lag = {"F1": 1, "F2": 4} # seconds behind the leaderAsynchronous followers apply changes a little later. Each has its own lag.
• followers = {f: [o for t, o in writes if t + d <= now] for f, d in lag.items()}A follower has a write only if the write's time plus its lag has passed.
• new_leader = max(followers, key=lambda f: len(followers[f]))On failover, the most up-to-date follower is promoted, losing as little as possible.

Run it on your PC: async vs semi-sync replication

This program simulates a leader and two followers on a clock. A client writes every two ticks; followers copy after a delay. It reads from a lagging follower, crashes the leader, promotes the best follower and lists acknowledged writes that were lost. Then it repeats with semi-synchronous replication and checks quorum settings. Save it as replication_sim.py and run it.

"""A leader with two followers, simulated step by step on a clock.
Writes go to the leader's log; followers copy the log after a delay (asynchronous replication).
We read from a follower, crash the leader, fail over, and count what was lost.
Then we repeat with synchronous replication to one follower (semi-sync)."""


def run(mode, lag={"F1": 2, "F2": 5}, crash_at=12):
    leader_log = []
    follower_log = {"F1": [], "F2": []}
    pending = []                         # (arrival_time, follower, entry)
    acked = []
    for t in range(0, crash_at):
        if t % 2 == 0:                   # a client write every 2 ticks
            entry = f"order@{t}"
            leader_log.append(entry)
            for f, d in lag.items():
                pending.append((t + d, f, entry))
            if mode == "semi-sync":      # wait until F1 has it before saying OK
                follower_log["F1"].append(entry)
                pending = [p for p in pending if not (p[1] == "F1" and p[2] == entry)]
            acked.append(entry)
        for p in [p for p in pending if p[0] <= t]:
            follower_log[p[1]].append(p[2])
            pending.remove(p)
        if mode == "async" and t == 7:
            print(f"  t={t}: read from F2 sees {len(follower_log['F2'])} of {len(leader_log)} orders (replication lag)")
    # leader crashes; promote the follower with the longest log
    new_leader = max(follower_log, key=lambda f: len(follower_log[f]))
    lost = [e for e in acked if e not in follower_log[new_leader]]
    print(f"  leader crashed at t={crash_at}; promoted {new_leader} with {len(follower_log[new_leader])} of {len(acked)} acknowledged orders")
    print(f"  acknowledged but LOST: {lost if lost else 'none'}")


print("Asynchronous replication:")
run("async")
print("\nSemi-synchronous (leader waits for F1 before acknowledging):")
run("semi-sync")

# Quorum check for leaderless replication
print("\nQuorum reads/writes with N = 3 replicas:")
for w, r in [(1, 1), (2, 1), (2, 2), (3, 1)]:
    print(f"  W={w} R={r}: {'always reads the latest write' if r + w > 3 else 'may read stale data'}")

# In a terminal:  python replication_sim.py

Followers applying the log, one line at a time

The leader's log has three entries. Follower F2 already has the first one. Watch it apply only the entries it is missing, in order.

Worked examples: real problems, solved and traced in memory

Four replication problems, from counting copies to choosing quorum settings. Each runs live and is traced.

Kafka 20 TB × 3, HDFS 50 TB × 3

Kafka: 20 TB of data x 3 copies = 60 TB of disk
HDFS: 50 TB of data x 3 copies = 150 TB of disk
Total raw disk: 210 TB

r1: 0.4 s, r2: 12 s, r3: 75 s

checkout: ['r1']
reports : ['r1', 'r2']

3,000 orders/s, lag 0.5 s normal, 4 s peak

normal: up to 1500 acknowledged orders could be lost
peak  : up to 12000 acknowledged orders could be lost
RPO target: 0 orders -> needs synchronous or semi-sync replication

N = 3

consistent settings: [(1, 3), (2, 2), (2, 3), (3, 1), (3, 2), (3, 3)]
write-heavy pick (W, R): (1, 3)
read-heavy pick  (W, R): (3, 1)

Check yourself

Quick revision

Interview and viva questions

Practice problems

Replication exercises, from lag to quorums. The last two are business tasks.

12 TB, RF 3

36 TB

print(12 * 3, "TB")

10500, 10320

180 entries behind

print(10500 - 10320, "entries behind")

3, 4, 5, 7

3 nodes -> majority 2
4 nodes -> majority 3
5 nodes -> majority 3
7 nodes -> majority 4

for n in [3, 4, 5, 7]:
    print(n, "nodes -> majority", n // 2 + 1)

3, 5, 7

3 nodes tolerate 1 failures
5 nodes tolerate 2 failures
7 nodes tolerate 3 failures

for n in [3, 5, 7]:
    print(n, "nodes tolerate", (n - 1) // 2, "failures")

F1: 880, F2: 905, F3: 870

F2

pos = {"F1": 880, "F2": 905, "F3": 870}
print(max(pos, key=pos.get))

lags [0.2, 0.4, 1.8, 0.3]

2160 orders at risk in the worst case

lags = [0.2, 0.4, 1.8, 0.3]
print(int(1200 * max(lags)), "orders at risk in the worst case")

3 replicas, 1 up to date

R=1: chance of stale result 67%
R=2: chance of stale result 33%
R=3: chance of stale result 0%

from math import comb
N, fresh = 3, 1
for R in [1, 2, 3]:
    stale = comb(N - fresh, R) / comb(N, R)
    print(f"R={R}: chance of stale result {stale:.0%}")

Rs 2.5 lakh/month, Rs 40 lakh/hour, 4 h/year

standby: Rs 30.0 lakh/year vs expected downtime: Rs 160 lakh/year
worth it

standby_per_year = 2.5 * 12
downtime_cost = 40 * 4
print(f"standby: Rs {standby_per_year} lakh/year vs expected downtime: Rs {downtime_cost} lakh/year")
print("worth it" if downtime_cost > standby_per_year else "not worth it")

2 s lag, 800/s, RPO 300 s

transactions at risk: 1600
RPO met

lag, rate, rpo = 2, 800, 300
print("transactions at risk:", lag * rate)
print("RPO met" if lag <= rpo else "RPO NOT met")

Real-world scenario questions (asked in company interviews)

Replication questions from real incidents and interviews. Decide your answer first, then open the model answer.