NoSQL databases: key-value, document, wide-column and graph

Five databases for one app?

What is a NoSQL database?

Words you’ll meet in this lesson

The four families and well-known examples

NoSQL at three levels

Key-value, document, wide-column and graph

SQL vs NoSQL

Same order, four models

One order across five databases

1. Key-value: Riya’s cart in Redis

   import time
   cache = {}
   def set_key(k, v, ttl):
       cache[k] = (v, time.time() + ttl)
   def get_key(k):
       v, exp = cache.get(k, (None, 0))
       return v if time.time() < exp else None
   set_key("cart:U9", {"thali": 1, "lassi": 2}, ttl=1800)
   set_key("otp:9876", "482913", ttl=-1)          # already expired
   print(get_key("cart:U9"), get_key("otp:9876"))

   {'thali': 1, 'lassi': 2} None
   A cart is read and written by one key, many times a minute, and is useless after half an hour. A key-value store with a TTL does exactly this in about a millisecond, and the expired OTP disappears by itself.

2. Document: each kitchen is one document

   kitchens = [
       {"_id": "K11", "name": "Annapurna", "veg": True, "menu": [{"dish": "thali", "price": 220}]},
       {"_id": "K12", "name": "Spice Box", "veg": False, "menu": [{"dish": "biryani", "price": 280}], "halal": True},
   ]
   veg = [k["name"] for k in kitchens if k["veg"]]
   print("veg kitchens:", veg, "| K12 extra field halal:", kitchens[1].get("halal"))

   veg kitchens: ['Annapurna'] | K12 extra field halal: True
   The menu lives inside the kitchen document, so one read fetches the whole page. Kitchens can have extra fields (like halal) without changing a schema for everyone.

3. Wide-column: rider pings partitioned by rider

   pings = {}   # partition key = rider, rows kept sorted by time
   for rider, t, lat in [("R7", 3, 18.52), ("R9", 1, 19.07), ("R7", 1, 18.50), ("R7", 2, 18.51)]:
       pings.setdefault(rider, []).append((t, lat))
       pings[rider].sort()
   print("R7 partition:", pings["R7"])
   print("latest for R7:", pings["R7"][-1])

   R7 partition: [(1, 18.5), (2, 18.51), (3, 18.52)] latest for R7: (3, 18.52)
   Each rider’s pings live together in one partition, kept sorted by time. “Latest position of R7” reads one partition, even when there are billions of pings in total spread over many nodes.

4. Graph: who referred whom

   refers = {"Riya": ["Kabir", "Meera"], "Kabir": ["Arjun"], "Meera": [], "Arjun": ["Zoya"]}
   level1 = refers["Riya"]
   level2 = [p for f in level1 for p in refers[f]]
   print("referred by Riya:", level1, "| their referrals:", level2)

   referred by Riya: ['Kabir', 'Meera'] | their referrals: ['Arjun']
   Relationships are stored directly, so “referrals of my referrals” is a short walk. In SQL each extra hop is another self-join on a large table.

5. Relational: the payment itself

   When Riya pays ₹420, the order row, the payment row and the wallet balance change together inside one PostgreSQL transaction: all or nothing.

   This is where ACID matters most. If the payment fails half-way, nothing is saved. That is why money stays in a relational database.

6. Polyglot persistence

   One order touched five databases, each doing what it does best. Events from all of them also flow into the data lake for analytics.

   The cost: more systems to run, monitor and keep in sync. Teams add a new database only when an access pattern truly needs it.

One app session: what Riya sees, and which database answers

Step through the six frames. Left: Riya using the app. Right: the database that serves each step, and the query it runs.

Real-time scenarios: NoSQL under pressure

Three live scenarios: a key-value cache that saves a flash sale, a Cassandra table designed around its query, and an advanced graph search that exposes a fraud ring.

Flash sale: a key-value cache saves the database

At midnight a sale starts and everyone opens the same few kitchen pages. Watch the cache-aside pattern take the load off the database.

Designing a Cassandra table around its query

TiffinGo stores every rider’s GPS ping, crores per day. Support agents need “the last 50 positions of rider R77”. Watch a natural-looking table fail, and a query-first design succeed.

Catching a cashback fraud ring with a graph

A promotion gives ₹500 to every new user. Fraudsters create many accounts. Each account looks innocent alone; the fraud only shows up in the connections between them.

NoSQL in Indian products you use

Four NoSQL models in plain Python

Each NoSQL family is simulated with ordinary Python data structures: a dictionary for key-value, nested dictionaries for documents, partitions of sorted rows for wide-column, and an adjacency list for a graph. Run it, then try the challenges: cache-aside, denormalisation, choosing a partition key, quorum consistency and graph hops.

# 1. Key-value: get/set by key only
kv = {"session:7f3a": "U9", "cart:U9": {"thali": 1, "lassi": 2}}
print("KV       :", kv["cart:U9"])

# 2. Document: nested, flexible records, queried by fields
kitchens = [{"_id": "K11", "city": "Pune", "veg": True, "menu": ["thali", "lassi"]},
            {"_id": "K12", "city": "Pune", "veg": False, "menu": ["biryani"], "halal": True}]
print("Document :", [k["_id"] for k in kitchens if k["city"] == "Pune" and k["veg"]])

# 3. Wide-column: partition by key, rows sorted inside the partition
pings = {"R77": [(1, "18.50,73.85"), (2, "18.51,73.86"), (3, "18.52,73.86")],
         "R12": [(1, "19.07,72.87")]}
print("Wide-col :", "latest R77 ->", pings["R77"][-1])

# 4. Graph: nodes and relationships
friends = {"Riya": ["Kabir", "Meera"], "Kabir": ["Arjun"], "Meera": ["Zoya"]}
two_hops = [ff for f in friends["Riya"] for ff in friends.get(f, [])]
print("Graph    :", "friends of Riya's friends ->", two_hops)

• print("KV :", kv["cart:U9"])Key-value stores answer one question very fast: “what is stored under this key?”
• {"_id": "K12", "city": "Pune", "veg": False, "menu": ["biryani"], "halal": True}]Documents can have different fields. K12 has halal; K11 doesn't, and nothing breaks.
• print("Wide-col :", "latest R77 ->", pings["R77"][-1])All of R77's rows sit in one partition, already sorted, so “latest” is the last row.
• two_hops = [ff for f in friends["Riya"] for ff in friends.get(f, [])]Following relationships hop by hop is what graph databases are built for.

Run it on your PC: key-value, documents and a graph

Python's standard library includes a real on-disk key-value store (dbm) and SQLite with JSON functions. This program uses them to store carts by key with an expiry, query JSON kitchen documents by their fields, and find a fraud ring in a small graph. Save it as nosql_lab.py in an empty folder and run it.

"""Three NoSQL ideas using only Python's standard library:
1. a real key-value store on disk (dbm) used as a shopping-cart store
2. JSON documents queried with SQLite's JSON functions (a document-store style)
3. a small graph searched for a fraud ring"""
import dbm, json, sqlite3
from collections import deque

# ---- 1. Key-value: carts stored by key, with an expiry time
with dbm.open("carts_db", "c") as kv:
    now = 1_000_000                       # fixed clock so the output is repeatable
    kv["cart:U9"] = json.dumps({"items": {"thali": 1, "lassi": 2}, "expires": now + 1800})
    kv["cart:U12"] = json.dumps({"items": {"dosa": 1}, "expires": now - 5})   # already expired
    for key in [b"cart:U9", b"cart:U12", b"cart:U99"]:
        raw = kv.get(key)
        if raw is None:
            print(key.decode(), "-> not found")
            continue
        cart = json.loads(raw)
        state = "expired" if cart["expires"] < now else "live"
        print(key.decode(), "->", cart["items"], f"({state})")

# ---- 2. Documents: each kitchen is one JSON document with a nested menu
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE kitchens (doc TEXT)")
docs = [
    {"_id": "K11", "name": "Annapurna", "city": "Pune", "veg": True,
     "menu": [{"dish": "thali", "price": 220}, {"dish": "lassi", "price": 100}]},
    {"_id": "K12", "name": "Spice Box", "city": "Pune", "veg": False,
     "menu": [{"dish": "biryani", "price": 280}], "delivery_partner": "own riders"},
    {"_id": "K13", "name": "Udupi Corner", "city": "Mumbai", "veg": True,
     "menu": [{"dish": "dosa", "price": 150}, {"dish": "filter coffee", "price": 60}]},
]
db.executemany("INSERT INTO kitchens VALUES (?)", [(json.dumps(d),) for d in docs])
rows = db.execute("""
    SELECT json_extract(doc, '$.name'), json_array_length(doc, '$.menu')
    FROM kitchens
    WHERE json_extract(doc, '$.city') = 'Pune' AND json_extract(doc, '$.veg') = 1
""").fetchall()
print("\nVeg kitchens in Pune (name, dishes on menu):", rows)
extra = db.execute("SELECT json_extract(doc, '$.name') FROM kitchens WHERE json_extract(doc, '$.delivery_partner') IS NOT NULL").fetchall()
print("Documents with an extra field nobody planned for:", extra)

# ---- 3. Graph: accounts linked by shared phones and UPI handles
edges = [("A1", "phone:98xx01"), ("A2", "phone:98xx01"), ("A2", "upi:x@ok"), ("A3", "upi:x@ok"),
         ("A3", "device:D7"), ("A4", "device:D7"), ("A5", "phone:77xx22")]
graph = {}
for a, b in edges:
    graph.setdefault(a, set()).add(b)
    graph.setdefault(b, set()).add(a)
start, seen, queue = "A1", {"A1"}, deque(["A1"])
while queue:
    node = queue.popleft()
    for nxt in graph[node]:
        if nxt not in seen:
            seen.add(nxt)
            queue.append(nxt)
ring = sorted(n for n in seen if n.startswith("A"))
print("\nAccounts connected to A1 through shared phones, UPI ids or devices:", ring)

# In a terminal:  python nosql_lab.py

Cache-aside, one line at a time

Three requests arrive for kitchen pages. Watch the first request for each kitchen miss the cache and read the database, and the repeat request hit the cache.

Worked examples: real problems, solved and traced in memory

Four NoSQL design tasks, from choosing a family to finding a ring in a graph. Each runs live and is traced.

payments, user sessions, product catalogue, chat history, friend suggestions

payments            -> relational (SQL)
user sessions       -> key-value
product catalogue   -> document
chat history        -> wide-column
friend suggestions  -> graph

menu (40 items, read together), reviews (grows forever), addresses (3, read together), orders (grows forever)

menu     : embed (small, read with parent)
reviews  : reference (large or read separately)
addresses: embed (small, read with parent)
orders   : reference (large or read separately)

3 groups with 1,200, 90 and 30 messages over 6 months

group       :  3 partitions, largest 1200 messages
group+month : 18 partitions, largest 200 messages

5 accounts; pairs that share a phone or device

Rings: [['A1', 'A2'], ['A3', 'A4', 'A5']]

Check yourself

Quick revision

Interview and viva questions

Practice problems

NoSQL modelling and access-pattern exercises. The last two are business tasks.

keys cart:U9, cart:U99

cart:U9 -> {'thali': 1}
cart:U99 -> not found

kv = {"cart:U9": {"thali": 1}}
for k in ["cart:U9", "cart:U99"]:
    print(k, "->", kv.get(k, "not found"))

3 documents

['Annapurna', 'Spice Box']

docs = [{"name": "Annapurna", "city": "Pune"}, {"name": "Udupi", "city": "Mumbai"}, {"name": "Spice Box", "city": "Pune"}]
print([d["name"] for d in docs if d["city"] == "Pune"])

otp1 exp 90, otp2 exp 120, cart exp 300

['otp2', 'cart']

expiry = {"otp1": 90, "otp2": 120, "cart": 300}
now = 100
print([k for k, exp in expiry.items() if exp > now])

R7: t=1,2,3,4

[(4, 'D'), (3, 'C')]

pings = {"R7": [(1, "A"), (2, "B"), (3, "C"), (4, "D")], "R9": [(1, "X")]}
print(sorted(pings["R7"], reverse=True)[:2])

small friend graph

['Arjun', 'Zoya']

g = {"Riya": ["Kabir", "Meera"], "Kabir": ["Arjun", "Meera"], "Meera": ["Zoya"]}
direct = set(g["Riya"])
two = {ff for f in direct for ff in g.get(f, [])} - direct - {"Riya"}
print(sorted(two))

3,000 vs 2,00,000

Minimum hit ratio: 98.5%

db_max, traffic = 3000, 200000
needed = 1 - db_max / traffic
print(f"Minimum hit ratio: {needed * 100:.1f}%")

N = 5

15 valid pairs
smallest R for W=3: 3

N = 5
pairs = [(w, r) for w in range(1, 6) for r in range(1, 6) if r + w > N]
print(len(pairs), "valid pairs")
print("smallest R for W=3:", min(r for w, r in pairs if w == 3))

5 crore, 30%, 2 KB, +30%

Memory needed: 37 GB

users = 5_00_00_000
active = users * 0.30
gb = active * 2 / 1024 / 1024 * 1.3
print(f"Memory needed: {gb:.0f} GB")

1 crore reads, 95% hits

Without cache: Rs 20000
With cache   : Rs 1950

reads = 1_00_00_000
no_cache = reads * 0.002
with_cache = reads * 0.95 * 0.0001 + reads * 0.05 * 0.002
print("Without cache: Rs", round(no_cache))
print("With cache   : Rs", round(with_cache))

Real-world scenario questions (asked in company interviews)

NoSQL design questions from real interviews and projects. Decide your answer first, then open the model answer.