---
name: system-design
description: 'Designs and reviews system architecture: scalability, CAP trade-offs, caching, sharding, queues, load balancing, availability math, capacity estimates. Use for architecture decisions, design docs, choosing between REST/gRPC/GraphQL/queues, or "how should we structure/scale this".'
license: MIT
metadata:
  author: lammanhhoang
  version: "1.0.0"
---

# System Design

Knowledge skill. Rules are numbered with RFC-2119 keywords — cite them in reviews ("violates rule #9").

## When to use

Use for: architecture proposals and reviews, design docs, scaling an existing system, choosing datastores/protocols/queues, capacity planning, availability targets, cache/shard strategy.

Do NOT use for: code-level refactors inside one service, UI design, algorithm puzzles, or writing Terraform/K8s manifests for an already-decided design.

## Method — 4 steps, always in this order

1. **Use cases, constraints, capacity.** Write the top 3 use cases, explicit non-goals, and the numbers: DAU, QPS (avg + peak), read:write ratio, latency SLO, availability target, object sizes, retention, growth. Do back-of-envelope math before drawing anything. For worked examples (Twitter-like, URL shortener), read references/ESTIMATION.md (load whenever you produce QPS/storage/bandwidth numbers).
2. **High-level design.** Boxes and arrows: client → DNS → LB → stateless services → cache → datastore → async workers. Name the source of truth for each datum. Walk every core use case end-to-end through the diagram.
3. **Core components.** Zoom into the 2–3 components the constraints stress: data model + access patterns, API signatures, cache policy, queue topology. Decide with the tables below.
4. **Scale the bottlenecks.** Find the bottleneck the step-1 numbers predict (single writer? fan-out? hot key?), fix with the cheapest adequate tool, repeat. Escalation order: indexes/algorithm → vertical scale → cache → read replicas → CDN → async/queues → sharding → multi-region. For mechanics (LB L4/L7, replication topologies, federation, denormalization, microservice split criteria), read references/PATTERNS.md.

Record every non-obvious decision as an ADR using assets/adr-template.md.

## Rules: capacity and structure

1. You MUST quantify constraints before proposing components. No numbers, no design.
2. You MUST estimate: QPS = daily actions ÷ 86,400 (≈10^5 for napkin math); peak = 2–5× average unless measured; storage = rate × object size × retention. Round to one significant digit.
3. You SHOULD design for 10× today's load and only *plan* (not build) for 100×.
4. Services MUST be stateless; all state lives in DB/cache/queue. Scaling then equals adding instances behind the LB.
5. Every datum MUST have exactly one source of truth. Caches, search indexes, projections, and denormalized copies are rebuildable derivatives, never authoritative.
6. Every network call MUST have a timeout, bounded retries with exponential backoff + jitter, and — if it mutates — an idempotency key. Retrying a non-idempotent write is how you double-charge a customer.

## Latency numbers (after Jeff Dean; SSD/network rows from later community updates) — memorize the ratios

| Operation | Latency |
|---|---|
| L1 cache reference | 0.5 ns |
| Branch mispredict | 5 ns |
| L2 cache reference | 7 ns |
| Mutex lock/unlock | 100 ns |
| Main memory reference | 100 ns |
| Send 2 KB over 10 Gbps network | ~2 µs |
| Compress 1 KB with Snappy | ~10 µs |
| Read 4 KB randomly from SSD | 150 µs |
| Read 1 MB sequentially from memory | 250 µs |
| Round trip within same datacenter | 500 µs |
| Read 1 MB sequentially from SSD | 1 ms |
| Disk seek | 10 ms |
| Read 1 MB sequentially from disk | 30 ms |
| Send packet CA → Netherlands → CA | 150 ms |

Derived rules of thumb: memory ~4 GB/s sequential, SSD ~1 GB/s, spinning disk ~30 MB/s; a datacenter round trip costs 5,000× a memory reference; you get at most ~6 sequential cross-ocean round trips inside a 1 s budget.

7. You MUST sanity-check designs against this table. A design needing 20 sequential intra-DC calls burns ≥10 ms on network alone; one needing chatty cross-region calls is dead on arrival.

## Availability math

| Target | Downtime/year | /month | /week | /day |
|---|---|---|---|---|
| 99.9% ("three 9s") | 8h 45min 57s | 43m 49.7s | 10m 4.8s | 1m 26.4s |
| 99.99% ("four 9s") | 52min 35.7s | 4m 23s | 1m 0.5s | 8.6s |

- Components **in series** multiply: `A_total = A1 × A2`. LB 99.99% → app 99.9% → DB 99.9% gives 0.9999 × 0.999 × 0.999 = **99.79%** (~18h 24m down/year). Chains only get worse.
- Components **in parallel** (redundant, independent failure): `A_total = 1 − (1 − A)^n`. Two 99.9% replicas → 1 − (0.001)² = **99.9999%**.

8. Your availability target MUST NOT exceed your worst hard-dependency chain. Compute the composite; don't promise four 9s on top of a three-9s database.
9. Each additional nine costs roughly 10× the engineering effort. You SHOULD challenge any target above 99.9% with "what does a 10-minute outage actually cost?"

## CAP: CP vs AP

10. During a partition you MUST choose per **dataset**, not per system: return an error (CP) or return possibly-stale data (AP)?

| CP (reject rather than lie) | AP (serve stale rather than fail) |
|---|---|
| Bank ledger / account balance | Social/news feed |
| Inventory decrement, seat booking | Like/view counters |
| Distributed locks, leader election | DNS, service discovery reads |
| Payment idempotency records | Shopping cart (merge on read) |
| Unique username claim | Presence / typing indicators |

CAP only bites during partitions. The everyday trade-off is PACELC: else, latency vs consistency.

**Consistency patterns:** weak (best effort, lost writes acceptable — live video, VoIP, memcached-as-only-store), eventual (async replication, reads stale for ms–seconds — DNS, S3 cross-region, feeds), strong (consensus/sync replication — RDBMS transactions, Spanner-class stores; paid for in latency and availability).

**Availability patterns:** failover — active-passive (heartbeat + promotion; un-replicated writes are lost; fence against split-brain) or active-active (both serve; needs conflict story); replication (topologies in references/PATTERNS.md); graceful degradation (feature-flag off expensive paths, serve from cache when the source of truth is down).

## Caching

Default cache: **Redis**. Escape hatch: Memcached when you need nothing but a flat multi-threaded LRU at extreme throughput.

| Strategy | Mechanics | Wins when | Failure mode |
|---|---|---|---|
| Cache-aside (default) | App reads cache; on miss, reads DB and populates | Read-heavy, tolerates brief staleness | Stale after DB write unless you invalidate; stampede on hot-key expiry |
| Write-through | Write cache; cache writes DB synchronously | Read-after-write consistency on hot data | Write latency ↑; churns cache with never-read data |
| Write-behind | Write cache; async flush to DB | Absorbing write bursts (counters, metrics) | Data LOSS if cache dies before flush; replay complexity |
| Refresh-ahead | Refresh hot keys before TTL expiry | Predictable hot set, tight latency SLO | Wasted refreshes on bad prediction; still needs miss path |

11. Every cache MUST define: TTL (always set one, even 24h, as a backstop), invalidation trigger, stampede defense (jittered TTL + single-flight/request coalescing), eviction policy (LRU default), and cold-start behavior.
12. You SHOULD invalidate by **deleting** the key on write and lazy-reloading, not by writing the new value from app code — two concurrent writers racing to set a cache produce permanent staleness.

**CDN:** default **pull** (CDN fetches from origin on first request, honors TTL — fits long-tail content, zero upload workflow; cost: first-hit latency and origin stampede at mass expiry). Use **push** when content is large, published rarely, and traffic spikes at release (installers, video drops): you upload and you invalidate.

## Sharding

13. You MUST NOT shard until caching, read replicas, and vertical scaling are exhausted. Sharding is a one-way door: cross-shard joins, transactions, and rebalancing become your problem forever.

| Scheme | Wins | Loses |
|---|---|---|
| Hash(key) | Even spread, no planning | No range scans; naive modulo reshuffles ~everything on reshard |
| Range | Range scans cheap (time-series, leaderboards) | Hot tail: newest range takes all writes |
| Directory | Arbitrary placement, easy migration | Extra hop; the directory itself must scale and not die |

Consistent hashing in two sentences: keys and nodes hash onto the same ring, and each key belongs to the first node clockwise, so adding or removing a node remaps only ~1/N of keys instead of nearly all. Give each physical node 100–200 virtual nodes to smooth the load.

**Hot keys** (celebrity problem): split the key with a suffix (`user123#0..15`) and fan-in at read, give hot keys a dedicated cache tier, or replicate them read-only to every shard. Detect with per-key QPS top-k, not averages.

## Queues and async

Default: **Kafka** for event streams / fan-out / replay; **SQS** (or RabbitMQ self-hosted) for plain task queues.

14. Introduce a queue only when at least one holds: (a) work exceeds the caller's latency SLO, (b) load is spiky but consumers can drain steadily, (c) ≥2 independent consumers need the same event, (d) side effects need durable retry (email, webhooks). A queue in a synchronous read path is pure added latency.
15. Backpressure is MANDATORY: bound the queue and shed at admission (429 / drop policy) instead of growing forever. Backlog is time debt — Little's Law: latency = backlog ÷ drain rate; 1M messages draining at 2k/s = 8.3 minutes of delay for every new message. Alert on message **age**, not just depth.
16. Exactly-once delivery across systems does not exist. Reality is at-least-once delivery + idempotent consumers: dedupe on message ID or business idempotency key (unique-constraint upsert or conditional write). Kafka's "exactly-once" covers Kafka-to-Kafka stream processing only. Use the transactional outbox pattern to publish "DB write + event" atomically. Dead-letter after ~5 attempts and monitor DLQ depth.

## Protocol choice

| Criterion | REST/JSON | gRPC | GraphQL | WebSocket |
|---|---|---|---|---|
| Latency/payload | Verbose text | Binary protobuf, HTTP/2 multiplexed | One round trip for composite reads | Lowest per-message after handshake |
| Typing | None native (OpenAPI bolt-on) | Strong, codegen | Strong schema | None — bring your own |
| Browser support | Universal | Needs gRPC-Web proxy | Universal | Universal |
| Streaming | No (SSE bolt-on) | Bidirectional native | Subscriptions (WS underneath) | Bidirectional native |
| Team/ops cost | Lowest; curl-able | Proto toolchain; LB must speak HTTP/2 | Resolver N+1, caching is hard, query cost limits | Connection state, sticky LB, reconnect logic |

17. Defaults: REST for public APIs, gRPC for internal service-to-service, GraphQL only when many client shapes aggregate many backends (mobile BFF), WebSocket only for server-push real-time (chat, live prices). Deviations get an ADR.

## SQL vs NoSQL

18. Default to **PostgreSQL**. It is the right answer until proven otherwise: transactions, ad-hoc queries, joins, and with replicas it comfortably serves ~10k+ simple TPS and several TB.
19. NoSQL is justified when ≥2 of these hold: (a) access is exclusively by known keys — no ad-hoc queries; (b) sustained write throughput or working set beyond a single writer (~>50k writes/s or >5–10 TB hot); (c) data is ephemeral/TTL'd; (d) payload is genuinely schemaless. Then: DynamoDB/Cassandra for write scale, Redis for ephemeral, Elasticsearch strictly as a secondary index (never source of truth).
20. With NoSQL you MUST design tables from the query list, not the entity model: enumerate access patterns first, denormalize to serve each with one lookup, accept no cross-partition joins or transactions.

## What NOT to do — red flags in any design

- Microservices for a team of 3, or before a monolith's seams are known (criteria in references/PATTERNS.md).
- Kafka for 10 QPS. A Postgres table with `SELECT ... FOR UPDATE SKIP LOCKED` is a fine queue at small scale.
- Sharding before replicas + cache (rule #13).
- Two sources of truth for one datum, "kept in sync" (rule #5).
- A cache with no TTL/invalidation/stampede story (rule #11).
- Distributed transactions (2PC) across services — use sagas + outbox instead.
- Multi-region active-active with no conflict-resolution story.
- Latency budgets that ignore the physics table (rule #7).
- Tech chosen from hype or résumés; every non-default choice needs an ADR with rejected alternatives (assets/adr-template.md).