← CatZep/Engineering Guides/System Design/Design Scenarios

Scaling a System from 10× to 100×

A system design interview guide to a repeatable method for growing a system by orders of magnitude — measure, find what breaks first, fix it with the cheapest change that works, and repeat — without rewriting everything at once.

“How would you scale this to 10×? To 100×?” is less a question about any one technique than about whether you have a method. The wrong answer is a shopping list — “add caching, sharding, and a queue” — recited without knowing which bottleneck each one relieves. The right answer is a loop: measure the system, find the single resource that saturates first, apply the least invasive fix that removes it, and repeat, because scaling never fails everywhere at once. It fails at one place — the database’s write capacity, a connection pool, a single hot key — and that place moves each time you grow. This guide lays out the loop and then walks the ladder of fixes in the order you actually reach for them, noting what tends to break first at each order of magnitude.

Concepts to understand first

Each definition includes a concrete example and the condition under which the mechanism is useful. You can read this guide without opening another guide.

1. The Method & Scope

Scaling is an iterative process, not a one-shot design. The method is the answer; the techniques are just its steps.

1. Measure. Instrument first. You cannot scale what you cannot see — find the resource that saturates (CPU, memory, disk I/O, network, database, locks) using latency percentiles and utilization, not guesses.
2. Find the bottleneck. At any given load, usually one resource saturates first (CPU, DB connections, disk, a hot key). Fixing anything else buys little. Once you fix it, the next bottleneck appears, so measure again after every change.
3. Apply the cheapest fix that works. Prefer the least invasive change — a bigger box, a cache, a read replica — over the most invasive — sharding, a rewrite. YAGNI: do not shard for a load you do not have.
4. Repeat. The fix moves the bottleneck somewhere else. Re-measure and go again.

Scope. The goal is to grow throughput by 10× then 100× while holding latency and error rate within SLO and keeping cost sane. Correctness and the multi-region story are assumed handled elsewhere; here it is about capacity.

2. Back-of-the-Envelope

Numbers tell you which order of magnitude breaks which resource, so you can predict the next bottleneck instead of discovering it in production.

The pattern is consistent: reads scale out cheaply with caching and replicas, but writes are the wall. Somewhere around 100× a single write primary cannot keep up, and that is when the expensive fix — sharding — finally earns its cost.

Worked example: sizing the 100× system

Assume 100,000 requests per second, 90% reads, a 90% cache hit rate, a DB node that handles about 5,000 reads/s or 5,000 writes/s, queries that take 5 ms, and app boxes that max out at 1,500 rps. These are round, made-up but realistic numbers. The method is the point.

Two lessons fall out. First, the cache did most of the work: without it the DB would see 90K reads/s, or 18 replicas. Second, only about 95 connections are busy at once, but naive pools would open 2,240. A shared pooler such as PgBouncer closes that gap. Note that replicas do not help writes at all. Every replica still applies every write, so write load is the reason to shard.

import math

rps, read_frac, hit = 100_000, 0.90, 0.90
reads, writes = rps * read_frac, rps * (1 - read_frac)
db_reads = reads * (1 - hit)                       # cache misses reach the DB

replicas = math.ceil(db_reads / 5_000) + 1         # 5K reads/s each, +1 spare
shards = math.ceil(writes / (5_000 * 0.5))         # 5K writes/s, run at 50%
boxes = math.ceil(rps / (1_500 * 0.6))             # 1.5K rps max, run at 60%

# Little's law: connections busy = query rate x query time
db_qps = db_reads + writes
conns_needed = db_qps * 0.005                      # 5 ms per query

print(f"DB reads/s {db_reads:,.0f}  replicas {replicas}  shards {shards}")
print(f"app boxes {boxes}  naive pool conns {boxes * 20:,}")
print(f"DB connections actually busy {conns_needed:.0f}")

3. The Scaling Ladder

The fixes come in a natural order, cheapest and least invasive first. You climb only as far as the load forces you — most systems never reach the top rungs.

• Vertical scaling — a bigger box — is the first and simplest move. It buys time with zero architectural change, but hits a ceiling (the biggest machine) and a price cliff, so it is a bridge, not a destination.
• Statelessness is the pivot that unlocks everything after it. Once the app holds no per-user state locally — sessions in a shared cache, files in object storage — any instance can serve any request, so you can add and remove instances freely.
• Horizontal scaling puts many stateless instances behind a load balancer. This is where growth becomes elastic: add boxes to add capacity.

Everything below the app tier — caching, replicas, sharding, async — addresses the data layer, which is where scaling gets genuinely hard, and which the rest of this guide covers.

4. A Scaled Architecture

Assembled, the rungs form a familiar layered architecture. Each layer absorbs load so the layer below it sees less.

• CDN / edge. Serves static assets and cacheable responses from points of presence near the user, taking the largest slice of traffic off your origin entirely.
• Load balancer. Spreads dynamic traffic across app instances and routes around unhealthy ones.
• Stateless app tier. Scales horizontally; because it holds no local state, autoscaling can add or drop instances with the load.
• Cache. A distributed cache (Redis/memcached) absorbs repeated reads before they reach the database.
• Queue. Moves slow or bursty work off the request path so a spike is absorbed, not dropped.
• Database. A write primary with read replicas for read scale, and sharding once the write volume exceeds a single node.

5. Caching Layers

Caching is the highest-leverage move because it removes work rather than adding capacity. It exists at every layer, and the discipline is to serve each request from the cheapest cache that can answer it.

A 90% hit rate cuts backend read load roughly tenfold — often the single biggest lever available. But caching adds its own hard problems, and naming them signals maturity:

• Invalidation. Stale data is the cost of caching; you need a TTL or explicit invalidation strategy, and to decide how much staleness each read tolerates.
• Stampede / thundering herd. When a hot key expires, thousands of requests miss at once and hammer the origin. Mitigate with request coalescing (one fetch fills the cache while others wait), early/jittered expiry, or serving stale while revalidating.
• Hot keys. A single wildly popular key can overload one cache node; replicate or locally cache it.

function read(key):
  v = cache.get(key)
  if v is not None:
    return v                          # fast path: the common case
  with single_flight(key):            # coalesce concurrent misses
    v = cache.get(key)                # re-check after acquiring
    if v is None:
      v = db.read(key)                # only one caller hits the DB
      cache.set(key, v, ttl=jitter()) # jitter avoids synchronized expiry
  return v

6. Replicas, Sharding & Async

Read replicas offload selected reads but still apply primary writes. Fresh/session reads need authority/apply barriers or dependency tokens; a primary flag or sticky session alone is not proof after failover. Measure replica lag and freshness failures.

Before sharding, optimize queries/indexes, batching, contention, data lifecycle, and write amplification. Sharding is a major scale lever, not the only possible throughput improvement. High-cardinality keys do not ensure even access; one hot key remains hot. Account for cross-shard transactions, joins, rebalancing, ownership versions, and consistent-hash migration.

Async work needs durable intent/outbox, idempotent effects, ordering, finite queues, deadlines, and recovery capacity. It smooths bursts and moves latency; it does not make total compute free or allow unlimited outages. Scale from measured bottlenecks rather than treating a fixed ladder as compulsory.

7. Connection Limits & Backpressure

Two failure modes bite specifically as you scale out, and both surprise people because they are not about raw compute.

Connection limits. Databases cap concurrent connections at a few hundred, but a horizontally scaled app tier can have thousands of threads each wanting one. Without a bound, the app exhausts the database’s connections and everything stalls. The fix is a connection pool — often a dedicated pooler — that multiplexes many app threads onto a small, bounded set of database connections. The pool size, not the thread count, is the real concurrency limit against the database.

Backpressure. When a downstream component saturates, an unbounded system keeps accepting work it cannot do, queues grow without limit, latency explodes, and it collapses — often taking callers with it in a cascading failure. A well-behaved system pushes back: bounded queues that reject when full, timeouts so callers do not wait forever, load shedding that drops low-priority work early, and circuit breakers that stop hammering a failing dependency. Failing fast under overload is healthier than failing slowly for everyone.

function handle(req):
  if inflight >= max_inflight:
    return 503                         # shed load early, don't queue forever
  with pool.acquire(timeout=50ms) as conn: # bounded DB concurrency
    return conn.query(req, timeout=200ms)  # deadline, don't wait forever

8. Bottlenecks & Tradeoffs

Scaling is a sequence of trades, and the discipline is to make the cheapest one that removes the current bottleneck — no more.

• Vertical vs horizontal. Vertical is simple but ceilinged and priced steeply; horizontal is elastic but demands statelessness and a load balancer. Climb vertically to buy time, then go horizontal.
• Consistency vs performance. Caches and read replicas trade freshness for throughput. Decide per read how much staleness is acceptable; send the few that must be current to the primary.
• Simplicity vs scale. Sharding and async unlock scale but add operational and cognitive cost forever. Defer them until the load is real (YAGNI); premature sharding is a common self-inflicted wound.
• Cost awareness. Every rung costs money — more boxes, more replicas, cross-node traffic. Scaling is an economics problem: the goal is the SLO at the lowest total cost, and sometimes the right answer is to make the workload cheaper (a better query, less data) rather than to buy more capacity.
• Availability under load. The system must degrade gracefully, not collapse. Backpressure, load shedding, and circuit breakers are what turn an overload into slow-but-alive instead of a cascading outage.

Worked example: more app servers exhaust the database pool

Start with 20 app instances, each configured for 20 DB connections.

Run a small, testable model

Download concepts.py. It uses only the Python 3.10+ standard library. From the directory where you saved it, run:

python3 concepts.py capacity
python3 concepts.py test

The model computes safe usable capacity and procurement; adapt its bound to the measured aggregate connection/work budget.

The program contains deterministic assertions and failure traces. It is a learning model, not a production service: it does not implement network consensus, distributed durability, authentication, or third-party delivery. Read its contracts and try changing an input before extending it.

9. Interview Walkthrough (45 Minutes)

10. Common Mistakes

11. Summary

Scaling from 10× to 100× is a method, not a menu: measure, find the one thing that breaks first, apply the cheapest fix, and repeat. Reads scale out easily; writes are the wall you climb last.

12. Check Yourself

1. Your cache hit rate drops from 95% to 90%. By how much does DB read load change?
   Answer

   The miss rate goes from 5% to 10%, so DB reads double. Small changes in hit rate have large effects near the top.
2. You add two read replicas and write latency does not improve. Why?
   Answer

   Replicas only offload reads. Every replica still applies all writes, and the primary still takes them all. Writes need batching, async, or sharding.
3. An app handles 3,000 rps and each DB query takes 20 ms. About how many DB connections are busy at once?
   Answer

   Little’s law: 3,000 × 0.02 = 60 connections (assuming one query per request).
4. Why must the app tier be stateless before you scale it horizontally?
   Answer

   If a box holds session state, requests must stick to that box. Then you can’t freely add, remove or replace boxes, and losing one loses its users’ state. Move state to a shared store.
5. What is backpressure, and give two ways to apply it?
   Answer

   Pushing back on callers when you are full, instead of accepting work you can’t finish. Examples: bounded queues that reject when full, returning 429/503 early (load shedding), and timeouts.
6. What makes a good shard key?
   Answer

   High cardinality, even spread of load (no hot shard), and alignment with the most common queries so they hit one shard. User id often works; a timestamp usually makes one hot shard.

13. Key Terms

Bottleneck

The resource that saturates first and caps throughput.

Vertical scaling

A bigger machine.

Horizontal scaling

More machines behind a load balancer.

Stateless

A server that keeps no per-user state between requests.

Cache hit rate

Share of reads served from cache.

Read replica

A copy of the DB that serves reads and lags slightly.

Sharding

Splitting data across nodes by a key.

Hot key

One key that gets far more traffic than others.

Connection pooler

Shares few DB connections among many app threads.

Little’s law

Items in flight = arrival rate × time in system.

Backpressure

Backpressure tells an upstream producer that the downstream stage cannot accept more work. Buffering delays that signal; shedding deliberately rejects or drops lower-priority work when a bound is reached. Example and limits above.

14. Further Reading

• Martin Kleppmann, Designing Data-Intensive Applications, chapters 1, 5 (replication) and 6 (partitioning).
• Brendan Gregg, “The USE Method” for finding the bottleneck.
• Site Reliability Engineering, chapter “Handling Overload”. Free at sre.google.
• David Yanacek, “Using load shedding to avoid overload”, Amazon Builders’ Library.
• Alex Xu, System Design Interview, chapter 1, “Scale from zero to millions of users”.