A system design interview guide to shipping a major change to a critical core platform without a single dropped request — and being able to undo it in seconds when the new version misbehaves.
Student edition · start with the guarantee
Change code and data in compatible stages while preserving a real rollback horizon and measuring user impact.
Correctness contract: Old/new readers and writers coexist under explicit schema/protocol compatibility. Publication, migration, and rollback are gated by sufficient evidence; irreversible effects remain accounted for.
Basic HTTP, databases, and programming are assumed. Specialized terms are explained below. Named products motivate the problem; this is an educational design, not a claim about their current internals. Capacity numbers are illustrative decimal payload estimates, not benchmarks. Add indexes, protocol overhead, replicas, headroom, and failure capacity before provisioning.
The goal is a rollout that preserves the defined availability and correctness objectives, not an absolute guarantee against every fault. Compatible versions, progressive exposure, enough surviving capacity, objective gates, connection draining, and a data-safe rollback horizon make the goal testable.
Intuition first
Think of changing the tires on a moving bus. You cannot stop the bus, so you never remove a wheel until a new one is already carrying weight. Zero-downtime deploys follow the same rule. The old version keeps serving until the new one has proven itself on real traffic, and there is always a way back to a known good state.
Each definition includes a concrete example and the condition under which the mechanism is useful. You can read this guide without opening another guide.
Expand-contract migrations
Change a schema in stages that allow old and new software to coexist: add a compatible representation, deploy readers/writers, backfill and reconcile, switch authority, then remove old fields only after the rollback horizon.
Example. Rename full_name by adding display_name first. Both app versions remain valid until backfill, dual-write consistency, and rollback checks are satisfied.
Limit. Two application writes are not automatically atomic. Prefer one transaction/change-log authority and detect divergence. Dropping the old field too soon makes code rollback unsafe.
Canaries, shadow traffic, and rollout gates
A canary sends a small real traffic cohort to a new version and compares user-relevant signals with a control. Shadow traffic executes copied inputs but isolates its effects; rollout gates stop promotion when evidence is inadequate or harmful.
Example. Release 1%, then 5%, then 25%, checking error rates, latency, queue lag, and correctness with sufficient observations per cohort.
Limit. No universal sample fraction or dwell time proves safety. Rare failures need enough samples; cohorts must be comparable. Rollback cannot automatically undo emitted events or incompatible data writes.
Optimistic concurrency and compare-and-set
Update state only if the expected version or status still matches, in the same atomic operation that changes it. A failed comparison means a conflicting operation won; reread or return a conflict.
Example. UPDATE seats SET status='held', version=version+1 WHERE id=? AND status='available' allows only one concurrent claimant to update the row.
Limit. Application-level GET then unconditional SET is not equivalent. Include acquisition identity to reject stale callbacks, and couple related invariants in a transaction.
Transactional outbox
A business transaction writes its state change and an event row to the same database. A relay publishes committed event rows later, retrying until the broker durably accepts them.
Example. An order and its OrderCreated outbox row commit together. A relay crash after publish but before marking sent publishes the same event ID again on restart.
Limit. The relay is at-least-once, not magically exactly-once. Consumers dedup transactionally; preserve per-entity sequence when order matters. CDC is change data capture: reading committed database changes.
SLIs, SLOs, and error budgets
An SLI measures a user-relevant outcome; an SLO sets its target over a window. For a success-ratio SLO, the allowed bad fraction is 1−SLO. An SLA is a contractual commitment that may use different definitions.
Example. 99.9% successful requests over 1 billion requests allows 1 million bad requests. A time-based 30-day availability target instead allows 43.2 unavailable minutes.
Limit. Request and time budgets are not interchangeable under variable traffic or partial failure. SLOs are not universally stricter than SLAs; choose both from their definitions and user needs.
Failure domains and redundant capacity
A failure domain is a set of components vulnerable to one shared event: process, host, rack, zone, or region. Redundancy must separate domains and leave enough surviving capacity to carry peak demand.
Example. Five replicas placed 2/2/1 across three regions leave at least three after any one region fails. A 3/1/1 layout can lose its quorum with the largest region.
Limit. Independent-failure probability formulas are illustrative assumptions. Shared code/config, dependencies, routing, and control planes can fail all replicas together. N+1 machines does not imply tolerance of a whole zone.
1. Requirements & Scope
Start by stating what “zero downtime” actually means for this platform, because it shapes every later choice. Frame the requirements crisply and confirm them with the interviewer.
Zero downtime. No window where the service is unavailable. In practice that means requests in flight during a deploy complete successfully, and new requests always land on a healthy instance — the availability SLO holds through the rollout, not just around it.
Fast, safe rollback. When the new version regresses, recovery must be measured in seconds to a few minutes, and must itself be low-risk — a rollback that requires a fresh build or a risky reverse-migration is not a rollback you can trust at 3 a.m.
No data loss or corruption. Acknowledged writes survive the transition. Migrations never leave data in a shape that either the old or the new code cannot read.
No broken clients. Callers — other services, mobile apps, SDKs you cannot force to upgrade — keep working across the change. For a core platform this is often the hardest constraint, because you do not control the clients and cannot deploy them atomically with the server.
Observable and reversible. Every stage emits the signals needed to decide go/no-go, and every stage can be undone.
Interview tip: Say explicitly that zero-downtime deployment is really a compatibility problem, not a tooling problem. Orchestrators, canaries, and flags are the machinery, but the thing that guarantees no downtime is that at every instant the versions and data shapes in production can coexist. Lead with that and the rest of the answer falls out of it.
Questions to ask first
Is the service stateless, or does it own data (a database, a cache, open connections)?
Does this release change a schema, an API or a message format that other services read?
How fast must we be able to undo it: seconds, or is a 20-minute redeploy acceptable?
How much traffic is there? (It decides how long a canary must bake to see a signal.)
Are there clients we do not control, such as mobile apps that stay on old versions for months?
2. Compatibility Is the Foundation
During any rollout there is a window where old and new code run side by side and talk to the same data and the same clients. Zero downtime is only possible if both versions can operate correctly during that overlap. That gives one non-negotiable rule: every change must be backward and forward compatible with the versions it will coexist with.
Backward compatible. New code can read data written by, and serve requests from, the old code. This is what lets you roll the new version out gradually.
Forward compatible. Old code can tolerate data written by, and requests shaped by, the new code — ignoring fields it does not understand rather than crashing. This is what lets you roll back to the old code after the new one has already written new-shaped data.
The cardinal sin, and the thing interviewers probe for, is combining a breaking change with a deploy in a single step. If you rename a column and ship the code that depends on the new name at the same moment, there is an instant where half the fleet expects the old name and half expects the new one, and neither the forward nor the backward direction holds. You have manufactured a window with no safe state and no clean rollback. The rule is absolute: never make a breaking change and a deploy in the same step. Split every breaking change into a sequence of individually compatible steps — which is exactly what the expand/contract pattern does. This is the deployment-time face of the same discipline covered in Reviewing API, Schema & Query Compatibility.
3. The Expand / Contract Pattern
Expand adds compatible representation; migrate moves writers/readers/data under a clear authority; contract removes old state only after the rollback horizon. Additive DDL can still lock/rewrite a live table depending on engine/version/defaults, so test the actual migration plan.
Dual-write in one transaction where fields share a database; otherwise use one authoritative change stream and reconciliation. Backfills use bounded resumable batches and version/snapshot rules so stale copies cannot overwrite new authoritative edits. A momentary parity check is not enough while old-only writers still run.
Keep the old representation current during the supported rollback period. If rollback reintroduces old-only writers, catch up/reconcile the new column before the next read cutover. Stop old writers/readers/jobs, verify parity under that boundary, then contract. The destructive step is not reversible without retained data/backups. 'Deploy readers before writers' is a compatibility strategy, not a universal order for every column migration.
4. Progressive Delivery
Rolling replaces bounded batches and may need surge capacity or enough spare survivor capacity; 'no extra capacity' is not universal if availability must stay constant. Blue-green retains a warm previous fleet but routing convergence/connection draining is not an atomic world-wide instant. Shared data must remain compatible.
Canaries compare representative cohorts with sufficient samples/dwell time and explicit abort conditions. A statistically nonsignificant difference is not proof the new version is no worse; rare failures need larger observations or dedicated tests. Ramp percentages/bake times are examples chosen from risk and traffic, not a universal safe sequence.
At 10k requests/s, a 1% canary over 900s sees 90k requests; a 5% bug creates 4500 failures versus 450,000 at 100% over the same duration. The monthly 99.9% budget is 25.92M under constant traffic. This explains a reduced direct blast radius, but shared state/resource failures can still affect non-canary traffic.
5. Feature Flags & Dark Launches
The single most powerful idea in modern rollout is decoupling deploy from release. A deploy puts new code on the servers; a release turns a new behavior on for users. A feature flag is the switch between them: you ship the new code paths dark (present but off), deploy them normally, and then enable the behavior with a config change you can flip in seconds — no rebuild, no redeploy.
This buys several things at once. Rollback of a feature becomes flipping a flag off, which is faster and far lower-risk than rolling back a binary. You can enable a feature for internal users first, then a cohort, independent of the deploy cadence. And you separate the risk of the deploy (does the new binary run?) from the risk of the release (does the new behavior work?), so you never debug both at once.
Dark launch. Ship and run the new code path in production with its user-visible effect suppressed, to prove it works at real scale before anyone sees it.
Shadow traffic. Mirror a copy of live requests to the new version and compare its responses against the old one — without the shadow’s responses ever reaching users. This validates correctness and load behavior of a risky rewrite against production traffic with zero user-facing risk. Be careful that shadowed requests do not cause side effects (double-writes, duplicate charges); shadow reads, not writes, unless the write path is fully idempotent and sandboxed.
Kill switch. A coarse flag that instantly disables a feature or subsystem under duress. Every risky launch should ship with one.
Interview tip: The line “deploy is not release” signals maturity. It lets you say: the binary rolls out through a canary with health gates, but the feature itself is behind a flag ramped independently — so a bad feature is turned off with a flag flip while the binary stays put, and a bad binary is rolled back while the flag state is preserved. Two independent safety controls, two independent failure modes.
6. Rings, Dogfooding & the Bug Bar
Canary percentages control how much traffic sees the change; deployment rings control who sees it, in order of increasing blast radius and decreasing tolerance for bugs. The change graduates outward through the rings, and each boundary is a gate.
Ring 0 — internal / dogfood. The team and then the whole company use the new version first (“eating your own dog food”). Employees hit real edge cases and report them, and the cost of a bug here is an annoyed colleague, not a customer.
Ring 1 — small external cohort. A small percentage of real users, ideally ones who have opted into early releases or a region you can watch closely.
Ring 2 — general availability. The full population, reached only after the earlier rings held.
What makes rings a discipline rather than a hope is an explicit bug bar: a pre-agreed set of severity thresholds that block promotion to the next ring. For example — any Sev1/Sev2, any regression in a core SLI beyond a set margin, or an error-rate delta above X% halts the rollout automatically; a cosmetic Sev3 does not. Defining the bar before the rollout removes the in-the-moment temptation to wave through a problem because a deadline looms. The bug bar turns “is this good enough to proceed?” from a judgment call under pressure into a rule checked against objective signals.
7. Rollback Strategy
Rollback needs a compatible old version, sufficient warm capacity, routing/connection handling, and data that the old code can still interpret. It does not undo already emitted events, external payments, irreversible migrations, or corrupted data. Recovery can require reconciliation/repair even after code rollback.
Keep the old data representation current until its declared rollback horizon expires. Backfills and dual writers need atomic/versioned semantics. Router/flag changes have propagation and long-lived-connection behavior to test; 'seconds' is a measured objective, not a universal guarantee. Separate code recovery from destructive schema/data cleanup and retain an explicit roll-forward plan.
8. Automated Guardrails
For a critical platform, humans watching a dashboard are too slow and too unreliable to be the safety mechanism. The rollout should defend itself with automated gates wired to objective signals.
SLO / error-budget gates. Tie promotion to the same error budget that governs the service (see Designing a Highly Reliable Service). If the canary would burn the budget at an elevated rate, the gate refuses to promote — reliability and rollout share one currency.
Automated canary analysis. A statistical comparison of the canary against a baseline fleet on the same traffic, over the metrics that matter (errors, latency percentiles, saturation, key business KPIs), producing a pass/fail rather than a human eyeballing graphs.
Auto-rollback on regression. When a gate fails, the system shifts traffic back automatically — recovery does not wait on a pager being acknowledged. This is the difference between a two-minute blip and a two-hour outage.
Kill switch. An always-available manual override to disable a feature or halt a rollout instantly, independent of the automated gates, for the cases the metrics do not catch.
Interview tip: When asked “how do you know the canary is bad?”, resist “we watch the dashboards.” The strong answer is an automated gate: define the SLIs and thresholds up front, compare canary vs baseline statistically, and wire failure to an automatic traffic-shift back. The machine should catch and revert the regression before a human is even paged.
9. Stateful vs Stateless & Tradeoffs
Everything above is easiest when the tier being deployed is stateless — any instance can serve any request, so you can freely add, drain, and remove instances and shift traffic at will. Stateful components need extra care, and calling that out separates a real answer from a checklist.
Draining connections. Before an instance exits, stop routing new requests to it (mark it unready) but let in-flight requests finish within a grace period, then shut down. Without draining, every deploy severs live connections and manufactures the very errors you were trying to avoid.
Stateful rollouts preserve quorum and measured survivor capacity, follow supported version-skew/membership rules, and account for leader transfer, connection draining, catch-up, and correlated failures. One-at-a-time replacement alone is not an unconditional availability guarantee.
Version skew & sessions. During any rollout both versions serve simultaneously, so a client can hit v1 then v2 on consecutive requests. Sessions must not be pinned to a version’s in-memory state (keep session state external), and any wire protocol or cache format shared across versions must itself be forward/backward compatible — version-skew handling is compatibility applied to the in-flight request, not just the data at rest.
The tradeoffs to name:
Speed vs safety. More ramp steps, longer bake times, and more rings catch more bugs but slow every release. Match the caution to the blast radius — a core platform earns slow, gated rollouts; a low-risk internal tool does not.
Cost vs rollback speed. Blue-green buys instant rollback at 2× the capacity; canary buys most of the safety for a fraction of the cost but rolls back a hair slower. State the tradeoff and pick per criticality.
Compatibility overhead. Expand/contract turns one change into several deploys and a window of dual-write complexity. That is real engineering cost — and it is the price of never having an unsafe moment. For a critical platform it is almost always worth paying.
Flag debt. Feature flags proliferate; stale flags become a maintenance and correctness hazard. Treat flag cleanup as part of the release, not an afterthought.
Worked example: renaming a live column safely
Rename full_name to display_name without making old code unreadable.
Read top to bottom. The table below explains the state and guarantee at each step.
Step
State, calculation, and reasoning
1. Expand
Add display_name with an online-safe DDL plan. Nullable/additive does not mean the database operation is lock-free or reversible in every engine.
2. Migrate under one authority
Keep full_name authoritative; dual-write atomically or derive via an ordered change log. Backfill in bounded resumable batches with version guards/reconciliation.
3. Switch readers with rollback checks
Verify parity and that old writers are gone. If rolling back to old-only writers, catch up/reconcile display_name before rolling forward again.
4. Contract after the horizon
Drop full_name only after old readers/writers/jobs and supported rollback versions no longer need it. Data deletion is a separate irreversible step, not undone by routing traffic.
Check your understanding: Is blue-green rollback always instant and lossless?
No. Router convergence, connections, warm capacity, shared data compatibility, and already-emitted effects remain. A traffic switch cannot undo incompatible writes.
Run a small, testable model
Download concepts.py. It uses only the Python 3.10+ standard library. From the directory where you saved it, run:
python3 concepts.py versions
python3 concepts.py test
The version model demonstrates compare-and-swap publication; the guide's migration steps add the cross-version writer and rollback conditions.
The program contains deterministic assertions and failure traces. It is a learning model, not a production service: it does not implement network consensus, distributed durability, authentication, or third-party delivery. Read its contracts and try changing an input before extending it.
Blue-green or canary, ramp steps and bake times, rings, feature flags to separate deploy from release.
30–38
Guardrails and rollback
SLIs compared canary vs baseline, automatic traffic shift back, when roll-forward is the only option.
38–45
Stateful tiers and tradeoffs
Connection draining, quorum-aware DB upgrades, cost of double capacity, flag debt.
11. Common Mistakes
Mistake
Why it hurts
Do this instead
Renaming a column or field in one deploy
Old and new instances disagree mid-rollout; errors until it finishes
Expand/contract across several deploys
Reading the new column before the backfill is verified
Stale or missing data, worse after a rollback
Keep one source of truth; flip reads behind a flag
One giant UPDATE backfill
Long locks, replica lag, can’t resume after a failure
Small throttled batches keyed by id; idempotent
Canary judged by a human watching dashboards
Slow, noisy, misses small regressions
Automated canary analysis with rollback wired in
Canary gets different traffic than the baseline
Comparison is meaningless
Route a random, comparable slice; compare against a same-age baseline
Flags that are never removed
Dead code paths and untested combinations pile up
Give every flag an owner and an expiry
12. Summary
Zero-downtime deployment is compatibility plus progressive delivery plus a fast, safe undo. You never run incompatible versions together, you change one variable at a time, you expose the change to production in widening slices gated on objective health, and you keep the previous good state one traffic-flip away.
Concern
Mechanism
How can old and new coexist during a rollout?
Backward + forward compatibility as a hard rule; never a breaking change and a deploy in one step.
How do we make a “breaking” schema/API change safely?
Expand / contract (parallel change): add new, dual-write, backfill, cut over, drop old last.
How do we move traffic without downtime?
Rolling, blue-green, or canary — canary with a 1%→10%→50%→100% ramp for a critical platform.
How do we decide to promote or abort?
Automated health/metric gates comparing canary vs baseline on SLIs and business KPIs.
How do we separate shipping code from releasing behavior?
Feature flags; dark launches and shadow traffic to validate at scale with no user risk.
How do we bound who is exposed and when?
Rings (internal → cohort → GA) with an explicit bug bar blocking promotion.
How do we roll back fast and safely?
Instant traffic shift to the warm old version; roll-forward only when a migration is irreversible.
How do we make rollback trustworthy?
Expand/contract so no dropped data is needed; idempotent, verified, resumable backfills.
The recurring theme: there must never be a moment in production with no safe state. Compatibility guarantees old and new can coexist, expand/contract guarantees every schema change is reversible until the last step, progressive delivery guarantees a bad change hurts a sliver before the whole, and automated guardrails guarantee the system reverts before a human is paged. Zero downtime is not one trick — it is the sum of always keeping a way back.
13. Check Yourself
Why must readers of a new message format be deployed before the writers?
AnswerDuring a rollout both versions run. If a new writer runs before every reader understands the new shape, an old reader receives something it can’t parse. Readers first means every instance can already read whatever any writer produces.
In expand/contract, which step is irreversible, and why does it come last?
AnswerThe contract step (dropping the old column or field). It destroys the data that old code needs, so it runs only after you are sure you will never roll back to code that uses it.
Blue-green vs canary: which needs more spare capacity, and which gives a better signal?
AnswerBlue-green keeps a full second fleet, so it needs about double capacity but switches all at once. A canary needs only a small extra slice and lets you compare new vs old on live traffic before most users see it.
At 50 requests/s, a 1% canary sees how many requests in 10 minutes? Is that enough to spot an error rate change from 0.1% to 0.3%?
Answer0.5 rps × 600 s = 300 requests. At 0.1% you expect 0.3 errors, at 0.3% about 0.9 errors. That is far too few to tell apart. Use a bigger slice, a longer bake, or synthetic traffic.
What does “deploy is not release” mean?
AnswerShipping the binary (deploy) and turning on the behaviour for users (release) are separate steps. Feature flags let you roll out code dark, then ramp the feature on its own schedule and turn it off without a redeploy.
When is roll-forward the only way to recover?
AnswerWhen something irreversible already happened, such as a destructive migration or data written in a shape old code can’t read. Then you ship a fix or flip a kill switch forward. Good design keeps you out of this state.
14. Key Terms
Backward compatible
New code can read data or requests made by old code.
Forward compatible
Old code can safely read data made by new code.
Expand / contract
Add the new shape, migrate, then remove the old one (also called parallel change).
Dual-write
Writing both the old and the new shape during a migration.
Backfill
Filling the new shape for rows written before the change.
Blue-green
Two full fleets; switch traffic from one to the other.
Canary
A small slice of traffic sent to the new version first.
Bake time
How long a stage runs before promotion.
Feature flag
A runtime switch that turns a code path on or off.
Ring
A stage of users (internal, beta, GA) the release moves through.
Connection draining
Letting in-flight requests finish before a server stops.
15. Further Reading
Danilo Sato, “Parallel Change”, martinfowler.com (the expand/contract pattern).