Moving dozens of teams onto a new platform or architecture — phased, backward-compatible, reversible, and coordinated across regions and timezones all the way to zero.
Student edition · understand, apply, verify
Coordinate a migration through tested compatible phases and finish under a safe retirement policy.
What you should be able to do: Separate validation from proof, and treat dual-write, shadow, and rollback as failure-sensitive protocols.
Context-dependent collaboration guidance. Examples/numbers are illustrative; first-person practice drafts are not verified personal history. Respect decision rights, safety/legal obligations, privacy, and the partners' actual capacity. No framework guarantees agreement or delivery.
Concepts explained in this guide
Read these definitions first, or use the linked terms as you go. The examples and limitations are local to this guide; companion pages are optional.
Expand-contract migrations
Change a schema in stages that allow old and new software to coexist: add a compatible representation, deploy readers/writers, backfill and reconcile, switch authority, then remove old fields only after the rollback horizon.
Example. Rename full_name by adding display_name first. Both app versions remain valid until backfill, dual-write consistency, and rollback checks are satisfied.
Boundary. Two application writes are not automatically atomic. Prefer one transaction/change-log authority and detect divergence. Dropping the old field too soon makes code rollback unsafe.
Transactional outbox
A business transaction writes its state change and an event row to the same database. A relay publishes committed event rows later, retrying until the broker durably accepts them.
Example. An order and its OrderCreated outbox row commit together. A relay crash after publish but before marking sent publishes the same event ID again on restart.
Boundary. The relay is at-least-once, not magically exactly-once. Consumers dedup transactionally; preserve per-entity sequence when order matters. CDC is change data capture: reading committed database changes.
Idempotency keys
A client-generated identity for one logical operation. The server binds it to caller scope, endpoint, request fingerprint, state, and result. A retry uses the same key and reuses the original outcome rather than repeating the effect.
Example. Payment p42 times out after the provider charged it. Retrying with p42 lets the provider return the same charge instead of creating a new one.
Boundary. A unique local claim is insufficient if the business effect and saved result can diverge. Keep unknown outcomes pending, use downstream idempotency, and state the retention/retry horizon.
Canaries, shadow traffic, and rollout gates
A canary sends a small real traffic cohort to a new version and compares user-relevant signals with a control. Shadow traffic executes copied inputs but isolates its effects; rollout gates stop promotion when evidence is inadequate or harmful.
Example. Release 1%, then 5%, then 25%, checking error rates, latency, queue lag, and correctness with sufficient observations per cohort.
Boundary. No universal sample fraction or dwell time proves safety. Rare failures need enough samples; cohorts must be comparable. Rollback cannot automatically undo emitted events or incompatible data writes.
Ownership, accountability, and decision rights
Ownership names who drives follow-through; accountability names who answers for outcomes. Decision rights specify who chooses/approves and who is consulted.
Example. A migration has a coordinating owner, service owners, and named release authority.
Boundary. DRI/RACI/DACI are local conventions, not guarantees. Role labels do not replace authority, capacity, and follow-through.
This question tests whether you can execute a large, risky migration that touches many teams without a big-bang outage and without losing the plot halfway through. Cloud moves, storage-engine swaps, and architecture migrations are where a lot of senior candidates reveal they’ve only ever done greenfield. The interviewer wants to see two intertwined skills: the technical de-risking — a phased plan, backward compatibility so there’s no forced flag day, shadow or dual-write to verify before cutover, incremental ramp with rollback at every step — and the program coordination to drive dozens of stakeholders, often split across timezones and with no reporting line to you, all the way to zero on the old system. The hard part of a platform-wide migration is rarely the code; it is getting many teams, each with its own roadmap, to actually do the work and finish it. The answer follows the CARL shape.
The spine: phased plan → backward compatibility → shadow / dual-write to verify → incremental cutover → rollback ready at every step → coordinate stakeholders → track the old path to zero.
What this question is really testing
Retirement follows verified consumer migration, rollback/recovery, retention/legal policy, and safe data cleanup. Deletion is not automatically the right immediate definition of Done.
How to answer
Phase the plan. Break the migration into milestones with owners and exit criteria — foundation, dual-run, ramp, decommission — so it’s a sequence of small verifiable steps, not one terrifying cutover.
Preserve backward compatibility. Build the new path behind the same interface so consumers don’t have to change on your schedule. No forced flag day — teams migrate when ready, within a deadline.
Shadow/load comparison validates selected cases; isolate side effects/resources/privacy and use invariant/compatibility reasoning plus fault tests. It is not a universal correctness proof.
Cut over incrementally with rollback. Ramp traffic 1% → 10% → 50% → 100% behind a flag, with a fast rollback at every step and clear health metrics gating each increase. Reversibility is what makes the migration safe to run during business hours.
Map dependencies and reduce per-team effort. Sequence the rollout off a dependency graph — foundational libraries first, then high-volume consumers, then the long tail — and invest your own team’s effort in a landing zone (codemods, migration scripts, a golden-path guide) so each team’s work is hours, not weeks. The cheaper you make migrating, the less you have to push.
Coordinate across timezones, async-first. When teams are split across regions, run written decisions of record, a regular written status post, and rotating office-hours instead of one meeting that’s midnight for a third of the teams — so no decision is trapped in one region’s working hours and any team can self-serve answers.
Coordinate stakeholders and track to zero. One tracker/scorecard with per-team state and a live percentage to 100%, a named DRI per workstream, and an exec sponsor whose borrowed authority moves teams you don’t manage. And finish it — drive the long tail off the old system and decommission it, because a migration that stalls at 90% leaves you paying for two systems and owning double the on-call.
What the interviewer is looking for
Risk decomposed into phases with exit criteria, not a heroic cutover.
Collect representative compatibility, replay, failure, and outcome evidence before cutover; finite testing does not prove all possible production behavior.
Incremental ramp with rollback and health gates — reversibility as a first principle.
Coordination machinery — scorecard, DRIs, sponsor, landing-zone tooling — for many teams over a long horizon.
An async, written default so the program isn’t bottlenecked on any one region or timezone.
The discipline to finish — the last 10% and the decommission, tracked to zero.
A worked example (CARL)
Practice draft: first-person narratives below are not independently verified autobiographical evidence. Use their structure, not their claimed results; replace with your genuine safe-to-share experience.
Context. Our ads events storage sat on an aging storage engine that was hitting scaling and cost limits, and we needed to migrate to a new tiered backend. The catch: more than 20 downstream teams read from this store — spread across three regions with an eight-to-twelve-hour spread, none reporting to me — several with strict freshness SLAs, and a billing pipeline where any data loss or duplication was unacceptable. A big-bang cutover was off the table — a bad migration here would be a company-level incident — and I owned the outcome with no authority over the people who had to do the work.
Actions. I designed it as a phased, reversible program rather than a project with a launch date. Phase one was foundation and compatibility: my team put the new backend behind the exact same read and write interface the 20 teams already used, so no consumer had to change code to benefit — that removed the “forced flag day” that kills these efforts. I also built the dependency map up front, ranking teams by traffic and migrating the shared libraries several teams depended on first, so downstream teams weren’t blocked when their turn came. Phase two was verification without dependence: we dual-wrote every event to both the old and new stores and ran a continuous diff job comparing them, so we caught correctness bugs against real production traffic while the old store was still the source of truth and nothing was at risk. We ran dual-write for weeks until the diff rate was effectively zero, including through peak load and a few gnarly edge cases the diff surfaced that we’d never have found in a test environment. Phase three was incremental cutover of reads: I moved read traffic behind a flag and ramped 1% → 10% → 50% → 100%, with each step gated on freshness and error-rate health metrics and a one-command rollback if anything regressed — and we did roll back once at 10% when a latency spike appeared, fixed it, and re-ramped. For the billing pipeline specifically, I kept it on the old store until every other consumer was cut over and stable, because its risk tolerance was lowest. The coordination was as much of the work as the engineering. The single highest-leverage move was a landing zone: a codemod and a golden-path guide that turned each team’s cutover from a multi-week task into a day or two, so most teams migrated with no direct help from us. On top of that I ran a scorecard — one row per team showing status and the live percentage of traffic still on the old store, visible to every team and the exec sponsor, because a public number is leverage a private ask never has — a named DRI per workstream, and a coordination model built for the timezone spread: a weekly written status post as the decision of record, an FAQ that grew every time someone hit a snag, and rotating office-hours in two timezone-friendly slots instead of one meeting that was midnight for a third of the teams. My sponsor set a company-visible target date and a dated freeze so the old path grew progressively costlier than the new one. The last 10% was the hardest — a few low-priority teams that kept deprioritizing — so I embedded engineers to concierge-migrate the genuinely blocked ones and used a short, factual escalation through the sponsor for the rest, converting “someday” into “this sprint.”
Results. We migrated all 20+ teams across the three regions with zero data-loss incidents and no consumer-facing SEV, decommissioned the old storage engine within the target half, and cut storage cost for the tier by roughly 35% once the old system was fully retired. The codemod alone accounted for most teams migrating with no direct help from mine, which is the only reason a group my size could drive that many teams at all. Because we never had a forced flag day and could roll back at every step, the whole migration ran during normal business hours.
Learnings. The two moves that made it safe were backward compatibility (no consumer had to move on my clock) and dual-write-plus-diff (correctness proven on real traffic before anyone depended on it). The move that made it finish was the landing zone: every hour spent making migration cheaper bought more adoption than any hour spent pushing teams to spend their own. And the migration wasn’t “done” at 90% — the discipline to drive the long tail to zero and actually decommission the old system is what turned it from a science project into realized cost savings.
Common follow-ups
How do you handle the last teams that keep deprioritizing the migration?
How to answer
Concierge the migration. Do the work for them so it’s a review, not a project on their backlog.
Set a hard decommission date. A dated shutoff of the old system converts “later” into a real deadline.
Make the cost of the old path visible. Show leadership the double on-call and dollar cost of keeping it alive.
Escalate the true holdouts. If a team won’t move despite an easy path, that’s a priority call for shared leadership.
How do you verify data correctness during the migration?
How to answer
Dual-write and diff. Write to both systems and continuously compare outputs on real traffic before cutover.
Shadow reads. Serve from the old path but replay against the new one and compare, so users are never at risk during validation.
Gate ramp on the diff rate. Don’t increase traffic until the discrepancy rate is at or near zero, including at peak.
Reconcile the edge cases. Investigate every diff — the last few percent are usually the real bugs.
Mid-migration you find the new system has a flaw. Do you roll back or push through?
How to answer
Roll back if users are at risk. Reversibility exists precisely for this — use it without ego.
Assess blast radius and reversibility. A two-way-door flaw you fix forward; a one-way-door risk you stop.
Fix, re-verify, re-ramp. Return to dual-write/diff to confirm the fix before resuming the ramp.
Communicate the pause. Tell stakeholders why you paused — a controlled hold builds more trust than a hidden push-through.
How do you run this when the teams are split across timezones?
How to answer
Async-first, written by default. Every decision lands in a doc or a status post so no region is excluded for being asleep, and any team can self-serve answers in its own working hours.
A scorecard, not a standup. A live per-team dashboard replaces the status meeting — anyone in any timezone reads progress without a sync, and you spend sync time only on blockers.
Rotate the sync burden fairly. Office-hours in two timezone-friendly slots instead of one meeting that’s midnight for a third of the teams.
Make the default do the work. New services born on the new path and a landing zone that self-serves mean teams unblock themselves rather than waiting on a handoff across the clock.
How do you drive this across dozens of teams you don’t manage?
How to answer
Make migrating cheap first. Invest your own team’s effort in codemods, scripts, and a golden path so each team’s work collapses to hours — the strongest lever there is.
Make status legible. A public per-team scorecard with a percentage to 100% changes behavior a private ask never can.
Borrow authority from a sponsor. A shared goal and an exec who has publicly backed the date give you the leverage your org chart doesn’t.
Shift the incentive. Make the new path the default and the old path progressively costlier — deprecation warnings, then a dated freeze, then removal.
Evidence to collect with your approved tools
A decision/review record with the problem, constraints, options, and your contribution.
Relevant tests, dashboards, or measurements with definitions, denominators, dates, and uncertainty.
Incident or delivery records showing actual outcomes and unresolved limitations.
Stakeholder agreements and follow-up actions with owners and review dates.
Only authorized, minimized, safe-to-share evidence. Do not copy confidential personnel notes into a public story.
Worked example: dual writes diverge before cutover
Use the following scenario to practice making goals, uncertainty, responsibility, and evidence explicit.
Sequence for this example. The table explains the evidence and decision at each step.
Step
Evidence, decision, and expected result
Choose an authority
Two separate writes can half-fail. Use a local transaction where possible or a durable authoritative change stream/outbox and reconciliation.
Validate an isolated shadow
Prevent real duplicated side effects and assess resource/privacy impact. Matching sampled outputs does not prove every case correct.
Cut over by supported gates
Use enough representative evidence, data consistency checks, client compatibility, and a tested rollback horizon—not fixed percentages alone.
Retire safely
Verify consumers, backups/retention, legal requirements, recovery, and data ownership before deletion. Keep justified exceptions with an owner and review date.
Check your understanding: Does a flag make an incompatible data migration reversible?
No. A traffic flag does not undo changed schemas, deleted data, or external effects.
Practice with your own situation
Choose a real cross-team decision. Record the shared problem, each party's constraints, options, assumptions, authority, agreed actions, owner/date, observed outcome, and reconsideration trigger. Use authorized evidence and distinguish contribution from attribution.
Use genuine, safe-to-share evidence. Example numbers here illustrate reasoning; they are not claims about your work or a guaranteed outcome.
