Designing Global Multi-Datacenter Routing & Failover

A system design interview guide to sending every user to the right region, keeping data safe across regions, and surviving the total loss of a datacenter with little or no data loss and a fast recovery.

A service that runs in one datacenter is one power failure, fiber cut, or regional outage away from being down for everyone. Running in several regions fixes that, but it trades one hard problem for two: you must route each user to a healthy nearby region, and you must replicate state between regions so that losing one does not lose data. These two problems pull against each other through the laws of physics — the speed of light makes cross-region coordination slow, and the CAP and PACELC tradeoffs mean you cannot have strong consistency, low latency, and full availability at once. This guide works through both halves: the routing layer (GeoDNS, anycast, and GSLB), the choice between active-active and active-passive, synchronous versus asynchronous replication and what each does to your recovery objectives, and finally how to fail an entire region over without splitting the brain or losing writes.

Concepts to understand first

Each definition includes a concrete example and the condition under which the mechanism is useful. You can read this guide without opening another guide.

RPO and RTO

Recovery Point Objective is the target maximum lost-data window; Recovery Time Objective is the target maximum recovery interval. Measured lag and drill duration are observations used to test those targets, not the targets themselves.

Example. At a constant 5000 acknowledged writes/s, an async standby 2 seconds behind lacks about 10,000 writes. Detecting, fencing, promotion, catch-up, and rerouting contribute separately to recovery time.

Limit. RPO=0 holds only for acknowledged durable commits under a stated failure model and safe recovery. DNS TTL and health checks do not give a universal hard upper bound on RTO.

Failure domains and redundant capacity

A failure domain is a set of components vulnerable to one shared event: process, host, rack, zone, or region. Redundancy must separate domains and leave enough surviving capacity to carry peak demand.

Example. Five replicas placed 2/2/1 across three regions leave at least three after any one region fails. A 3/1/1 layout can lose its quorum with the largest region.

Limit. Independent-failure probability formulas are illustrative assumptions. Shared code/config, dependencies, routing, and control planes can fail all replicas together. N+1 machines does not imply tolerance of a whole zone.

Quorum

A set of replicas whose responses are sufficient for an operation. In a fixed set of N replicas, read size R and write size W overlap whenever R+W>N. A majority is one particular quorum, not the definition of every quorum.

Example. With N=3, writes to {A,B} and reads from {B,C} share B. If the write completed durably and replicas preserve a comparable version, B carries that version forward.

Limit. Overlap alone does not provide linearizability: concurrent or failed writes, sloppy replica sets, and version selection matter. Consensus adds protocol rules to preserve decisions.

Fencing tokens

A strictly increasing grant number attached to operations. The protected resource atomically compares it with its durable highest accepted token and rejects older operations, making superseded clients unable to overwrite newer effects.

Example. A has token 33 and pauses. B acquires 34 and writes. When A sends token 33 later, the resource rejects it because its fence is already 34.

Limit. A new grant must be installed at the resource before it promises exclusion there. Token 33 can still pass until 34 is observed. Equal-token operations also need sequence/idempotency rules if they can arrive out of order.

Replication

Maintaining copies of data on independent storage nodes. Synchronous acknowledgement waits for specified durable copies; asynchronous replication sends changes after the client has already received success.

Example. A primary acknowledges after two durable copies in different zones. A follower that applies updates later can serve an old value even though the write is durable.

Limit. Replica count is not a complete durability or availability guarantee. State the failure domains, acknowledgement policy, promotion rules, repair process, and what happens if all copies fail.

CAP and consistency models

During a network partition, a distributed service cannot offer both linearizable operations and successful completion of every eligible request at nonfailed nodes. CAP consistency is linearizability; it is not durability or 'no bytes lost'.

Example. A minority that cannot contact the authority blocks or rejects a linearizable read/write. A mergeable append log can accept local writes on both sides and preserve both records, while not giving one-copy ordering.

Limit. Causal/session/eventual guarantees are not a universal availability ladder. Sticky routing alone fails after failover unless the new replica has satisfied the client's dependencies.

1. Requirements & Scope

The requirements split cleanly into the two halves — getting users to a region, and keeping data correct across regions — plus the disaster case that ties them together.

2. Back-of-the-Envelope

Two sets of numbers drive the design: the physics of cross-region latency, and the objectives that flow from the business.

The single most important number is the cross-region RTT: ~70 ms. Every design choice below is really a decision about whether to pay that latency on the write path (synchronous, zero data loss) or to avoid it and accept a small data-loss window (asynchronous). Physics, not preference, forces the trade.

3. RPO, RTO & Key Definitions

The vocabulary of disaster recovery is precise, and interviewers expect it. Two numbers frame every failover design.

Quantity	Assumption	Result
Cross-region round trip	US east–west ~3,000 mi = 4,828 km; fiber ~⅔ c ≈ 200,000 km/s gives a 24 ms one-way floor	~30–40 ms one way, ~60–80 ms RTT on real routes
Sync write penalty	every write waits one cross-region RTT	+~70 ms per write — on the hot path
Async replication lag	ship changes continuously	sub-second to seconds of exposure
DNS-based failover	record TTL 30–60s + resolver caching	minutes of stale routing worst case
Anycast reconvergence	BGP withdraws a failed POP	seconds to reroute
Target RPO / RTO	tier-1 service	RPO ~0 (sync) or seconds (async); RTO ~minutes

Term	Meaning	Driven by
RPO (Recovery Point Objective)	How much recent data you can afford to lose — the age of the last safe copy.	Replication mode (sync → 0, async → lag).
RTO (Recovery Time Objective)	How long you can be down — time to detect, promote, and reroute.	Failover automation and routing TTLs.
Active-active	All regions serve live traffic simultaneously.	Multi-master or partitioned ownership.
Active-passive	One region serves; others stand by, ready to promote.	Single primary + hot standbys.
Split brain	Two regions both act as writable primary and diverge.	Prevented by quorum / fencing.
Quorum	A majority must agree before acting, so only one side can win.	Consensus (Raft / Paxos).

The mental model: RPO is a data question answered by replication, and RTO is a time question answered by automation and routing. Sync replication buys RPO = 0 at a latency cost; fast, well-drilled failover buys a low RTO.

4. High-Level Architecture

The shape is a global routing layer on top of two or more self-sufficient regions, with a replication channel between their data stores.

5. Routing: GeoDNS, Anycast, GSLB

There are three common ways to get a user to a region, and mature systems layer them. Each makes a different tradeoff between control, speed of failover, and connection stability.

In practice you combine them: anycast to reach the nearest edge/POP fast, GSLB (health-aware DNS) to choose the healthy origin region, and short DNS TTLs so a policy change propagates quickly. Whatever the mechanism, the routing layer must be driven by real health signals, because routing to a region that is failing is worse than routing nowhere.

6. Replication & Consistency

Synchronous means acknowledgement waits for the configured durable copies/commit rule; asynchronous means local commit can be acknowledged before remote catch-up. RPO=0 requires placement and promotion that preserve every acknowledged commit under the stated failure budget. A bare remote ACK or majority counter without protocol/placement is not sufficient.

Async lag is an observed gap, not automatically a fixed bound. Semi-sync implementations may have timeout fallback, so check actual modes. A consensus majority follows election/log rules, including Raft's current-term commit condition. Partitioned write ownership needs fenced ownership transfer; it does not sidestep failover conflicts merely by choosing one home region initially.

7. Failover & Split-Brain

Timeouts and monitor agreement establish suspicion, not proof a primary died. Obtain safe promotion authority through the storage/consensus protocol. Fence the old primary at the actual effect/resource boundary; a DNS change or cooperative lease revoke alone does not stop delayed writes.

Choose a candidate using committed history and recovery rules, not simply the largest observed log position. An uncommitted divergent suffix can be longer. Synchronous replication protects acknowledged commits only if the configured durable placement and failover protocol recover all of them under the fault model. A candidate can catch up from surviving quorum members; it need not itself have acknowledged every original commit.

Async failover may lose acknowledged data not available from survivors. Last observed primary position and sampled lag are incomplete evidence; distinguish bytes/log entries from seconds and expose uncertainty. Reroute with client/session handling, warm the survivor, verify critical read/write journeys, and resync the old region under a new epoch. Drills validate achieved recovery, not an absolute timing guarantee.

8. Worked Example: RTO and RPO Budgets

A sample staged budget of 30s detection+30s confirmation+10s fencing+30s promotion+60s DNS+60s client/resolver allowance totals 220s≈3.7 minutes. It is an illustrative budget, not a hard worst-case bound. Some stages overlap; some clients cache longer or retain old connections.

DNS-based GSLB still depends on resolver behavior; anycast also has route convergence and connection considerations. Measure detection, authority, catch-up, fencing, traffic convergence, and critical-operation recovery independently. Async missing-write count is approximately rate×lag only under steady rate and a trustworthy lag sample; outages/spikes can enlarge it. State RPO/RTO targets and measured drill results separately.

9. Failure Modes

10. Bottlenecks & Tradeoffs

The tensions here are fundamental — they come from the network and from CAP/PACELC, not from implementation.

11. Summary

Global multi-region design is two problems bound by the speed of light: route users to a healthy region, and replicate data so losing a region does not lose the data. Every choice is a placement of the same latency-versus-durability trade.

Worked example: async lag and a safe promotion

Run a small, testable model

Failure	Why it hurts	Defense
Gray failure (slow, not dead)	Health checks pass while users time out	Check real user paths and latency, not just "port open"
Flapping failover	Traffic bounces back and forth, each time risking data	Confirmation window, hysteresis, and a human gate for fail-back
Fencing fails	Old primary keeps writing: split brain	Don't promote until fencing is confirmed; storage rejects old epochs
Survivor region lacks capacity	New region falls over under double load	Plan N+1 capacity; shed low-priority traffic
Control plane lives in the dead region	You cannot run the failover	Run failover tooling and its config store in every region
Clients ignore DNS TTL	Some users stay on the dead region	Anycast, client-side retry to a second endpoint

Concern	Mechanism
How do users reach a region?	GeoDNS for proximity, anycast for fast reroute, GSLB for health-aware steering — layered.
How much data can we lose?	RPO — set by replication: sync/quorum → 0, async → the lag window.
How long can we be down?	RTO — set by detection, automated promotion, and routing TTLs.
All regions live or standby?	Active-active for latency and utilization; active-passive for simplicity and lower cost.
How do we replicate correctly?	Sync/quorum for RPO 0 at a latency cost; async for speed; partition write ownership to avoid conflicts.
How do we avoid two primaries?	Quorum-based promotion plus fencing of the old primary — the cure for split brain.
How do we know it works?	Regular DR drills / game days that actually evacuate a region and measure RPO and RTO.

Step	State, calculation, and reasoning
1. Estimate missing writes	About 10,000 under constant rate/known lag. Bursts, unseen commits, and stale monitoring can make that estimate wrong; record uncertainty.
2. Obtain promotion authority	Monitors suspect unreachability; they cannot prove death. A consensus/ownership protocol and fencing govern promotion, not majority health votes alone.
3. Select and catch up a valid candidate	Use committed-log rules and available durable copies. The most advanced raw log position can include uncommitted divergent entries and is not automatically safe.
4. Reroute and measure recovery	DNS TTL, resolver caches, persistent connections, cache warmup, and client retries affect convergence. RTO is measured by successful critical operations, not just a DNS change.

Download concepts.py. It uses only the Python 3.10+ standard library. From the directory where you saved it, run:

The majority-placement model compares 3/1/1 with 2/2/1 and shows surviving quorum capacity.

The program contains deterministic assertions and failure traces. It is a learning model, not a production service: it does not implement network consensus, distributed durability, authentication, or third-party delivery. Read its contracts and try changing an input before extending it.