A system design interview guide to building a service that stays up when its dependencies, machines, and whole datacenters do not — the toolkit an interviewer expects when the prompt is “design a highly reliable service for X.”
Student edition · start with the guarantee
Define reliability through user-visible outcomes, independent capacity, and bounded request-path behavior.
Correctness contract: SLIs match user journeys; failure assumptions and remaining capacity are explicit. Retries are bounded and effects are safe under uncertain outcomes.
Basic HTTP, databases, and programming are assumed. Specialized terms are explained below. Named products motivate the problem; this is an educational design, not a claim about their current internals. Capacity numbers are illustrative decimal payload estimates, not benchmarks. Add indexes, protocol overhead, replicas, headroom, and failure capacity before provisioning.
Reliability is not a feature you bolt on at the end; it is a set of deliberate choices made at every layer, from how you define “up” to how a single call times out. A highly reliable service assumes that everything fails — processes crash, disks fill, networks partition, dependencies slow to a crawl — and is engineered so that no single failure, and ideally no single class of failure, takes the whole system down. This guide walks the toolkit: first defining reliability precisely with SLIs, SLOs, and an error budget; then the structural defenses of redundancy across failure domains and health-aware load balancing; then the request-path patterns — timeouts, retries, circuit breakers, bulkheads — that keep a slow dependency from becoming an outage; and finally rollout safety and disaster recovery for when a whole region is lost.
Intuition first
Think of a hospital. Machines break, staff get sick, the power fails. A good hospital still treats patients because it has spare beds, backup generators and clear rules for who gets seen first. A reliable service works the same way: it expects parts to fail and plans what happens next, so one broken part never stops the whole thing.
Each definition includes a concrete example and the condition under which the mechanism is useful. You can read this guide without opening another guide.
SLIs, SLOs, and error budgets
An SLI measures a user-relevant outcome; an SLO sets its target over a window. For a success-ratio SLO, the allowed bad fraction is 1−SLO. An SLA is a contractual commitment that may use different definitions.
Example. 99.9% successful requests over 1 billion requests allows 1 million bad requests. A time-based 30-day availability target instead allows 43.2 unavailable minutes.
Limit. Request and time budgets are not interchangeable under variable traffic or partial failure. SLOs are not universally stricter than SLAs; choose both from their definitions and user needs.
Burn-rate alerting
Burn rate is the observed bad-event fraction divided by the allowed bad fraction. Multi-window alerts require sustained long-window burn and a confirming short window so old incidents can clear.
Example. For a 99.9% SLO, 2% errors is 20× burn. In a 30-day window, 14.4× time-normalized burn for one hour consumes 14.4/720=2% of budget.
Limit. That conversion assumes time-based measurement or roughly representative request volume. Low traffic, missing telemetry, and denominator-zero need policy; an alert alone does not guarantee response.
Failure domains and redundant capacity
A failure domain is a set of components vulnerable to one shared event: process, host, rack, zone, or region. Redundancy must separate domains and leave enough surviving capacity to carry peak demand.
Example. Five replicas placed 2/2/1 across three regions leave at least three after any one region fails. A 3/1/1 layout can lose its quorum with the largest region.
Limit. Independent-failure probability formulas are illustrative assumptions. Shared code/config, dependencies, routing, and control planes can fail all replicas together. N+1 machines does not imply tolerance of a whole zone.
Backpressure and load shedding
Backpressure tells an upstream producer that the downstream stage cannot accept more work. Buffering delays that signal; shedding deliberately rejects or drops lower-priority work when a bound is reached.
Example. A sink accepts 100 events/s while input is 150/s. A 1000-event queue fills in 20 seconds. The system must then block, reject, or apply its documented drop policy.
Limit. A finite queue cannot absorb an infinite outage. Prioritized messages still need hard quotas, deadlines, and bounded client queues; otherwise an exemption can exhaust the system.
Idempotency keys
A client-generated identity for one logical operation. The server binds it to caller scope, endpoint, request fingerprint, state, and result. A retry uses the same key and reuses the original outcome rather than repeating the effect.
Example. Payment p42 times out after the provider charged it. Retrying with p42 lets the provider return the same charge instead of creating a new one.
Limit. A unique local claim is insufficient if the business effect and saved result can diverge. Keep unknown outcomes pending, use downstream idempotency, and state the retention/retry horizon.
Wall and monotonic clocks
A wall clock maps to civil/epoch time and can jump because of correction. A monotonic clock measures elapsed time locally without wall-clock jumps, but its origin is not a shared global epoch.
Example. Use an epoch timestamp to label an event and a monotonic deadline to bound how long a process waits for the next usable timestamp.
Limit. Neither establishes causal order across machines. Monotonic readings cannot be compared between hosts or assumed to survive reboot with the same origin.
1. Requirements & Scope
Start by pinning down what reliability means for this specific service, because the answer drives every later choice. Frame it as functional and non-functional requirements.
Functional. The service exposes an API — say a request/response endpoint backed by a datastore. It must accept writes, serve reads, and never corrupt or lose acknowledged data.
Availability. A target like 99.9% (three nines) or 99.99% (four nines). State it explicitly: three nines allows roughly 43 minutes of downtime a month, four nines allows about 4 minutes. The number sets the ambition of the whole design.
Durability. Acknowledged writes must survive machine and disk loss — usually stated as a number of nines of durability, achieved with replication.
Latency. A tail-latency target, for example p99 ≤ 100 ms, because a reliable service is one that is fast and up; slow is a kind of down.
Blast radius & recovery. No single failure domain should take the service down, and after a disaster it must recover within a stated RTO/RPO (covered in section 7).
Interview tip: Do not promise 100% availability — it signals inexperience. Pick a target with the interviewer, translate it into a downtime budget out loud, and let that budget justify how much redundancy and complexity you add. Over-engineering to five nines a service that only needs three is a real design smell.
Questions to ask first
Who calls this service, and what happens to them when it is down? (Checkout down loses money; a reporting page down loses patience.)
What counts as “up”: any 200 response, or a correct response under a latency bound?
Is the traffic read-heavy or write-heavy, and can reads be slightly stale?
How much data can we lose in a disaster (RPO), and how fast must we be back (RTO)?
One region or many? Are there data-residency rules?
2. SLIs, SLOs & the Error Budget
An SLI measures a defined outcome, often successful eligible requests/total eligible requests. An SLO is the target over a named window. An SLA is a contractual commitment whose scope/penalties may differ; it is not universally looser than an SLO.
For a 99.9% request-success SLO over one billion eligible requests, budget is one million bad requests. For a time-based 99.9% 30-day availability target, budget is 43.2 minutes. These are not interchangeable under uneven load or partial failures. Define slow requests, exclusions, probes, no traffic, and missing telemetry deliberately.
Burn rate is observed bad fraction divided by allowed bad fraction. Multi-window alerts and error-budget policies should reflect actual request volume/window and urgency. A 100% target has zero error budget, but it can be a valid safety aspiration for a narrowly defined correctness property; do not conflate that with a practical probabilistic availability SLO.
3. Redundancy & Failure Domains
Separate redundant instances across the failure domains you promise to survive. Preserve enough capacity after the specified domain loss and maintenance case. N+2 spare machines do not automatically mean surviving two availability zones; replica placement, traffic skew, quorum, and safe utilization determine that.
Stateless compute simplifies routing but still depends on shared state and control planes. A synchronous replica protects only the configured durable acknowledgement/recovery model. Shared software/config/DNS/credentials can fail all replicas together. Health checks should distinguish liveness from readiness and verify useful critical journeys without causing synchronized restart storms.
4. Availability Math: Series vs Parallel
Two simple rules let you estimate the availability of a whole design from its parts. Use them out loud in an interview; they turn “we add redundancy” into a number.
Series (A needs B needs C). Every part must be up, so multiply availabilities: A = A1 × A2 × A3. Each extra hard dependency makes the total worse.
Parallel (any one copy will do). The system is down only if every copy is down, so multiply the unavailabilities: A = 1 − (1 − a)n.
Chaining parts in series multiplies their availabilities, so the whole is worse than its weakest part. Putting copies in parallel multiplies their unavailabilities, so two 99% replicas give 99.99%, but only if they fail independently.
Design
Math
Availability
LB → app → DB in series
0.9999 × 0.9995 × 0.9995
99.890%
Two independent app replicas at 99%
1 − 0.01²
99.99%
Three independent app replicas at 99%
1 − 0.01³
99.9999%
Two regions at 99.9% each, active-active
1 − 0.001² (ignores shared causes)
99.9999% on paper
The first row is a warning: three parts that each beat 99.9% still combine to less than 99.9%. That is why a service rarely has more nines than its least reliable hard dependency.
def series(*parts): # all must be up
total = 1.0
for a in parts:
total *= a
return total
def parallel(a, n): # any one of n copies is enough
return 1 - (1 - a) ** n
print(round(series(0.9999, 0.9995, 0.9995), 5)) # 0.9989
print(round(parallel(0.99, 2), 6)) # 0.9999
Common mistake
Trusting the parallel formula blindly. It assumes copies fail independently. A bad config push, a shared DNS provider or one bug in every replica takes all copies down at once. Real multi-region availability is limited by these shared causes, not by the formula.
5. Health Checks & Load Balancing
Redundancy only helps if traffic actually avoids the broken copies, and that is the job of health checks feeding a load balancer. A load balancer continuously probes each backend and routes requests only to those reporting healthy, so a crashed or degraded replica is removed from rotation within seconds rather than serving errors.
The subtlety is in what “healthy” means. A shallow (liveness) check confirms the process is running and answering; a deep (readiness) check confirms it can actually do useful work — its dependencies are reachable, its caches are warm, it is not overloaded. Readiness is what should gate load balancing, but it must be designed carefully: if a deep check marks every replica unhealthy the instant a shared dependency blips, the load balancer will drain all traffic and turn a minor dependency wobble into a total outage. The fix is to fail health checks gradually and to distinguish “I am broken” from “a thing I depend on is briefly slow.”
Balancing algorithm. Round-robin is simple; least-connections or least-load handles uneven request costs better; consistent hashing preserves cache locality and minimizes reshuffling when the backend set changes.
Outlier detection. Beyond pass/fail health, eject a backend that is returning elevated errors or latency even if it still claims to be healthy.
Draining. On deploy or scale-down, stop sending new requests to a replica but let in-flight ones finish before it exits, so a rollout is not itself a source of errors.
Interview tip: Call out the health-check failure mode explicitly — a deep check that is too aggressive can cause a correlated, self-inflicted outage where the whole fleet drains at once. Interviewers love that you know redundancy can be defeated by a health check that treats a shared-dependency blip as every replica being dead.
6. The Request-Path Resilience Toolkit
Structural redundancy protects against a replica dying. A different, equally common threat is a dependency that does not die but turns slow — and a slow dependency, without defenses, will exhaust your threads, queue requests, and cascade into a full outage. The request-path toolkit is the set of patterns that contain that.
Every outbound call is wrapped in defenses: a timeout bounds how long it can hang, retries with backoff and jitter recover from transient blips without stampeding, a circuit breaker stops calling a failing dependency, bulkheads isolate resource pools, load shedding rejects low-priority work under stress, and graceful degradation serves a reduced result rather than none.
Timeouts & deadlines
Never make an unbounded call. Every request to a dependency gets a timeout, and better still a deadline that propagates down the call chain: if the caller has 100 ms left, the callee must not spend 200 ms. Deadline propagation prevents wasted work on requests whose caller has already given up, and it is the single most effective guard against cascading slowness.
Retries with backoff and jitter
Transient failures — a dropped packet, a brief blip — are worth retrying, but naive retries are dangerous. Retrying immediately, or retrying in lockstep across many clients, produces a retry storm that hammers an already-struggling dependency exactly when it is weakest. The fix is exponential backoff (wait longer after each attempt) plus jitter (randomize the wait so clients do not synchronize). Cap the number of attempts, and only retry idempotent operations.
Common mistake: retries at every layer
If a web tier, a service tier and a data tier each try 3 times, one user request can become 3 × 3 × 3 = 27 calls at the bottom exactly when that bottom layer is struggling. Retry at one layer only (usually the one closest to the user or the one that knows the operation is idempotent), and cap total retries with a retry budget, for example “retries may add at most 10% extra load.”
function call_with_retry(op, deadline):
attempt = 0
while attempt < MAX_ATTEMPTS and now() < deadline:
try:
return op(timeout = remaining(deadline)) # bound each try
except Transient:
base = min(MAX_DELAY, BASE_DELAY * 2 ** attempt) # exponential, capped
sleep(min(random(0, base), remaining(deadline))) # full jitter, never past the deadline
attempt += 1
raise Unavailable # give up, do not storm
Circuit breakers
When a dependency is clearly down, retrying at all is counterproductive. A circuit breaker tracks the recent error rate to a dependency; once it crosses a threshold the breaker opens and calls fail fast without even attempting the network, giving the dependency room to recover and freeing the caller’s threads. After a cooldown it goes half-open, letting a trickle of requests through to test recovery, and closes again if they succeed. This converts a slow, thread-exhausting failure into an instant, cheap one.
Bulkheads
Named after a ship’s watertight compartments, a bulkhead isolates resources so a failure in one area cannot sink the whole vessel. Give each downstream dependency its own connection pool and thread pool, so that if dependency A hangs and saturates its pool, calls to healthy dependency B still have threads to run on. Without bulkheads, one slow dependency consumes the shared thread pool and stalls every request, related or not.
Why a slow dependency is worse than a dead one: Little’s law
Little’s law says: requests in flight = arrival rate × time each request takes. At 2,000 requests/s and 50 ms per call you need about 2,000 × 0.05 = 100 threads busy at once. If the dependency slows to 2 s, you suddenly need 2,000 × 2 = 4,000 threads. A pool of 200 fills in a fraction of a second and every request waits. A dead dependency fails fast and frees the thread. A slow one holds it. Timeouts and bulkheads cap this damage.
7. Idempotency & Backpressure
Two more properties make the difference between a service that survives load and one that melts under it.
Idempotency. Retries, at-least-once delivery, and client reconnects all mean the same operation can arrive more than once. An operation is idempotent if applying it twice has the same effect as applying it once. Reads and overwrites are naturally idempotent; increments and appends are not. The standard fix is an idempotency key: the client attaches a unique id to each logical operation, the server records which keys it has already applied, and a duplicate returns the original result instead of doing the work again. This is what lets you retry safely without double-charging a customer or double-writing a record.
Backpressure. A reliable service must protect itself from more load than it can handle rather than accepting everything and collapsing. Backpressure is the family of mechanisms that push load back: bounded queues that reject when full, concurrency limits, and load shedding that drops low-priority requests first when the system is saturated. The counterintuitive but essential move is to reject early and cheaply — a fast rejection preserves capacity for the requests you do accept, whereas accepting everything guarantees that all requests get slow and then all fail. Admission control at the front door is more reliable than unbounded queues behind it.
Interview tip: When asked “what happens under overload?” the strong answer is not “autoscale” (which is too slow for a spike) but “shed load and degrade gracefully.” A service that serves 90% of traffic well and rejects 10% cleanly is far more reliable than one that tries to serve 100% and fails all of it.
8. Rollout Safety & Disaster Recovery
The leading cause of outages is not hardware — it is change. So a reliable service treats deployment as a first-class reliability concern, and prepares for the rare day it loses an entire region.
Safe rollouts
Canary. Deploy the new version to a small slice of traffic first, watch its error and latency SLIs against the old version, and only proceed if they hold. A bad change then hurts 1% of traffic, not 100%.
Progressive / staged rollout. Ramp from 1% → 10% → 50% → 100%, region by region, with automated health gates between stages.
Automatic rollback. Wire the gates to roll back automatically when SLIs regress, so recovery does not wait on a human noticing.
Feature flags. Decouple deploy from release: ship code dark, then turn a feature on with a flag you can flip off instantly without a redeploy.
Disaster recovery: RTO & RPO
For catastrophic loss, two numbers define the target. RTO (Recovery Time Objective) is how quickly you must be back up; RPO (Recovery Point Objective) is how much recent data you can afford to lose. They drive the DR strategy directly and trade cost against recovery.
Strategy
RTO / RPO
Cost
Backup & restore
Hours / hours
Cheapest — restore from backups on demand.
Pilot light
Tens of min / minutes
Core replicated, rest spun up on failover.
Warm standby
Minutes / seconds
Scaled-down copy always running, scale up on failover.
Active-active
Seconds / ~zero
Most expensive — both regions serve live.
Whichever you pick, the discipline that matters is testing it: run regular failover drills and game days, because a DR plan that has never been exercised is a hypothesis, not a capability.
9. Bottlenecks & Tradeoffs
Every reliability mechanism has a cost, and naming the tradeoffs is what separates a checklist from a design.
Reliability vs cost. Each extra nine roughly multiplies cost — more replicas, more regions, more on-call. Match the target to the business need; do not buy five nines for a service that needs three.
Redundancy vs consistency. Multi-region active-active gives the best availability but forces you to confront the CAP tradeoff: under a partition you choose between serving possibly-stale data (AP) or refusing writes to stay consistent (CP). State which you pick and why.
Retries vs stampedes. Retries improve success rates but, done wrong, cause the very overload they were meant to survive. Backoff, jitter, caps, and circuit breakers are mandatory, not optional.
Timeouts too tight vs too loose. Too tight and you fail healthy-but-slow requests; too loose and you let slowness cascade. Set them from observed latency percentiles, not guesses.
Complexity is itself a risk. Every mechanism you add is more code that can fail and more behavior on-call must understand. The most reliable design is the simplest one that meets the SLO.
Worked example: request budgets are not downtime budgets
The service has a 99.9% request-success SLO over one billion eligible requests.
Read top to bottom. The table below explains the state and guarantee at each step.
Step
State, calculation, and reasoning
1. Compute allowed bad requests
1,000,000,000×0.001=1,000,000. This is a request budget, not automatically 43 minutes of downtime.
2. Observe 5% failures
Burn=0.05/0.001=50×. Conversion to elapsed exhaustion time assumes representative request rate and the defined window.
3. Bound retry amplification
Three attempts at each of three layers can create 3^3=27 bottom-layer calls. Choose one informed retry owner and a shared budget/deadline.
4. Lose a failure domain
Verify the remaining fleet can carry peak load under safe utilization. Independent probability arithmetic omits shared config/code/dependencies.
Check your understanding: Can two replicas guarantee the independence formula's availability?
No. Common-cause failures, routing, shared state, and insufficient survivor capacity violate its assumptions.
Run a small, testable model
Download concepts.py. It uses only the Python 3.10+ standard library. From the directory where you saved it, run:
python3 concepts.py observability
python3 concepts.py test
The model prints request/time budget arithmetic separately and checks the burn-rate denominator.
The program contains deterministic assertions and failure traces. It is a learning model, not a production service: it does not implement network consensus, distributed durability, authentication, or third-party delivery. Read its contracts and try changing an input before extending it.
A reliability prompt is easy to ramble on. A simple time plan keeps you on track and shows the interviewer a method.
Minutes
What you do
What to say out loud
0–5
Clarify scope and target
“Let us agree on 99.95% over 28 days, p99 under 200 ms, RPO under 1 minute.”
5–10
Define SLIs and the error budget
“That is about 20 minutes of budget a month; I will alert on burn rate.”
10–20
Draw the redundant architecture
Stateless replicas across 3 AZs, replicated datastore, second region, health-checked LBs. Do the series/parallel math.
20–30
Walk the request path defenses
Deadlines, one layer of retries with jitter, circuit breakers, bulkheads, load shedding.
30–38
Failure drills
“An AZ dies”, “the DB primary dies”, “a dependency gets slow”, “a bad deploy”: say what each defense does.
38–45
Tradeoffs and what you would cut
Cost of each extra nine, CAP choice for multi-region, complexity risk.
11. Common Mistakes
Mistake
Why it hurts
Do this instead
Promising 100% or “five nines” by default
Zero or tiny budget; huge cost; no room to ship
Choose the target from user impact and state the downtime it allows
Replicas all in one zone or one rack
One power event takes every copy
Spread copies across failure domains
Retries without backoff, jitter or caps
Retry storms amplify an outage
Exponential backoff, full jitter, one retrying layer, retry budget
No timeouts, or one fixed timeout everywhere
Slow calls hold threads until the pool is gone
Deadlines that shrink as the request goes deeper
Deep health checks that fail on any dependency blip
The whole fleet drains at once
Separate liveness from readiness; fail gradually
A DR plan never tested
It fails on the day you need it
Regular failover drills and game days
12. Summary
A highly reliable service is built by assuming failure everywhere and layering defenses: define reliability precisely, remove single points of failure, keep traffic away from broken copies, contain slow dependencies on the request path, and rehearse recovery.
Concern
Mechanism
How do we define “reliable”?
SLIs measured from telemetry, an SLO target, and an error budget that governs release risk.
How do we survive component failure?
Stateless N+2 replicas spread across AZs and regions — independent failure domains.
How do we keep traffic off broken replicas?
Health-aware load balancing with readiness checks, outlier ejection, and draining.
Canary + progressive rollout with automated health gates and rollback; feature flags.
How do we survive losing a region?
DR strategy sized to RTO/RPO, tested with regular failover drills.
The recurring theme: assume everything fails, and make each failure local, cheap, and recoverable. Redundancy handles the parts that die; timeouts, breakers, and bulkheads handle the parts that merely slow down; error budgets and safe rollouts keep humans from becoming the outage. Reliability is the sum of many small, deliberate defenses, sized to a target you stated up front.
13. Check Yourself
A service has an SLO of 99.95% over 30 days. How many minutes of downtime does the budget allow?
Answer30 × 24 × 60 × 0.0005 = 21.6 minutes.
A request passes through three parts with availability 99.9%, 99.9% and 99.99%. Can the whole path meet a 99.9% SLO?
AnswerNo. 0.999 × 0.999 × 0.9999 ≈ 0.9979, which is 99.79%. You need redundancy for the weak parts or fewer hard dependencies.
Your error rate jumps to 2% against a 99.9% SLO. What is the burn rate, and how long until a 30-day budget is gone?
AnswerBurn rate = 0.02 ÷ 0.001 = 20. The budget lasts 720 h ÷ 20 = 36 hours.
Why add jitter to exponential backoff?
AnswerWithout jitter, clients that failed together retry together, in synchronized waves that hit the dependency at the same moments. Random waits spread the retries out.
A circuit breaker is open. What does a call do, and how does the breaker ever close again?
AnswerThe call fails fast without touching the network. After a cooldown the breaker goes half-open and lets a few trial requests through; if they succeed it closes, if they fail it opens again.
Your RPO is 0 and your RTO is 30 seconds. Which DR strategy fits, and what does it cost you?
AnswerActive-active with synchronous (or quorum) replication across regions. It costs double capacity and adds cross-region write latency to every commit.
14. Key Terms
SLI
An SLI measures a user-relevant outcome; an SLO sets its target over a window. For a success-ratio SLO, the allowed bad fraction is 1−SLO. An SLA is a contractual commitment that may use different definitions. Example and limits above.
SLO
An SLI measures a user-relevant outcome; an SLO sets its target over a window. For a success-ratio SLO, the allowed bad fraction is 1−SLO. An SLA is a contractual commitment that may use different definitions. Example and limits above.
SLA
An SLI measures a user-relevant outcome; an SLO sets its target over a window. For a success-ratio SLO, the allowed bad fraction is 1−SLO. An SLA is a contractual commitment that may use different definitions. Example and limits above.
Error budget
An SLI measures a user-relevant outcome; an SLO sets its target over a window. For a success-ratio SLO, the allowed bad fraction is 1−SLO. An SLA is a contractual commitment that may use different definitions. Example and limits above.
Burn rate
Burn rate is the observed bad-event fraction divided by the allowed bad fraction. Multi-window alerts require sustained long-window burn and a confirming short window so old incidents can clear. Example and limits above.
Failure domain
A set of things that can fail together (host, rack, zone, region).
N+2
Enough capacity to survive two units lost (one failed, one in maintenance).
Circuit breaker
Stops calling a failing dependency and fails fast instead.
Bulkhead
Separate resource pools so one failure cannot drain them all.
Load shedding
Backpressure tells an upstream producer that the downstream stage cannot accept more work. Buffering delays that signal; shedding deliberately rejects or drops lower-priority work when a bound is reached. Example and limits above.
Idempotent
A client-generated identity for one logical operation. The server binds it to caller scope, endpoint, request fingerprint, state, and result. A retry uses the same key and reuses the original outcome rather than repeating the effect. Example and limits above.
RTO / RPO
How fast you must recover / how much data you may lose.
15. Further Reading
Site Reliability Engineering (Google, O’Reilly), chapters on SLOs, handling overload and addressing cascading failures. Free at sre.google.
The Site Reliability Workbook, chapter “Alerting on SLOs” for burn-rate alerts. Free at sre.google.
Marc Brooker, “Timeouts, retries, and backoff with jitter”, Amazon Builders’ Library.
Martin Kleppmann, Designing Data-Intensive Applications, chapter 1 (reliability) and chapter 5 (replication).
Michael Nygard, Release It!, for circuit breakers and bulkheads.
