A system design interview guide to moving and tracking money correctly, where the defining requirement is that the system must never lose, invent, or double-count a single unit of currency.
Student edition · start with the guarantee
Represent money as auditable postings and enforce balances, currency, and retries transactionally.
Correctness contract: Each transaction balances per currency. Both account effects, no-overdraft checks, postings, and local request outcomes share an atomic boundary.
Basic HTTP, databases, and programming are assumed. Specialized terms are explained below. Named products motivate the problem; this is an educational design, not a claim about their current internals. Capacity numbers are illustrative decimal payload estimates, not benchmarks. Add indexes, protocol overhead, replicas, headroom, and failure capacity before provisioning.
A payment or wallet system is the textbook case where correctness outranks every other quality. Most services can tolerate a dropped request or a stale read; a money system cannot tolerate a balance that is off by a cent. That single constraint reshapes the whole design. Instead of optimizing for raw availability, you lean toward strong consistency on the money path; instead of mutating balances in place, you record an immutable, balanced trail of entries; and instead of hoping retries are harmless, you make them harmless with idempotency. This guide develops the core of such a system: the double-entry ledger, the locking that prevents double-spends, idempotent payment requests, sagas for multi-step flows, and the reconciliation that proves it all adds up.
Each definition includes a concrete example and the condition under which the mechanism is useful. You can read this guide without opening another guide.
Double-entry ledger
An append-only journal where postings balance within each transaction and currency. Account types determine debit/credit meaning; a signed wallet-delta model is a simplifying convention, not the full accounting definition.
Example. A simplified transfer records Alice −1000 cents and Bob +1000 cents, atomically, with transaction sum zero. Funding, fees, and withdrawals post against explicit counterparty accounts.
Limit. Balanced postings detect some errors but do not prove the right amount/counterparty was chosen. Mutable balances, no-overdraft checks, currency, and idempotency must also be transactional.
Optimistic concurrency and compare-and-set
Update state only if the expected version or status still matches, in the same atomic operation that changes it. A failed comparison means a conflicting operation won; reread or return a conflict.
Example. UPDATE seats SET status='held', version=version+1 WHERE id=? AND status='available' allows only one concurrent claimant to update the row.
Limit. Application-level GET then unconditional SET is not equivalent. Include acquisition identity to reject stale callbacks, and couple related invariants in a transaction.
Idempotency keys
A client-generated identity for one logical operation. The server binds it to caller scope, endpoint, request fingerprint, state, and result. A retry uses the same key and reuses the original outcome rather than repeating the effect.
Example. Payment p42 times out after the provider charged it. Retrying with p42 lets the provider return the same charge instead of creating a new one.
Limit. A unique local claim is insufficient if the business effect and saved result can diverge. Keep unknown outcomes pending, use downstream idempotency, and state the retention/retry horizon.
Sagas and compensation
A durable workflow of local transactions with corrective actions for failures. Compensation is a new business operation, not a rollback of history. The workflow exposes intermediate states and must recover after crashes.
Example. Reserve a seat, authorize payment, confirm booking, then capture. A declined authorization releases the seat; an uncertain charge stays pending until reconciled.
Limit. Timeout does not mean the remote effect failed. Compensation can fail too and needs retries/escalation. A saga does not provide isolation or guarantee every external effect is reversible.
Transactional outbox
A business transaction writes its state change and an event row to the same database. A relay publishes committed event rows later, retrying until the broker durably accepts them.
Example. An order and its OrderCreated outbox row commit together. A relay crash after publish but before marking sent publishes the same event ID again on restart.
Limit. The relay is at-least-once, not magically exactly-once. Consumers dedup transactionally; preserve per-entity sequence when order matters. CDC is change data capture: reading committed database changes.
CAP and consistency models
During a network partition, a distributed service cannot offer both linearizable operations and successful completion of every eligible request at nonfailed nodes. CAP consistency is linearizability; it is not durability or 'no bytes lost'.
Example. A minority that cannot contact the authority blocks or rejects a linearizable read/write. A mergeable append log can accept local writes on both sides and preserve both records, while not giving one-copy ordering.
Limit. Causal/session/eventual guarantees are not a universal availability ladder. Sticky routing alone fails after failover unless the new replica has satisfied the client's dependencies.
1. Requirements & Clarifying Questions
"Design a payment system" can mean a wallet, a checkout or a whole processor. Ask:
What are we building? An internal wallet with its own balances, or checkout that calls an outside processor such as Stripe or Adyen? This guide covers both: a ledger, plus calls out to a processor.
Which flows? Pay, refund, transfer between users, payouts?
How many currencies? Multi-currency needs FX entries and a currency on every amount.
Volume and peaks? Payments per day, and the peak on a sale day.
Compliance? Card data triggers PCI DSS scope. Most designs never touch raw card numbers and use processor tokens instead.
Functional
Non-functional
Charge a customer; refund all or part
Correctness: no money created, lost or moved twice
Transfer between internal accounts
Exactly-once effect under retries
Show balance and transaction history
Full audit trail kept for years
Reconcile with processors and banks daily
Strong consistency on the write path; high availability for reads
Intuition
Think of the ledger as a bank's paper journal written in pen. You never erase a line. You fix a mistake by writing a new line that reverses it. Every line moves money from somewhere to somewhere, so the columns always add up to zero.
2. Back-of-Envelope Estimation
100M payments/day averages 1157/s; a stated 5× peak gives 5787/s. Two 200-byte postings each give 40 GB/day and 14.6 TB/year payload. Seven years are about 102 TB before indexes, replicas, reconciliation, and headroom.
These numbers do not establish a universal primary limit. Hot accounts, commit policy, transaction shape, and storage determine capacity. Account sharding makes arbitrary transfers cross-shard; design that boundary first. Retain ledger history under explicit audit/legal policy rather than asserting it must literally remain forever.
3. API & Data Model
POST /v1/payments Idempotency-Key: <uuid>
{ "amount": 1999, "currency": "USD", "source": "tok_visa", "merchant": "m_42" }
-> 201 { "id": "pay_81", "status": "succeeded" | "pending" | "failed" }
POST /v1/payments/pay_81/refunds Idempotency-Key: <uuid> { "amount": 500 }
GET /v1/accounts/acc_7/balance
GET /v1/accounts/acc_7/entries?cursor=...
Amounts are integers in minor units (1999 = $19.99). The status on a payment is a small state machine: pending → succeeded or failed, with unknown for timeouts that reconciliation resolves later.
The payment API claims the key, writes balanced entries, updates balances and adds an outbox row in one transaction. It calls the processor with the same key. Reconciliation compares the ledger with the processor's settlement file. History screens read from replicas.
4. Correctness Above All
The first thing to say in an interview about a payment system is that its success criterion is unusual. For most systems we negotiate trade-offs — a little staleness for a lot of availability, some duplicates for simplicity. For money there is no such negotiation on the core invariant: every unit of currency that enters the system must be accounted for at every instant, and none may be created or destroyed except through an explicit, recorded transaction. A balance that is silently wrong is not a bug to be triaged later; it is a failure of the system's entire reason for existing.
This requirement is what justifies design choices that would look over-engineered elsewhere. We use transactions and locks where another system might use eventual consistency, we keep an immutable history where another system might overwrite a value, and we run continuous reconciliation where another system might trust its own writes. Each of these costs throughput or latency, and each is worth it because the alternative is losing or duplicating money. Stating this priority up front tells the interviewer you understand the domain rather than reflexively reaching for the scaling playbook.
A sharp framing: in a payment system, correctness is the feature and everything else — throughput, latency, even availability — is subordinate to it. Decisions that trade a little speed for certainty about the money are almost always right.
5. The Double-Entry Ledger
A double-entry journal balances debit and credit postings for each transaction and currency. Debit/credit are accounting directions whose effect depends on account type: debits increase asset accounts but decrease liability accounts; credits do the reverse. They are not universally synonyms for 'money sent' and 'money received'.
For a simplified wallet-delta model, a $10 internal transfer can use Alice −1000 cents and Bob +1000 cents with sum zero. A real ledger also names funding, withdrawal, bank, fee, suspense, and other counterparty accounts so every movement has balanced postings. Multi-currency movements require explicit per-currency accounts and an exchange/rounding policy; do not sum unlike currencies as one invariant.
Commit all postings and relevant balance/materialization/idempotency effects atomically. Append-only entries are corrected by new linked postings rather than silent history edits. Balancing catches an unbalanced journal, not a wrong-but-balanced amount, destination, duplicated transfer, or missing external settlement; validation and reconciliation remain necessary.
Use integer smallest units or an exact decimal with the correct currency scale/rounding rules. Binary floats can introduce rounding errors. Retention and immutability follow audit/legal policy, with access controls and explicit corrections rather than an unqualified promise to keep all data forever.
6. Append-Only & Balances
Ledger entries are append-only and immutable: once written, an entry is never updated or deleted. A mistake is corrected not by editing the past but by writing a new, compensating entry that reverses it. This is a deliberate constraint, and it buys two things that are hard to overstate for a money system. First, a complete audit trail: every change to every balance is a row you can point to, with a cause and a timestamp, which is exactly what auditors and regulators require. Second, the elimination of an entire class of bugs — there is no in-place update to race on, no value to silently clobber.
That raises an obvious question: if you only ever append entries, how do you know an account's current balance? There are two answers, and real systems often use both. You can derive the balance by summing all of an account's entries — always correct, but slow once the history is large. Or you can materialize the balance, keeping a running total that is updated as part of the same transaction that appends the entry. The materialized balance is fast to read, and the append-only entries remain the source of truth you can always recompute against to catch drift.
Approach
How balance is known
Trade-off
Derived
Sum all entries for the account on demand.
Always correct; slow as history grows.
Materialized
Keep a running total, updated with each entry.
Fast reads; must update atomically with the entry.
Both
Materialize for speed; derive periodically to verify.
Fast and self-checking; a little extra work.
7. Preventing Double-Spend
Validate positive exact money units, currency, and account pair. Serialize conflicting changes with row locks or atomic conditional updates. Lock multiple accounts in stable ID order where supported; bound deadlock/serialization retries.
A transfer updates both source and destination materialized balances, writes balanced postings, enforces relevant no-overdraft rules, and saves idempotency outcome in one transaction. Source deduction plus ledger append without destination materialization is inconsistent. Account types determine whether negative balances are valid.
Account-ID sharding does not co-locate arbitrary transfers. Use a reviewed distributed transaction, centralized posting boundary, or explicitly pending/reconciled protocol. A saga is not immediate cross-shard ledger atomicity. Benchmark throughput instead of treating 6000 TPS as a universal database limit.
8. Idempotent Payment Requests
Payments travel over networks, and networks drop responses, so a client that does not hear back will retry — and a retried payment must never charge twice. The defense is the same idempotency-key pattern used throughout reliable systems: the client generates a unique key for each logical payment and reuses it on every retry of that payment. The payment service records each key's outcome and, on seeing a key it has already completed, returns the stored result instead of charging again.
The payment API claims the idempotency key with an INSERT before doing anything else. If the claim wins, it charges once, passing the same key to the processor, and marks the key COMPLETED. A retry of a completed key gets the stored result; a retry while the first attempt is still running gets 409. If the charge times out, retry with the same key, or mark it UNKNOWN and let reconciliation settle it.
function pay(request, idempotency_key):
try:
dedup.claim(idempotency_key) # claim FIRST: unique constraint, one winner
except UniqueViolation:
prior = dedup.get(idempotency_key)
if prior.status == "COMPLETED":
return prior.response # retry: do not charge again
return 409 # first attempt still running
result = charge(request, idempotency_key) # processor dedups on the same key
dedup.complete(idempotency_key, result)
return result
Claim the key before checking for a prior result. If you check first and claim second, two duplicates can both see "no result yet" before either claims. Passing the same key to the external processor matters just as much. If the processor times out, you do not know whether it charged. Retry with the same key, and the processor returns the first result instead of charging again. If you still cannot tell, mark the payment UNKNOWN and let reconciliation settle it. Never guess.
This is the bridge between the payment system and general idempotency design: the money path is simply the highest-stakes place to apply it. A duplicate notification is an annoyance; a duplicate charge is a refund, a support ticket, and a dent in trust, so the dedup store here is not optional polish but part of the correctness guarantee.
9. Multi-Step Payments & Sagas
Durable payment states distinguish reserved, authorized, captured, settling, succeeded, rejected, and unknown. Authorization/capture differ by provider; the 'pivot' is business-specific. A timeout is unknown, not a decline. Query/retry the same provider key before compensation.
Refunds are new money movements with their own keys and may fail, incur fees, or take days. Compensations need retries, deadlines, and escalation; permanent external failures can leave unresolved workflows. 2PC provides atomic commit for participating resources and can block after prepare; isolation is a separate property. Banks/processors generally do not participate in your DB transaction.
10. Reconciliation & Auditing
Even a careful system must continually prove to itself that it is correct, because bugs, partial failures, and discrepancies with external processors are inevitable over time. Reconciliation is the routine that does this proving. Internally, it re-derives balances from the append-only ledger and checks them against the materialized balances, and it verifies the global invariant that debits equal credits. Externally, it compares the system's records against statements from banks and payment processors — the money you think you moved against the money they say moved — and flags any mismatch for investigation.
Processors report many things only later: a charge that timed out on your side but succeeded on theirs, a chargeback, or a settlement that nets out fees. Reconciliation is how those UNKNOWN and late states get resolved, usually from a daily settlement file.
This is where the append-only, double-entry design pays off most. Because the ledger is an immutable, balanced history, reconciliation has a trustworthy source of truth to check against, and any drift surfaces as a concrete discrepancy with a clear location rather than a vague sense that the numbers feel off. The same immutable trail is the audit record regulators require: a complete, tamper-evident account of every movement of money, queryable long after the fact.
# reconciliation: the system checks itself
for account in accounts:
derived = sum(ledger.entries(account)) # recompute from source of truth
assert derived == account.materialized_balance
assert sum(all debits) == sum(all credits) # global money conservation
diff = compare(internal_records, processor_statement) # external check
11. Consistency Over Availability
Every distributed system eventually faces the choice the CAP theorem describes: when the network partitions, you can stay consistent or stay available, but not both. Most consumer systems lean toward availability — better to serve a slightly stale feed than an error page. The money path makes the opposite call. If the system cannot be sure a payment will be recorded correctly and exactly once, it should refuse the operation rather than risk corrupting the ledger. A declined payment that the user can retry is recoverable; a double charge or a lost transfer is not.
In practice this means the core money path is built on strongly consistent storage and synchronous, transactional writes, accepting the latency and the occasional unavailability that come with it. It does not mean the entire product must be consistency-first — read-heavy, non-monetary surfaces like transaction history or analytics can be served from replicas and caches, where staleness is harmless. The art is drawing the line: strong consistency for anything that moves or guards money, relaxed consistency for everything that merely reports on it.
A useful boundary to articulate: be strict on the write path that changes balances, and relaxed on the read paths that only describe them. You buy correctness exactly where it is non-negotiable and pay for it nowhere else.
12. Failure Modes
What fails
What happens
How to handle it
Processor call times out
Unknown whether the card was charged
Retry with the same key; else mark UNKNOWN and reconcile
Client retries a payment
Risk of double charge
Idempotency key, claimed first
Crash between debit and credit
Half a transfer
Both entries and balances in one DB transaction
Two transfers lock accounts in opposite order
Deadlock
Lock in account-id order; retry on deadlock errors
Saga step fails after the charge
Money taken, order not placed
Retry forward steps after the pivot; refund only as a last resort
Ledger and processor disagree
Silent drift
Daily reconciliation; alert and hold on any mismatch
Hot merchant account
Lock contention caps throughput
Split into sub-accounts and sum them, or batch credits
13. Tradeoffs & Alternatives
Choice
Option A
Option B
Usual pick
Concurrency control
Pessimistic row locks: simple, waits under contention
Optimistic versions: no locks, retries on conflict
Pessimistic for hot accounts, optimistic otherwise
Cross-service consistency
2PC: atomic, blocking
Saga: non-blocking, visible middle states
Saga, with the pivot step placed carefully
Balance
Derived from entries: always right
Materialized: fast
Materialize, verify by deriving
Storage
Relational DB: transactions, constraints
NoSQL: scale, weaker transactions
Relational (or NewSQL) for the ledger
Partition behavior
Refuse writes (CP)
Accept and fix later (AP)
CP for money movement
Worked example: two spends against one ten-dollar balance
Alice has 1000 cents; two distinct payments each try to spend 1000.
Read top to bottom. The table below explains the state and guarantee at each step.
Step
State, calculation, and reasoning
1. Serialize the debit
Lock the source row in a transaction or conditionally debit WHERE balance≥amount. Two different idempotency keys do not stop overspending by themselves.
2. Commit the first transfer
Debit Alice, credit Bob, append balanced postings, and update both materialized balances with request outcome atomically.
3. Reject the second transfer
It sees zero funds or updates zero rows. Stable account lock ordering avoids avoidable deadlocks.
4. Reconcile remote settlement
A local balanced journal cannot prove a bank settled. Preserve provider-keyed pending/unknown state and reconcile evidence.
Check your understanding: Does a balanced ledger rule out every money bug?
No. Wrong amounts, currencies, counterparties, or duplicate balanced transactions still balance. Validate those invariants too.
Run a small, testable model
Download concepts.py. It uses only the Python 3.10+ standard library. From the directory where you saved it, run:
python3 concepts.py ledger
python3 concepts.py test
The model checks conservation, destination credit, no overdraft, duplicate replay, and rollback.
The program contains deterministic assertions and failure traces. It is a learning model, not a production service: it does not implement network consensus, distributed durability, authentication, or third-party delivery. Read its contracts and try changing an input before extending it.
Clarify: wallet or checkout, flows, currencies, volume. State "correctness first."
5–10
Estimate TPS and ledger growth. Note that it fits a sharded relational DB.
10–20
API, the double-entry data model, integer cents, the high-level diagram.
20–32
Deep dives: double-spend and locking, idempotency with the processor, sagas vs 2PC.
32–40
Failure modes: timeouts and UNKNOWN, reconciliation, deadlocks.
40–45
CAP stance and the tradeoffs table; recap against the requirements.
Interview tip
Expect "what if the processor times out?" It is the favorite follow-up. The strong answer has three parts: retry with the same idempotency key, never assume failure, and let reconciliation settle what is still unknown.
15. Common Mistakes
Floats for money. Use integer minor units or DECIMAL.
Updating a balance with no ledger entry. Every change is a balanced pair of entries in the same transaction.
Treating a processor timeout as a failure and charging again with a new key.
Checking the dedup store before claiming the key. Claim first.
Saying a saga is atomic. It has visible middle states, and some steps cannot be undone.
Serving balances from a cache for spend decisions. Reads that guard money go to the primary.
16. Summary
A payment system is, above all, a correctness machine — a set of choices that together make it impossible to quietly lose or double-count money:
Concern
Mechanism
What is the overriding requirement?
Correctness — never lose, invent, or double-count money — above throughput or availability.
How is money tracked?
A double-entry ledger: every transaction is balanced debits and credits.
How is history kept honest?
Append-only, immutable entries; balances derived or materialized from them.
How is double-spend prevented?
One DB transaction with row-level locking or optimistic versioning on the balance.
How are retried payments made safe?
Scoped payload-bound idempotency plus atomic local outcome/effect or provider dedup protects repeated charges within its defined retention/failure model.
How are multi-step payments coordinated?
Sagas expose intermediate states and recover through new corrective operations, retries, and reconciliation. Compensation is not a rollback and can fail or remain pending.
How do we know it all adds up?
Reconciliation against the ledger and external statements; immutable audit trail.
What do we sacrifice for correctness?
Consistency over availability on the money path — refuse rather than corrupt.
The recurring theme: a payment system spends complexity to buy certainty. Double-entry makes errors visible, immutability removes a class of bugs, locking serializes the dangerous writes, idempotency tames retries, sagas keep distributed flows whole, and reconciliation continuously proves the money is all there.
17. Check Yourself
Alice pays Bob $12.50. Write the ledger entries in cents. What must they sum to?
Answer
Alice −1250, Bob +1250. They must sum to 0.
Alice has $10 and sends two $10 payments at the same moment. Show how the double-spend happens, and one fix.
Answer
Both requests read a balance of $10, both pass the check, and both deduct, so the balance hits −$10. Fix: SELECT … FOR UPDATE on the account row inside the transaction, so the second request waits and then sees $0.
Transfer 1 goes A→B and transfer 2 goes B→A at the same time. How can this deadlock, and how do you prevent it?
Answer
Transfer 1 locks A and waits for B. Transfer 2 locks B and waits for A. Neither can finish. Prevent it by always locking rows in a fixed order, such as the lower account id first.
The processor call times out. Should you mark the payment failed and let the user retry?
Answer
No. The charge may have gone through. Retry with the same idempotency key so the processor returns the first result. If it is still unclear, mark it UNKNOWN and let reconciliation settle it.
Why is "charge the card" usually the pivot step of a checkout saga?
Answer
It is costly to undo, since a refund is a new money movement with delay and maybe fees. So cheap, reversible steps like reserving stock go before it. Steps after it are retried until they succeed rather than undone.
At 100M payments/day, 2 entries each at 200 B, how much ledger do you store in 7 years?
Answer
100M × 2 × 200 B = 40 GB/day. × 365 = 14.6 TB/year. × 7 ≈ 102 TB.
18. Key Terms
Double-entry ledger
An append-only journal where postings balance within each transaction and currency. Account types determine debit/credit meaning; a signed wallet-delta model is a simplifying convention, not the full accounting definition. Example and limits above.
Debit / credit
The two sides of an entry; signed amounts that sum to zero.
Minor units
The smallest currency unit, such as cents, stored as an integer.
Materialized balance
A running total updated with each entry.
Double-spend
Spending the same funds twice through a race.
Pessimistic / optimistic locking
Lock before reading / check a version when writing.
Saga
A durable workflow of local transactions with corrective actions for failures. Compensation is a new business operation, not a rollback of history. The workflow exposes intermediate states and must recover after crashes. Example and limits above.
Pivot step
The saga step after which you only go forward.
2PC
Two-phase commit: prepare everywhere, then commit everywhere.
Reconciliation
Checking your ledger against itself and against outside statements.
PCI DSS
Card-industry security rules for handling card data.
19. Further Reading
M. Kleppmann, Designing Data-Intensive Applications, chapter 7 ("Transactions") on isolation and lost updates, and chapter 9 on atomic commit and 2PC.
H. Garcia-Molina, K. Salem, "Sagas," ACM SIGMOD, 1987.
C. Richardson, Microservices Patterns (Manning, 2018), chapter 4, "Managing transactions with sagas."
M. Fowler, Analysis Patterns (Addison-Wesley, 1996), chapter 6, accounting patterns.
Stripe API documentation, "Idempotent requests"; and the PCI Security Standards Council's PCI DSS documents.
