← CatZep/Engineering Guides/System Design/Coordination, Money & Markets

Designing for Idempotency & Exactly-Once

A system design interview guide to making retried operations safe, so that a request that lands twice never charges a card twice, places two orders, or otherwise double-applies a side effect.

A lost response leaves the caller uncertain whether an operation committed. The server/provider may know, but the caller must retry or query using the same logical identity. A durable, scoped, payload-bound key plus an atomic local effect/result or downstream idempotency contract protects against repeated effects within its stated retention/failure model.

Concepts to understand first

Each definition includes a concrete example and the condition under which the mechanism is useful. You can read this guide without opening another guide.

1. Requirements & Clarifying Questions

An interviewer may say "make the payments API safe to retry." Pin down the scope first:

• Which operations have side effects? Reads are safe already. Focus on creates, charges, sends and transfers.
• Who retries? Mobile clients, a gateway, a message consumer, or all three? Each needs its own dedup point.
• How long can a retry arrive late? Seconds for HTTP, but days for a queue replay. This sets the key TTL.
• Must the retry see the same response, or just "already done"? Storing responses costs space.
• Do downstream services accept a key? Payment processors usually do. Email providers often do not.

2. Back-of-Envelope Estimation

Take a payments API at 5K unsafe requests/s with a 24-hour key TTL.

3. API & Data Model

POST /v1/payments
Idempotency-Key: 9f1c2e7a-5b4d-4c1e-8a0f-2d6b3c9e1a77
{ "amount": 5000, "currency": "USD", "card": "tok_123" }

200 OK             first run, or a replay of a finished run
409 Conflict       same key still running: retry after a moment
422 Unprocessable  same key, different body: a client bug

CREATE TABLE idempotency_keys (
  key          TEXT PRIMARY KEY,         -- the unique index that claims a key
  request_hash TEXT NOT NULL,            -- fingerprint of the body
  status       TEXT NOT NULL,            -- IN_PROGRESS | COMPLETED
  response     BYTEA,                    -- status code + body to replay
  locked_until TIMESTAMP,                -- lease on IN_PROGRESS
  expires_at   TIMESTAMP NOT NULL        -- TTL, reaped by a sweeper
);

Scope the key to the caller. Store it as (account_id, key), so one customer's keys can never collide with another's.

4. Why Retries Are Unavoidable

The root cause of every duplicate problem is that a network call has three possible outcomes, not two. It can clearly succeed, it can clearly fail, or — the troublesome third case — it can leave the caller with no answer at all. A request may have reached the server and been fully processed, but the response was lost on the way back. From the caller's chair, a lost response is indistinguishable from a request that never arrived.

Given that ambiguity, the caller has only two choices, and both are bad if the operation is not idempotent. It can give up, risking that a legitimate operation silently never happened. Or it can retry, risking that an operation which already succeeded gets applied a second time. Real systems almost always choose to retry, because losing a customer's order is worse than the rare double, and because timeouts are common enough that a no-retry policy would make the system feel broken. Once you accept that retries are a permanent feature of the landscape, the design question becomes: how do we make a second, third, or tenth identical request harmless?

5. How Retries Create Duplicates

It helps to make the failure concrete. Imagine a checkout endpoint that charges a card and creates an order. The client calls it, the server charges the card and writes the order, and then the connection drops before the success response gets back. The client, seeing only a timeout, retries. The server happily charges the card again and creates a second order. The customer is now out double the money and has two shipments on the way — a failure that is both expensive and deeply eroding of trust.

The same shape appears anywhere a write has a side effect that the world can observe: sending money, placing an order, sending an email, incrementing a counter, provisioning a resource. Note that pure reads are naturally safe — fetching a record twice changes nothing — so the entire problem is confined to writes that do something. The job of an idempotency design is to let those writes be retried as freely as reads.

6. The Idempotency Key Pattern

The standard solution puts the client in charge of identity. Before sending a request that has a side effect, the client generates a single unique value — an idempotency key — that names this one logical request, and attaches it (typically as a header such as Idempotency-Key). The crucial property is that the same key is reused across all retries of the same logical request, but a brand-new logical request gets a brand-new key. A UUID generated once per "user clicked Pay" is the canonical choice.

On the server, the key turns "have I done this already?" from an unanswerable question into a simple lookup. The server records, for each key, the outcome of the request. The first time it sees a key, it executes the operation and stores the result against the key. On any later request bearing the same key, it skips execution entirely and returns the stored result. The client cannot tell whether its request was the one that did the work or a replay — it just gets the same answer, which is exactly what it wanted.

function handle(request, idempotency_key):
  existing = store.get(idempotency_key)
  if existing and existing.status == "COMPLETED":
    return existing.response          # replay: do not re-execute
  result = do_the_work(request)       # the side effect, run once
  store.put(idempotency_key, status="COMPLETED", response=result)
  return result

The pattern shifts a small responsibility to the client — it must generate and reuse a key — in exchange for a large guarantee from the server. It also keeps the contract explicit: a key represents one logical intent, so reusing a key for a genuinely different request would be a client bug, not the server's problem to disambiguate.

7. The Dedup Store

Store caller/tenant scope, endpoint, key, canonical operation fingerprint, status, result, and retention. One key represents one logical request; an untrusted global string is not a safe caller scope. Stable response replay may require saving the original body/status rather than rebuilding from mutable data.

States include in-progress, unknown/reconciling, completed, and confirmed-failed-with-no-effect. Timeout/5xx does not prove no effect, so deleting the claim can double-charge. Expired leases do not prove the old worker stopped. Resolve via local atomic transaction, provider idempotency, or a status/reconciliation protocol.

A TTL bounds protection; late retries can run again. Size it from clients, queues, provider windows, and financial policy. HTTP 409/422 and failure caching are API choices; providers may cache 500 outcomes. Follow the actual provider contract.

8. Concurrent Retries & Locking

For a local effect, claim the key, validate fingerprint, apply business state, and save outcome in one transaction with uniqueness and suitable isolation. Request dedup complements rather than replaces business constraints.

begin transaction
  insert operation(scope, key, fingerprint)  # UNIQUE(scope,key)
  apply_local_effect()                       # same DB transaction
  save_original_response()
commit

Remote effects need durable pending work and the same downstream identity on every attempt. Unknown outcomes stay reconciling, not automatically deleted. Worker takeover must be versioned/fenced and safe to repeat. Canonicalize operation-defining fields including principal/path; arbitrary JSON formatting is not semantic canonicalization. Bound retries and escalate uncertainty when the provider cannot deduplicate/report status.

9. Delivery vs Effect

Specify the observation boundary. A network can retransmit bytes/requests while TCP or an application exposes a deduplicated logical stream. 'Exactly-once delivery is impossible' is too broad as a statement about every API.

Across independent systems, an uncertain ACK makes retry ambiguous. At-least-once processing plus durable transactional dedup can expose exactly-once effects within a stated failure/retention model. Dedup alone, finite retries, or a lossy source does not prove every intent happens exactly once. Separate at-most-once safety from eventual completion/liveness.

10. The Transactional Outbox

Idempotency on the receiver handles duplicates, but there is a matching problem on the sender: how do you reliably publish an event exactly when a database change commits, with no chance of the two diverging? The tempting approach — write to the database, then publish to the message broker — has a fatal gap. If the process crashes between the two steps, you have committed the data but never announced it, and downstream systems never learn. Publish first and the opposite happens: you announce a change that the database then fails to commit.

The transactional outbox closes the gap by making the event part of the same database transaction as the business write. In one atomic transaction you write the business row and insert a row into an outbox table describing the event. Because they commit together, you can never have one without the other. A separate relay process then polls the outbox, publishes each pending event to the broker, and marks it sent. If the relay crashes mid-publish, it simply republishes on restart — which is harmless precisely because the consumers are idempotent on the event id.

# inside one DB transaction
begin()
  orders.insert(order)                # the business change
  outbox.insert(event="order_placed", payload=order, status="PENDING")
commit()                              # both rows, or neither

# relay, running separately
for event in outbox.where(status="PENDING"):
  broker.publish(event)               # at-least-once; consumers dedup
  outbox.mark(event.id, status="SENT")

Polling is not the only relay. With change data capture (CDC), a tool such as Debezium tails the database's write-ahead log and publishes each committed outbox row. That avoids polling load and keeps commit order. Either way, send the event id with the message so consumers can dedup.

11. Naturally Idempotent Design

The cheapest idempotency is the kind you do not have to bolt on, because the operation is idempotent by construction. Whenever you can express a write so that applying it twice yields the same end state as applying it once, you sidestep the whole machinery of keys and dedup stores. Several common shapes have this property.

• Upserts. "Set this record to this value" is naturally idempotent — running it again writes the same value and changes nothing. Contrast with "insert a new record," which duplicates on retry. Where the data model allows it, prefer set-to-state over append-new.
• Conditional writes. "Update only if the current version is N" (compare-and-set) makes a retry a no-op: the first write moves the version to N+1, so the retried write's condition no longer holds and it does nothing. This also guards against lost updates from concurrent writers.
• Deterministic identifiers. Deriving a record's primary key from the request content — rather than auto-generating one — means a retried create collides with the existing row instead of making a second one, turning the database's uniqueness into your dedup.

# naturally idempotent: set absolute state, not a delta
profile.set_email("a@b.com")          # safe to run twice

# conditional write: retry becomes a no-op
update order set status="PAID"
  where id = 42 and status = "PENDING" # second run matches nothing

12. Failure Modes

13. Tradeoffs & Alternatives

Worked example: a successful charge with a lost response

The caller uses key p42; the provider charges, but its reply is lost.

Run a small, testable model

Download concepts.py. It uses only the Python 3.10+ standard library. From the directory where you saved it, run:

python3 concepts.py dedup
python3 concepts.py test

The SQLite example covers transactional local dedup and payload mismatch; the workflow model represents remote uncertainty.

The program contains deterministic assertions and failure traces. It is a learning model, not a production service: it does not implement network consensus, distributed durability, authentication, or third-party delivery. Read its contracts and try changing an input before extending it.

14. A 45-Minute Interview Walkthrough

15. Common Mistakes

• Check, then run, then save. Two duplicates both pass the check. Claim the key first.
• A new key on every retry. Make the key once per user intent and reuse it.
• Caching timeouts or 5xx errors. Every retry then replays the failure.
• Ignoring the body. Same key with a different body must be an error, not a replay.
• Reclaiming a stuck key without knowing whether the side effect ran.
• Claiming the outbox gives exactly-once delivery. It gives at-least-once; consumers dedup.

16. Summary

Idempotency is the discipline that lets a system retry freely without doubling its effects. The pieces reinforce one another:

17. Check Yourself

1. A client times out on POST /payments. Why can it not tell whether it was charged?
   Answer

   A call has three outcomes: success, failure or no answer. A lost response looks the same as a lost request, so the client cannot know if the server ran the charge.

2. Two copies of one request arrive 1 ms apart. Both do get(key), find nothing and run. What single change fixes it?
   Answer

   Claim the key before the work with an insert guarded by a unique constraint. Only one insert can win. The loser gets a violation and returns 409 or the stored result.

3. A retry comes with the same key but amount changed from 50 to 500. What should the server do?
   Answer

   Return 422. The stored fingerprint does not match the new body, so this is a different request reusing a key, which is a client bug. Replaying the old answer would hide it.

4. The first attempt got a 503 from the processor. Should the server store that response against the key?
   Answer

   No. A 503 is a transient failure. If it were stored, every retry would replay the 503 and the payment would never go through. Delete or release the key so a retry can run.

5. At 5K requests/s with a 24-hour TTL, how many keys are stored, and how big is the store at 200 B per key?
   Answer

   5,000 × 86,400 = 432M keys. 432M × 200 B = 86.4 GB.

6. Why is the outbox relay at-least-once and not exactly-once?
   Answer

   The relay can publish an event and crash before it marks the row sent. On restart it publishes again. Consumers make the result exactly-once in effect by deduping on the event id.

18. Key Terms

Idempotent

A client-generated identity for one logical operation. The server binds it to caller scope, endpoint, request fingerprint, state, and result. A retry uses the same key and reuses the original outcome rather than repeating the effect. Example and limits above.

Idempotency key

A client-generated identity for one logical operation. The server binds it to caller scope, endpoint, request fingerprint, state, and result. A retry uses the same key and reuses the original outcome rather than repeating the effect. Example and limits above.

Dedup store

Table mapping each key to its status and result.

Request fingerprint

A hash of the body, stored to catch key reuse.

At-least-once delivery

Retry until acked; duplicates are possible.

Exactly-once effect

At-least-once delivery plus dedup on the receiver.

Transactional outbox

A business transaction writes its state change and an event row to the same database. A relay publishes committed event rows later, retrying until the broker durably accepts them. Example and limits above.

CDC

A business transaction writes its state change and an event row to the same database. A relay publishes committed event rows later, retrying until the broker durably accepts them. Example and limits above.

Conditional write

Update only if a condition, such as a version, still holds.

Two generals problem

Proof that two parties cannot be sure of agreement over a lossy link.

19. Further Reading

• M. Kleppmann, Designing Data-Intensive Applications, chapter 11 ("Stream Processing"), sections on idempotence and exactly-once; and chapter 12 on end-to-end operation ids.
• Stripe API documentation, "Idempotent requests."
• IETF HTTPAPI working group, Internet-Draft "The Idempotency-Key HTTP Header Field."
• RFC 9110, HTTP Semantics, section 9.2.2 "Idempotent Methods."
• C. Richardson, Microservices Patterns (Manning, 2018), chapter 3, the transactional outbox pattern; and the Debezium documentation on the outbox event router.
• P. Helland, "Idempotence Is Not a Medical Condition," ACM Queue, 2012.