A system design interview guide to building a notification service that reliably fans out push notifications, SMS, and email to millions of users without losing messages, sending duplicates, or spamming anyone.
Student edition · start with the guarantee
Accept notification intent reliably and distinguish provider acceptance, device delivery, and a user's observation.
Correctness contract: Dispatch intent is recoverable and retries preserve identity. Provider dedup/retention limits and unknown outcomes are explicit.
Basic HTTP, databases, and programming are assumed. Specialized terms are explained below. Named products motivate the problem; this is an educational design, not a claim about their current internals. Capacity numbers are illustrative decimal payload estimates, not benchmarks. Add indexes, protocol overhead, replicas, headroom, and failure capacity before provisioning.
A notification system is one of those services that looks trivial from the outside ("just send a message") and turns out to be a surprisingly rich design problem once you account for scale, third-party gateways you do not control, and the hard requirement that you neither lose a message nor send it twice. The interesting work is not the sending itself but everything around it: deciding what contact information to keep, decoupling the callers from the slow external gateways, retrying without duplicating, respecting user preferences, and being able to prove afterwards whether a notification was actually delivered. This guide walks through a design that handles all of that, building up from the channels outward.
Each definition includes a concrete example and the condition under which the mechanism is useful. You can read this guide without opening another guide.
Transactional outbox
A business transaction writes its state change and an event row to the same database. A relay publishes committed event rows later, retrying until the broker durably accepts them.
Example. An order and its OrderCreated outbox row commit together. A relay crash after publish but before marking sent publishes the same event ID again on restart.
Limit. The relay is at-least-once, not magically exactly-once. Consumers dedup transactionally; preserve per-entity sequence when order matters. CDC is change data capture: reading committed database changes.
Idempotency keys
A client-generated identity for one logical operation. The server binds it to caller scope, endpoint, request fingerprint, state, and result. A retry uses the same key and reuses the original outcome rather than repeating the effect.
Example. Payment p42 times out after the provider charged it. Retrying with p42 lets the provider return the same charge instead of creating a new one.
Limit. A unique local claim is insufficient if the business effect and saved result can diverge. Keep unknown outcomes pending, use downstream idempotency, and state the retention/retry horizon.
Sagas and compensation
A durable workflow of local transactions with corrective actions for failures. Compensation is a new business operation, not a rollback of history. The workflow exposes intermediate states and must recover after crashes.
Example. Reserve a seat, authorize payment, confirm booking, then capture. A declined authorization releases the seat; an uncertain charge stays pending until reconciled.
Limit. Timeout does not mean the remote effect failed. Compensation can fail too and needs retries/escalation. A saga does not provide isolation or guarantee every external effect is reversible.
Backpressure and load shedding
Backpressure tells an upstream producer that the downstream stage cannot accept more work. Buffering delays that signal; shedding deliberately rejects or drops lower-priority work when a bound is reached.
Example. A sink accepts 100 events/s while input is 150/s. A 1000-event queue fills in 20 seconds. The system must then block, reject, or apply its documented drop policy.
Limit. A finite queue cannot absorb an infinite outage. Prioritized messages still need hard quotas, deadlines, and bounded client queues; otherwise an exemption can exhaust the system.
Leases
Time-limited permission issued by an authority. Expiry lets new work proceed after a client stops renewing; it does not prove that the previous holder is dead or unable to act.
Example. A 30-second lease expires while A is paused for 45 seconds. B can acquire a new lease, but A can still wake up and send a delayed write.
Limit. Heartbeats improve recovery time, not safety by themselves. Compare-and-renew/delete must verify a unique acquisition token atomically, and protected effects need fencing or equivalent serialization.
Pub/sub and durable queues
Publish/subscribe routes an event to interested subscribers; it may be ephemeral or durable. A queue/log with retained records and offsets allows replay after a consumer is disconnected.
Example. An online gateway misses a comment while restarting. Ephemeral Redis Pub/Sub cannot replay it; a durable comment log can fetch after the gateway's last sequence.
Limit. Pub/sub alone does not promise persistence, global order, or exactly-once effects. Broker choice, acknowledgements, retry horizon, and client catch-up protocol define those guarantees.
1. Requirements & Clarifying Questions
Clarifying questions to ask
Which channels: push, SMS, email? Real-time alerts, bulk campaigns, or both? How many per day? Must a notification never be lost, and is a rare duplicate OK? Do users control what they get? Any quiet hours? Assume: all three channels, 100M push + 5M SMS + 50M email a day, campaigns of up to 50M users, soft real time (seconds), opt-out required.
Functional
Non-functional
Internal services send by user id + template
No accepted notification is lost
Push (iOS, Android), SMS, email
Duplicates are rare (at-least-once, deduped)
Users opt out per category and channel
Transactional alerts arrive within seconds, even during a campaign
Track sent, delivered, clicked
One gateway outage does not affect other channels
2. Back-of-the-Envelope Estimation
Quantity
Assumption
Result
Push per day
given
100M
SMS per day
given
5M
Email per day
given
50M
Average push rate
100M ÷ 86,400 s
~1.2K/s
Average total rate
155M ÷ 86,400 s
~1.8K/s
Campaign burst
50M users in 10 minutes (600 s)
~83K/s
Notification log
155M × ~500 B per record
~77.5 GB/day
Takeaway
Average load is small; bursts are ~46× the average
queues absorb bursts
Intuition
A post office is quiet most of the day, then a store mails a million flyers at once. It does not hire a million carriers. It stacks the flyers in bins (queues) and sends them out as fast as the trucks allow, while letters marked urgent jump ahead.
Tip: separate urgent from bulk
Put transactional messages (password reset, "your ride is here") and marketing on different queues or priorities. Otherwise a 50M-user campaign sits in front of a login code for many minutes.
3. API Design
POST /v1/notifications
{
"idempotency_key": "order-482-shipped", # caller retries are safe
"user_ids": [123],
"template_id": "order_shipped",
"params": { "order_id": 482, "eta": "Friday" },
"channels": ["push", "email"], # or "auto"
"priority": "transactional" # or "bulk"
}
-> 202 Accepted { "notification_ids": ["n_9f1c..."] }
GET /v1/notifications/{id} # status for support/debugging
PUT /v1/users/{id}/preferences { "marketing": { "email": false } }
POST /v1/devices { "user_id":123, "platform":"ios", "token":"..." }
The server replies 202 Accepted, not 200 OK: the request is stored and queued, but nothing has been sent yet.
4. Channels and Gateways
The first thing to pin down is that a notification system almost never delivers messages itself. Instead it hands each message to a third-party gateway that owns the last mile to the device or inbox. There are several distinct channels, each with its own gateway, its own credentials, and its own quirks. Your design has to treat them as separate pipelines rather than assuming one uniform "send" path.
Channel
Gateway
What it does
iOS push
APNs (Apple Push Notification service)
Delivers push notifications to iPhones and iPads. You authenticate to APNs and send the payload addressed to a per-device token.
Android push
FCM (Firebase Cloud Messaging)
Google's equivalent for Android devices (and a cross-platform option). Again addressed by a device registration token.
SMS
An SMS provider (e.g. a commercial messaging API)
Sends a text message to a phone number. Usually metered per message and subject to carrier rules.
Email
An email provider / transactional email service
Sends email to an address, handling deliverability concerns like SPF, DKIM, and bounce tracking.
The common thread is that all four are external, networked, and unreliable from your point of view. They can be slow, rate-limited, or temporarily down, and you cannot fix them — you can only react. That single fact drives most of the architecture that follows: you must isolate yourself from each gateway so that one misbehaving channel cannot stall the others, and you must be able to retry safely when a gateway returns an error.
A useful reframing for an interview: you are not building a "message sender," you are building a resilient dispatch layer in front of four flaky external services. Every design choice — queues, retries, dedup — exists to absorb the unreliability of the gateways.
5. Gathering Contact Info
Before you can send anything you need to know where to send it, and that information has to be collected and kept up to date well before the first notification fires. The natural collection points are when a user signs up and when they install the app on a new device.
For each user you want to store:
User id — the stable internal identifier that everything else hangs off.
Device token / device id — the per-device address that APNs or FCM requires. A single user may have several, one per device, and tokens can rotate, so this is a one-to-many relationship that needs refreshing.
Phone number — for the SMS channel, ideally validated and stored in a normalized international format.
Email address — for the email channel.
When the mobile app is installed and registers for push, it obtains a device token from the OS and sends it to your backend; you persist it against the user id. Sign-up captures the phone number and email. All of this lands in a database that the notification system reads at send time to resolve a user id into concrete addresses.
# on app install / push registration
function register_device(user_id, platform, device_token):
db.devices.upsert(user_id, platform, device_token, last_seen=now())
# on sign-up
function register_contact(user_id, phone, email):
db.users.upsert(user_id, normalize(phone), email)
A schema sketch keeps the one-to-many device relationship explicit:
Table
Key fields
Notes
users
user_id, phone, email, locale, timezone
One row per user. Locale and timezone matter later for templating and quiet hours.
devices
device_id, user_id, platform, token, last_seen
Many rows per user. Stale tokens get cleaned up when a gateway reports them invalid.
6. End-to-End Architecture
With channels and contact info in place, the central design is the path a notification takes from the service that wants to send it to the device that receives it. The key move is to put a queue between the request and the actual sending, so the fast, synchronous part (accepting the request) is decoupled from the slow, unreliable part (talking to the gateway).
A service triggers a notification; the notification servers authenticate the caller and apply rate limits, look up device and user data from cache and DB, render from the template store, and enqueue onto a per-channel queue. Workers pull from the queue, call the third-party gateway with retry on error, and the notification reaches the device. A notification log and analytics service track the message through send-pending, sent, and click events.
Reading the flow left to right:
Calling service. Any internal service — a payments service, a social feature, a marketing job — triggers a notification by calling the notification system's API with a user id, a channel (or "any available"), and the content or a template reference.
Notification servers. These are the front door. They authenticate the caller (only trusted services may send), apply rate limits, look up the recipient's device token / phone / email from the cache (backed by the DB of device settings and user info), render the content using the template store, and write the notification to the notification log. Crucially they do this quickly and then return — the real send happens asynchronously.
Per-channel message queue. The server enqueues the prepared message onto the queue for its channel (an iOS PN queue, an Android queue, an SMS queue, an email queue). The queue is the buffer that absorbs spikes and decouples the servers from the gateways.
Workers. A pool of workers pulls messages off each queue and makes the actual call to the third-party gateway. If the gateway returns an error, the worker retries (with backoff) rather than dropping the message.
Gateway and device. The worker calls APNs (or FCM, or the SMS/email provider), which delivers to the user's iOS device, Android device, phone, or inbox.
Analytics service. Throughout, an analytics service records lifecycle events — send-pending, sent, and click — so the whole flow is observable end to end.
function notify(caller, user_id, channel, template_id, params):
authenticate(caller) # reject untrusted services
category = templates.category(template_id) # e.g. "orders", "marketing"
if not should_send(user_id, channel, category):
return DROPPED # opted out or rate limited (see Settings)
contact = cache.get_contact(user_id) # falls back to DB
body = templates.render(template_id, params, contact.locale)
msg = build(notification_id=uuid(), user_id, channel, contact, body)
log.write(msg, status="send-pending")
queue[channel].enqueue(msg) # return fast; send is async
return ACCEPTED
7. Why a Queue per Channel
It is tempting to use one big queue for everything, but giving each channel its own queue is one of the highest-leverage decisions in the design. The reasons compound:
Isolation. If APNs is slow or down, the iOS queue backs up — but the email and SMS queues keep draining normally. One sick channel cannot poison the others, which a shared queue would allow.
Buffering. The queue absorbs bursts. A marketing campaign that triggers a million notifications at once does not overwhelm the gateways; the messages sit in the queue and drain at a sustainable rate.
Independent scaling. Channels have wildly different throughput and cost profiles. You can run many workers on the push queues and a smaller, carefully throttled pool on the SMS queue (where each message costs money), tuning each channel separately.
Backpressure. Queue depth is a direct, observable signal. A growing iOS queue tells you the gateway or the worker pool is the bottleneck, and it gives you a natural place to apply backpressure rather than failing requests at the front door.
The per-channel queue is what turns "we depend on four flaky external services" into "each flaky service is contained behind its own buffer." It is the structural reason a single gateway outage degrades one channel instead of taking the whole system down.
8. Reliability: No Lost Data
Persist intent through a business-transaction outbox, then use durable dispatch records. Scope keys by recipient/channel/logical notification and bind payload fingerprints. Atomically claim work. Bounded retries, expiry, and dead-letter/manual states are visible behavior, not a guarantee every device receives every intent.
A crash after remote acceptance but before local recording creates uncertainty. A lease cannot make a remote call and local write atomic. Use documented provider idempotency/status lookup where available and reconcile unknown states.
APNs apns-collapse-id and FCM collapse_key are coalescing features, not universal dedup or one-visible-alert guarantees. Email/SMS support varies; do not assume every provider accepts keys. Record accepted, delivered where measurable, read, failed, and expired separately. Check preferences, authorization, suppression, and expiry at dispatch; keep urgent and bulk queues bounded and isolated.
9. Templates
Most notifications are not bespoke one-off strings; they are the same message structure filled in with different values — "Your order #{order_id} has shipped," "{name} liked your post." Embedding that text inline in every calling service is a maintenance and consistency nightmare. A notification template store solves this by keeping reusable, parameterized content in one place.
A template is a named, versioned piece of content with placeholders. Callers reference a template by id and supply the parameters; the notification servers render the final body at send time. This buys several things at once:
Consistency. Wording, formatting, and branding live in one place and can be updated without redeploying every caller.
Localization. A template can have per-locale variants, so the same notification renders in the user's language using the locale stored with their contact info.
Per-channel rendering. The same logical notification can have a short SMS form and a richer email form, chosen by the channel.
# template: "order_shipped" with placeholders
templates["order_shipped"] = {
"push": "Your order {order_id} is on the way!",
"email": "Hi {name}, order {order_id} shipped and arrives {eta}.",
}
render("order_shipped", {order_id: 482, name: "Sam", eta: "Friday"}, channel="email")
# -> "Hi Sam, order 482 shipped and arrives Friday."
10. Settings and Rate Limiting
Just because you can send a notification does not mean you should. Two controls protect the user from being overwhelmed, and they are also what keep your sending reputation (and SMS bill) healthy.
User settings and opt-out. Users decide which categories of notification they want and on which channels — they may want push but not email, or order updates but not marketing. These preferences live in the settings DB and are checked before every send. A notification the user has opted out of is silently dropped rather than queued. Honoring opt-out is frequently a legal requirement, not just a courtesy.
Rate limiting. Even for opted-in users, you cap how many notifications a given user receives in a window so a buggy or overzealous caller cannot flood them. Rate limits apply per user and often per category, so a flurry of one type does not crowd out genuinely important alerts. Hitting the limit drops or defers the excess.
function should_send(user_id, channel, category):
prefs = settings.get(user_id)
if not prefs.opted_in(channel, category):
return False # respect opt-out
if not rate_limiter.allow(user_id, category):
return False # don't spam
return True
A user who gets spammed turns off notifications entirely — or marks your email as spam, which damages deliverability for everyone. Restraint is a feature: the settings and rate-limiting layer protects the long-term value of the channel.
11. Observability
Because the actual delivery happens inside third parties you do not control, you cannot reason about the system without recording what happened at each step. Two pieces give you that visibility.
Notification log. A durable record of every notification and its status transitions — send-pending → sending → sent → delivered, or failed. This is both the reliability backbone (it is what makes persist-first and dedup possible) and the audit trail you consult when a user asks "why didn't I get the alert?"
Analytics service. Aggregates the lifecycle events across all notifications to answer product and operational questions: how many were sent, how many were delivered, and how many were clicked. Click and delivery rates per template and per channel tell you what is working; a sudden drop in delivery rate on one channel is an early warning of a gateway problem.
Tracking sent, delivered, and clicked as distinct events matters because the gaps between them are diagnostic. A high sent-but-not-delivered rate points at the gateway or stale tokens; a high delivered-but-not-clicked rate points at content. Without these events you are flying blind through services you cannot otherwise inspect.
Where does "delivered" come from?
A 200 from APNs means only that Apple accepted the push; APNs sends no per-message delivery receipt. So "delivered" for iOS usually comes from your app: a notification service extension (or the app on open) calls your backend with the notification_id. FCM offers aggregate delivery data in its reporting tools. For email, providers report bounces and complaints by webhook; SMS providers send delivery receipts (DLRs) from carriers, which are not always reliable.
analytics.record(notification_id, "send-pending") # accepted, queued
analytics.record(notification_id, "sent") # handed to gateway
analytics.record(notification_id, "delivered") # app / provider receipt, if any
analytics.record(notification_id, "clicked") # user engagement
12. Failure Modes
What fails
Effect
Handling
APNs or FCM slow / down
Push queue grows.
Other channels unaffected (own queues). Workers back off with jitter; a circuit breaker stops hammering the gateway; alert on queue age.
Worker crashes mid-send
Message may be sent again.
Queue redelivers; dedup by notification_id; collapse id limits visible duplicates.
Caller-supplied idempotency_key; the server returns the first result.
Campaign floods the system
Urgent alerts delayed.
Separate priority queues and per-caller quotas.
Provider rate limit hit
429 / throttling errors.
Token-bucket limiter per provider in the workers, sized just under the provider's limit.
13. Tradeoffs & Alternatives
Decision
Option A
Option B
Usual choice
Delivery guarantee
At-most-once: never duplicate, may lose
Durable accepted intent; bounded retry/expiry and possible duplicate provider delivery under the defined failure model.
At-least-once plus dedup
Queues
One shared queue
One per channel (and priority)
Per channel and priority
Template rendering
At accept time (snapshot)
In the worker at send time (latest data)
Accept time for alerts; send time for long-delayed campaigns
Build vs buy
Own dispatch layer
Hosted service (for example Amazon SNS/SES, Twilio)
Buy early; build when volume, cost or control demand it
Preference checks
At the API server
Again in the worker
Both: preferences can change while a message waits
14. Summary
A notification system is a resilient dispatch layer in front of several unreliable external gateways. Its design is a handful of decisions that reinforce each other:
Concern
Mechanism
How do messages actually reach devices?
Third-party gateways per channel: APNs (iOS push), FCM (Android push), an SMS provider, an email provider.
Where do we send to?
Contact info — user id, device tokens, phone, email — collected at sign-up and install, stored in a DB.
How do callers trigger sends?
Notification servers authenticate the caller, apply rate limits, render the template, persist, and enqueue.
How do we stay isolated from flaky gateways?
A message queue per channel: isolation, buffering, independent scaling, and backpressure.
How do we avoid losing messages?
Persist first to the notification log, then retry on error with backoff.
How do we avoid sending twice?
Dedup on a unique notification id before calling the gateway.
How do we keep content consistent?
A template store with reusable, parameterized, localizable content.
How do we avoid spamming users?
User settings / opt-out plus per-user, per-category rate limiting.
How do we know what happened?
A notification log plus an analytics service tracking sent, delivered, and clicked.
The recurring theme: every part of the design exists to absorb the unreliability of services you do not control. Queues isolate them, persistence and retries survive them, dedup tames the retries, and the log and analytics make the whole opaque chain observable.
Worked example: the crash between send and record
Provider accepts n42, but the worker crashes before saving that acceptance.
Read top to bottom. The table below explains the state and guarantee at each step.
Step
State, calculation, and reasoning
1. Persist intent
Business state and n42's outbox/work record commit together; queue publication can retry.
2. Claim work
An atomic claim/lease prevents ordinary concurrent sends. Expiry or timeout does not prove the old remote send failed.
3. Send n42
Use a documented provider idempotency key where supported. Acceptance means queued for handling, not delivered/read.
4. Recover
Retry/query using the same remote identity. Without provider dedup/status, disclose the chosen duplicate-vs-loss tradeoff.
Check your understanding: Does a collapse key guarantee exactly one visible alert?
No. APNs/FCM collapse features coalesce appropriate pending notifications; they are not a universal exactly-once delivery protocol.
Run a small, testable model
Download concepts.py. It uses only the Python 3.10+ standard library. From the directory where you saved it, run:
python3 concepts.py workflow
python3 concepts.py test
The model keeps an uncertain remote outcome pending and retries the same logical operation identity.
The program contains deterministic assertions and failure traces. It is a learning model, not a production service: it does not implement network consensus, distributed durability, authentication, or third-party delivery. Read its contracts and try changing an input before extending it.
Channels, volume, campaigns vs alerts, loss vs duplicate, opt-out.
5–10 min
Estimation
Average rate, campaign burst, log size. Conclude: queues needed.
10–15 min
API + data
POST /notifications with idempotency key; users, devices, preferences, log.
15–25 min
High-level design
API servers → per-channel queues → workers → gateways; templates and settings.
25–40 min
Deep dives
Retries and dedup (and the crash window), priorities, token cleanup, rate limits.
40–45 min
Wrap-up
Monitoring (queue age, error rate per provider, click rate), quiet hours, build vs buy.
16. Common Mistakes
Sending inside the API request. A slow gateway then slows every caller. Accept, persist, queue, return 202.
Claiming exactly-once delivery. A crash between send and mark can repeat a push. Say at-least-once, deduped, with collapse ids.
Treating an APNs 200 as "delivered". It only means accepted by Apple.
One queue for all traffic. One broken channel or one big campaign blocks everything.
Retrying permanent errors. Invalid tokens and bad numbers never succeed; delete or flag them.
Forgetting opt-out and quiet hours. Users disable notifications, and marketing email without consent can break laws such as CAN-SPAM and GDPR.
17. Check Yourself
A campaign targets 50M users and must finish in 10 minutes. What send rate is that?
Answer
50,000,000 ÷ 600 s ≈ 83,000 per second, about 46 times the ~1.8K/s daily average.
Why give each channel its own queue?
Answer
Isolation (an APNs outage does not stall email), independent scaling and throttling (SMS costs money), and a clear backlog signal per channel.
The worker sends a push, then crashes before writing "sent". What happens and how do you limit the damage?
Answer
The message is retried and sent again. Pass the same collapse id (apns-collapse-id / FCM collapse_key) so the second push replaces the first on the device.
APNs returned HTTP 200. Was the notification delivered?
Answer
Not necessarily. 200 means APNs accepted it. The phone may be off. Delivery is known only if the app reports it back.
Why check user preferences in the worker too, not just at the API?
Answer
A message can wait in a queue for minutes during a campaign. If the user opts out in that time, the worker check stops the send.
A gateway says a device token is unregistered. Retry?
Answer
No. It is a permanent error. Remove the token from the devices table and stop sending to it.
18. Key Terms & Further Reading
APNs / FCM
Apple's and Google's push delivery services.
Device token
The per-app, per-device address a push service needs.
At-least-once
Retry until success; rare duplicates are possible.
Idempotency key
A client-generated identity for one logical operation. The server binds it to caller scope, endpoint, request fingerprint, state, and result. A retry uses the same key and reuses the original outcome rather than repeating the effect. Example and limits above.
Collapse key
Push header so a newer notification replaces an older one.
Exponential backoff
Waiting 1 s, 2 s, 4 s… between retries, with random jitter.
Circuit breaker
Stop calling a failing service for a while, then test it again.
Dead-letter queue
Where messages go after all retries fail, for inspection.
Alex Xu, System Design Interview – An Insider's Guide, Volume 1, "Design a Notification System".
Apple Developer documentation: "Sending notification requests to APNs" and "Handling notification responses from APNs" (status codes, apns-collapse-id).
