A system design interview walkthrough for building a real-time chat service like WhatsApp or Messenger: the connection protocol that carries messages, the split between stateless and stateful services, how chat history is stored and ordered, and how messages reach a recipient whether they are online or away.
Student edition · start with the guarantee
Deliver and synchronize messages across intermittent devices without confusing message identity with committed delivery order.
Correctness contract: A durably accepted logical message is replayable within retention; each device advances only a contiguous committed cursor, and duplicate sends share one result.
Basic HTTP, databases, and programming are assumed. Specialized terms are explained below. Named products motivate the problem; this is an educational design, not a claim about their current internals. Capacity numbers are illustrative decimal payload estimates, not benchmarks. Add indexes, protocol overhead, replicas, headroom, and failure capacity before provisioning.
A chat system looks deceptively simple from the outside — type a message, it shows up on someone else's screen — but it pushes on a specific set of design problems. The server has to push data to a client at an arbitrary moment, not just respond when asked. Messages must be durable so history survives a restart, ordered so a conversation reads correctly, and delivered fast when both parties are connected. And the same conversation may need to reach a recipient who is offline, on three devices, or part of a 500-person group. This guide builds the system up one decision at a time, in the order an interview tends to follow.
Each definition includes a concrete example and the condition under which the mechanism is useful. You can read this guide without opening another guide.
Sequence numbers and replay cursors
A sequence number records committed order within a stream. A cursor is a consumer's durable position in that stream, so it can resume without skipping missing events. An opaque identity is not automatically such a position.
Example. A device receives message sequence 10 then 12. It acknowledges contiguous 10, buffers 12, and fetches 11 before advancing to 12.
Limit. A Snowflake timestamp can arrive late behind an already-acknowledged cursor. Use per-stream commit order or a durable inbox offset, define retention gaps, and use separate device cursors.
Idempotency keys
A client-generated identity for one logical operation. The server binds it to caller scope, endpoint, request fingerprint, state, and result. A retry uses the same key and reuses the original outcome rather than repeating the effect.
Example. Payment p42 times out after the provider charged it. Retrying with p42 lets the provider return the same charge instead of creating a new one.
Limit. A unique local claim is insufficient if the business effect and saved result can diverge. Keep unknown outcomes pending, use downstream idempotency, and state the retention/retry horizon.
Fanout on write and read
Fanout distributes one event to many recipients. On-write fanout precomputes recipient inbox entries; on-read fanout merges source events when a recipient asks. A hybrid chooses using measured write amplification and read cost.
Example. One post to 200 followers creates up to 200 inbox inserts. A celebrity's million followers can instead read the author's shared stream when they open their feed.
Limit. Fanout is not atomic across inboxes. Persist the post and delivery work via an outbox, dedup references, catch up after misses, and recheck privacy/deletion at serving time.
Presence and failure detection
Presence is an estimate of recently observed connection activity. Heartbeats refresh session expiry, and a timeout suspects that a session is no longer live. A user may have several sessions/devices.
Example. A phone disconnects while its laptop keeps sending heartbeats. The user remains online if the product defines online as any active session.
Limit. Timeouts cannot distinguish a dead process from a paused or partitioned one. Counts should say sessions, users, or recently active viewers; privacy and stale snapshots need explicit policy.
Transactional outbox
A business transaction writes its state change and an event row to the same database. A relay publishes committed event rows later, retrying until the broker durably accepts them.
Example. An order and its OrderCreated outbox row commit together. A relay crash after publish but before marking sent publishes the same event ID again on restart.
Limit. The relay is at-least-once, not magically exactly-once. Consumers dedup transactionally; preserve per-entity sequence when order matters. CDC is change data capture: reading committed database changes.
Pub/sub and durable queues
Publish/subscribe routes an event to interested subscribers; it may be ephemeral or durable. A queue/log with retained records and offsets allows replay after a consumer is disconnected.
Example. An online gateway misses a comment while restarting. Ephemeral Redis Pub/Sub cannot replay it; a durable comment log can fetch after the gateway's last sequence.
Limit. Pub/sub alone does not promise persistence, global order, or exactly-once effects. Broker choice, acknowledgements, retry horizon, and client catch-up protocol define those guarantees.
1. Requirements
Before drawing boxes, pin down what the system must actually do. A focused feature list keeps the design honest and gives the interview a clear scope.
Requirement
What it means
1:1 chat
Two users exchange text messages in near real time, with low end-to-end latency when both are connected.
Group chat
A message sent to a group is delivered to every member. Group sizes are bounded (say, up to a few hundred) to keep fanout tractable.
Online presence
Users can see whether their contacts are online, away, or last seen at some time.
Push when offline
If a recipient is not connected, the message is still delivered later, and a push notification alerts them on their device.
Message history
Messages are persisted durably and can be re-read across sessions and devices, ordered by time.
The non-functional shape matters just as much. Chat is write-heavy (every message is a write, and group messages multiply that), latency-sensitive on the delivery path, and must keep a persistent connection open per active client. History grows without bound, so storage has to scale horizontally and read access is dominated by recency — people scroll the most recent messages far more often than ancient ones.
A good early framing question: is this a small-group messenger or a large broadcast system? The answer caps group size and fanout strategy. This guide assumes bounded groups, where per-member fanout is acceptable.
Clarifying questions to ask
1:1 only, or groups too, and how big? Text only, or images and video? Which platforms (mobile, web)? How many daily users? Do we keep history forever? Do we need read receipts, typing indicators, end-to-end encryption? Assume: 1:1 and groups up to 500, text first, 500M DAU, history forever, sent/delivered/read ticks.
Non-functional goal
Target
Delivery latency (both online)
Under ~200 ms end to end, same region
Durability
Acknowledged messages are durable under the documented replica/failure/retention contract, and recoverable through the accepted history.
Delivery guarantee
At least once on the wire, exactly once on screen (dedup by message id)
Ordering
Same order for every member of a conversation
Availability
Sending keeps working when some servers fail
2. Back-of-the-Envelope Estimation
Quantity
Assumption
Result
Messages per day
500M DAU × 40 messages
20 B/day
Average write rate
20B ÷ 86,400 s
~231K msg/s
Peak write rate
×2 for evening peaks
~463K msg/s
Storage per day
20B × ~200 bytes (text + metadata)
~4 TB/day
Storage per year
4 TB × 365 (×3 replicas ≈ 4.4 PB)
~1.46 PB/yr
Concurrent connections
30% of DAU online at peak
~150M sockets
Chat servers
150M ÷ 100K sockets per server
~1,500 servers
Memory per server
100K × ~10 KB per connection
~1 GB
Heartbeats
150M ÷ one every 30 s
~5M/s
Takeaway
Connection count, per-socket memory, fanout, TLS, heartbeats, bandwidth, and measured CPU together determine gateway capacity; a socket-only estimate does not prove one server meets the load.
shard by conversation
Intuition
A chat server is like a hotel switchboard. It does very little work per call, but it must keep a line open to every guest. That is why we count sockets and memory, not requests per second, to size the fleet.
3. API Design
Stateless calls go over plain HTTPS. Live traffic goes over the WebSocket as small JSON (or protobuf) frames.
# HTTPS (stateless tier)
POST /v1/login -> { token, chat_server_url }
GET /v1/conversations/{cid}/messages?before=<msg_id>&limit=50 # scroll back
POST /v1/groups { name, member_ids } -> { group_id }
# WebSocket frames (stateful tier)
client -> server { type:"send", client_msg_id:"c-9f2e", cid, body }
server -> client { type:"ack", client_msg_id:"c-9f2e", msg_id, seq } # stored: 1 tick
server -> client { type:"msg", cid, msg_id, seq, from, body }
client -> server { type:"delivered" | "read", cid, up_to_seq } # 2 ticks / blue
client -> server { type:"ping" } # heartbeat
The client_msg_id is made on the phone before sending. If the network drops and the app resends, the server sees the same id and does not store a second copy.
A 1:1 conversation id can be derived from the two user ids, for example min(a,b) + ":" + max(a,b). Both users then find the same conversation without a lookup.
5. Connection Protocols
The defining challenge of chat is the server-to-client direction. A client can always open a request to send a message, but how does the server deliver a message that arrives for a client at an unpredictable time? Plain request/response does not push. Three techniques bridge that gap, and they sit on a spectrum from wasteful to efficient.
Technique
How it works
Tradeoff
Polling
The client asks the server "anything new?" on a fixed interval and the server answers immediately, empty or not.
Simple, but most polls return nothing — wasted requests. Poll too often and you burn resources; poll too rarely and messages are delayed.
Long polling
The client asks, and the server holds the request open until it has something to send or a timeout expires, then the client immediately re-asks.
Far fewer empty responses, but each message needs a fresh HTTP request, connections are held server-side, and there is still no clean server-initiated path — the server cannot push to a client that is between requests.
WebSocket
A single long-lived, full-duplex TCP connection, upgraded from HTTP once at the start. Either side can send a frame at any time.
One persistent connection per client to manage, but messages flow in both directions with minimal overhead and true server push.
WebSocket is the preferred choice for the message path. After a one-time HTTP Upgrade handshake, the connection stays open and becomes bidirectional: the client sends outgoing messages over it, and the server pushes incoming messages down the same socket the instant they appear. There is no polling delay and no repeated handshake cost. The price is that the connection is stateful — the server must keep it alive and remember which user is on the other end — which shapes the rest of the architecture.
# client opens one persistent socket and reuses it both ways
ws = websocket_connect("wss://chat.example.com") # HTTP Upgrade once
ws.on_message(msg -> render(msg)) # server pushes inbound
ws.send(outbound_message) # client sends outbound# the same socket carries both directions until it closes
Polling and long polling are still fine for low-frequency, mostly-client-initiated features. WebSocket earns its keep specifically because chat needs frequent, low-latency, server-initiated delivery.
6. Service Decomposition
The WebSocket requirement forces a clean split in the backend. Most of the system is ordinary request/response work that any web tier handles well. Only the live connection is special. Separating the two lets each scale on its own terms.
Tier
Examples
State
Stateless services
Authentication, user profile, contacts, group management, the general API.
Hold no per-connection state. Any instance can serve any request, so they sit behind a load balancer and scale by adding identical replicas.
Stateful chat service
The WebSocket servers that terminate live client connections.
Each one holds a set of open sockets and knows which user owns each. A client is bound to one specific server for the life of its connection.
The stateless tier is the easy part: a load balancer spreads requests across interchangeable instances, and login, profile lookups, and group edits all flow through it. The chat service is the hard part precisely because it is sticky. Once a client establishes its WebSocket to a particular chat server, every message for that user must be delivered through that server, because that is where the socket physically lives. You cannot round-robin an inbound message to a random instance and hope it lands on the one holding the connection.
This stickiness is why we need two more pieces that a stateless system would not: a way to find which chat server holds a given user's connection (service discovery), and a way to route a message from the sender's chat server to the recipient's chat server (the message sync queue, the message flow).
7. Storage and Message IDs
Chat history has an unusual access shape, and it points firmly at a key-value / NoSQL store rather than a relational database.
Enormous volume. Every message from every user, kept indefinitely. The dataset only grows, so it must shard horizontally across many nodes.
Write-heavy. Each message is a write, and group messages amplify that. The store must absorb a high, steady write rate without contention.
Accessed by recency. Reads are dominated by "give me the latest messages in this conversation" and scrolling backward from there — a range scan over recent keys, not arbitrary joins or analytics.
Keyed by conversation, then message id. The partition key is the conversation (channel) id, so one conversation lives together on one shard; inside it, rows are sorted by message id. "Latest 50 in this chat" becomes one cheap range read. There are no complex relational joins that would justify a SQL engine's overhead.
A NoSQL store fits all four: it scales out, handles heavy writes, and serves recency-ordered range reads cheaply. The remaining question is the message id, and it carries two hard requirements at once. The id must be globally unique (no two messages collide) and sortable by time (sorting by id sorts the conversation chronologically), so that ordering a thread is just "read keys in order" with no extra timestamp field to coordinate.
An auto-increment column from a single database would give ordering but does not scale across shards. A random UUID scales but loses time ordering. The standard answer is a Snowflake-style 64-bit id: a single integer packed from a timestamp in the high bits, a machine/shard id in the middle, and a per-millisecond sequence number in the low bits.
# 64-bit id: time-ordered AND unique without coordination
id = (timestamp_ms << 22) # high bits: sorts by time
| (machine_id << 12) # which generator node
| sequence_number # disambiguates within the same ms# sorting ids ascending ~= sorting messages chronologically# (exact only per generator; clocks on different machines drift)
Because the timestamp occupies the most significant bits, simply sorting ids ascending yields chronological order, while the machine id and sequence number guarantee uniqueness even when many servers generate ids in the same millisecond. That is exactly the property the conversation read path wants.
Watch out: clock skew
Snowflake ids from different machines are only roughly time-ordered, because server clocks drift by milliseconds. Two messages sent a moment apart in the same chat can get ids in the "wrong" order. Within one conversation that matters, so many real systems add a per-conversation sequence number (1, 2, 3, …) assigned by the one server that owns the conversation. It gives a strict order inside the chat, which is all users ever see.
8. Service Discovery
With connections pinned to specific chat servers, the system needs a directory: which chat server currently holds a given user's WebSocket? This is the job of a service discovery / coordination component, commonly ZooKeeper (or an equivalent) for the list of healthy chat servers, plus a fast key-value store such as Redis for the per-user connection map.
Service discovery has two responsibilities. First, when a client connects, it picks the best chat server for that client — typically the geographically closest one with capacity — and records the client → chat-server mapping. Second, it exposes that mapping so any part of the system can look up where a user is connected and route a message to the correct server. When a client disconnects or fails over, the mapping is updated so stale routes do not linger.
Two different directories
The server list is small and changes rarely, which suits ZooKeeper's strongly consistent, low-write design. The user → server map has one entry per online user (about 150M in our estimate) and changes every time a phone reconnects. That churn belongs in a sharded in-memory KV store with a TTL (for example 60 s, refreshed by heartbeats), so a crashed server's entries expire on their own.
# on connect: choose a server and register the mapping
server = discovery.pick_chat_server(user, region) # closest with capacity
discovery.register(user_id, server) # user -> server# on delivery: find where the recipient is connected
target = discovery.lookup(recipient_id) # which chat server?
if target is None:
route_to_offline_path(recipient_id) # they are not online
Service discovery is what makes the stateful tier usable. Without a reliable user-to-server map, the system has no way to send an inbound message to the one server that can actually deliver it.
9. The 1:1 Message Flow
Now the pieces connect. The diagram below traces a single message from User A to User B, numbered step by step. Each number corresponds to one of the components introduced above.
A message travels from User A to Chat server 1 (1) and gets a time-sortable id plus a per-conversation sequence number (2). It enters the message sync queue, partitioned by conversation id (3), and is written to the KV store under (conversation id, message id) (4). The Redis connection map, whose entries expire on a TTL, says where User B is connected: their own chat server if online (5.a, 6) or the push notification servers if offline (5.b).
Walking the numbered path:
(1) User A → Chat server 1. User A sends the message over their open WebSocket to the chat server that holds their connection.
(2) Chat server 1 → ID generator. The chat server requests a unique, time-sortable message id from the Snowflake-style ID generator. This stamps the message with its place in the conversation's order (see the clock-skew note in the Storage section).
(3) Chat server 1 → message sync queue. The message, now with an id, is handed to the message sync queue. This queue decouples receiving a message from delivering and persisting it, absorbing bursts and acting as the routing backbone between chat servers.
(4) → KV store. The message is written to the key-value store under (conversation id, message id), so it becomes durable history immediately — independent of whether the recipient is currently reachable.
(5.a) Online path → Chat server 2 → (6) User B. If User B is online, service discovery says which chat server holds their connection (Chat server 2). The message is routed there and pushed down B's WebSocket in real time.
(5.b) Offline path → PN servers. If User B is not connected, there is no socket to push to. The message goes to the push notification (PN) servers, which alert B's device. The message is already safely in the KV store, so B receives it the next time they come online and sync.
The key insight is that persistence (step 4) happens regardless of presence. Delivery is best-effort and immediate when possible, but durability is guaranteed first. Whether the recipient is online only changes how they are reached — a live socket versus a push notification plus a later sync — never whether the message is kept.
10. Delivery Guarantees and Receipts
The sender retains an outbox until server acknowledgement under the chosen durable-replica/failure contract. Namespace client_msg_id by sender/conversation and claim it atomically with persistence and sequence assignment. A lost acknowledgement retries the same identity, not a new message.
Server acceptance, device receipt, and user reading are different events. A cumulative device receipt covers only a contiguous stored prefix; read receipts are separate. Reconnection fetches durable history; gateway push is an optimization. Idempotent storage exposes one logical message within a defined retention horizon, not survival of every conceivable fault.
Persist group membership decisions and pending fanout via transactional work/outbox records. A crash after saving the message but before enqueueing every member must not strand the remaining recipients.
11. Online Presence
Presence — showing whether a contact is online, away, or last seen at a time — rides on the same persistent connection. The chat server uses a heartbeat to know the connection is alive: the client periodically sends a small keep-alive frame, and the server resets a timer each time one arrives.
If heartbeats keep coming, the user is online. If they stop for longer than a threshold (the socket dropped, the app was backgrounded, the network died), the server marks the user offline and records a last seen timestamp. Using a threshold rather than reacting to the first missed beat avoids flapping a user's status on a brief network blip.
# server tracks liveness per connection
on heartbeat(user):
last_seen[user] = now()
status[user] = ONLINE
every few seconds: # sweep
for user where now() - last_seen[user] > THRESHOLD:
status[user] = OFFLINE
fanout_presence(user, OFFLINE, last_seen[user])
When a status changes, it is fanned out to that user's contacts — but only to contacts who are themselves online and would actually display it, via a presence/fanout path much like the message path. Pushing presence to everyone all the time would be wasteful, so the fanout is scoped to interested, connected viewers. For very large contact lists, presence is often fetched on demand (when you open a conversation) rather than pushed eagerly.
12. Group Chat
Persist one message with a per-conversation committed sequence, then fan out references to member inboxes through durable, retryable work. Message and fanout intent commit together; inbox insertion is idempotent. Define membership at send time and historical access after join/leave.
Devices render canonical sequence order and repair gaps, not just arrival order or timestamp-ID order. Independent fanout can be delayed/duplicated. Bounded groups can use per-member inboxes; large channels may use shared logs and on-read catch-up. A shared ID or a loop over members alone does not guarantee complete, ordered delivery.
13. Multi-Device Sync
Keep a separate durable sync cursor for each device, scoped to a committed inbox or conversation stream. Resume by sequence/offset, not an assumed globally monotonic Snowflake ID. A timestamp-sorted ID can arrive or commit behind an already-seen higher ID and be skipped by a naive id > last_seen query.
Acknowledge the highest contiguous stored sequence. Receiving 12 while 11 is missing must not advance a contiguous cursor from 10 to 12. Fetch the gap, dedup replay, and only then advance. If retention removed the needed prefix, return an explicit resync/snapshot requirement. Membership rules govern history access after join/leave; authorize replay as well as live delivery.
14. Failure Modes
What fails
Effect
Handling
A chat server crashes
~100K sockets drop at once.
Clients reconnect with jittered backoff to another server, re-register in the session store, and resume from their last seq. Stale session entries expire by TTL.
Message store shard slow or down
Sends in those conversations cannot be acked.
Replicated store (for example 3 replicas, quorum writes). The client keeps the message in its outbox and retries; no ack means no lie to the user.
Recipient's server lookup is stale
Push goes to a server that no longer has the socket.
That server drops it; the message is already stored, so the recipient gets it on reconnect sync. Delivery is best effort, storage is not.
Duplicate send after timeout
Two copies could appear.
Dedup on client_msg_id at the server and on msg_id at the client.
Push notification provider down
Offline users are not alerted.
Queue and retry the push; messages are still in history, so nothing is lost, only delayed.
Reconnect storm after a network blip
Millions of clients reconnect in seconds.
Random jitter on reconnect, connection rate limits at the load balancer, spare server capacity.
15. Tradeoffs & Alternatives
Decision
Option A
Option B
Usual choice
Server push
Long polling: works everywhere, more overhead
WebSocket: one socket, low latency
WebSocket, long polling as fallback
Message order
Snowflake id: no coordination, approximate
Per-conversation sequence: strict, needs one owner
Sequence per conversation
Group delivery
Fan out a copy per member inbox
Members read one shared group log
Per-member for small groups, shared log for huge ones
History store
Relational DB: familiar, harder to scale writes
Wide-column KV: scales writes, simple queries
Wide-column KV (Cassandra, HBase)
Encryption
Server can read messages (search, moderation)
End-to-end: server sees only ciphertext
Product choice; E2E makes server-side search impossible
16. Summary
A chat system is a handful of decisions that build on each other, each one made to solve the previous one's consequence:
Concern
Mechanism
How does the server push to a client?
A persistent, bidirectional WebSocket connection per active client, chosen over polling and long polling.
How do services scale?
Stateless services (auth, profile, API) behind a load balancer; a separate stateful chat service holds the live sockets.
Where is history stored?
A key-value / NoSQL store: enormous, write-heavy, read by recency, partitioned by conversation id and sorted by message id.
How are messages ordered uniquely?
A 64-bit Snowflake-style id — timestamp in the high bits makes ids unique and sortable by time.
How is a user's chat server found?
ZooKeeper tracks healthy chat servers; a Redis-style KV store with TTLs holds the user → chat-server map.
How does a 1:1 message travel?
Chat server → id generator → message sync queue → KV store → recipient's chat server (online) or push servers (offline).
How is presence tracked?
Heartbeats detect online/offline; status changes fan out to connected contacts.
How does group chat differ?
Fanout: one copy enqueued into each member's per-user queue, then delivered like 1:1.
How do multiple devices stay in sync?
Per-user message sync queue with a per-device last-seen cursor; each device replays what it missed.
The recurring theme: persist first, deliver second. Durability through the message sync queue and KV store never depends on the recipient being reachable; presence and connection state only decide how a message is delivered — a live socket, a push notification, or a later catch-up sync — never whether it survives.
Worked example: message 12 arrives before message 11
The device has stored committed conversation sequences through 10.
Read top to bottom. The table below explains the state and guarantee at each step.
Step
State, calculation, and reasoning
1. Receive sequence 12
Buffer it. Do not acknowledge a contiguous cursor of 12 while message 11 is missing.
2. Reconnect after a gateway crash
Ask the durable stream for entries after contiguous cursor 10, not for IDs greater than the maximum Snowflake seen.
3. Receive 11 and replay 12
Dedup stored IDs; contiguous state can now advance through 12. Preserve device-specific cursors.
4. Retry a send with a lost ACK
Reuse (sender,conversation,client_msg_id). A transactional unique constraint returns the original committed message and sequence.
Check your understanding: Why is a highest-seen timestamp not a safe resume cursor?
A delayed or concurrently committed message can have an older timestamp/ID. A cursor must represent a complete committed prefix, not approximate time.
Run a small, testable model
Download concepts.py. It uses only the Python 3.10+ standard library. From the directory where you saved it, run:
python3 concepts.py delivery
python3 concepts.py test
The model buffers a gap, deduplicates replay, and verifies that acknowledgement does not skip a missing entry.
The program contains deterministic assertions and failure traces. It is a learning model, not a production service: it does not implement network consensus, distributed durability, authentication, or third-party delivery. Read its contracts and try changing an input before extending it.
1:1 and group size, history, receipts, DAU. State "at least once, dedup to exactly once on screen".
5–10 min
Estimation
Messages/s, storage/year, concurrent sockets → server count.
10–15 min
API + data model
WebSocket frames with client_msg_id; messages keyed by (cid, seq).
15–25 min
High-level design
Stateless vs stateful tiers, discovery, the 1:1 flow with online and offline paths.
25–40 min
Deep dives
Acks and retries, ordering, group fanout, presence cost, multi-device cursors.
40–45 min
Wrap-up
Failure handling, E2E encryption as an extension, metrics (send-to-deliver latency, reconnect rate).
18. Common Mistakes
Using HTTP polling for everything. Fine for a toy, but it wastes requests and adds seconds of delay at scale.
Acking before the message is durable. If the server crashes, the sender sees a tick for a message that is gone.
No dedup id. Retries are required for reliability, and without an id they create duplicates.
Trusting timestamps for order. Clocks differ between phones and between servers; use a server-assigned sequence per conversation.
Fanning out presence to every contact. With large contact lists that is a storm of tiny writes; send only to online contacts who are looking, or fetch on demand.
Keying messages only by message id. Then "latest 50 in this chat" touches many shards. Partition by conversation.
19. Check Yourself
Why is the chat service stateful while the profile service is not?
Answer
A chat server holds open WebSockets. A user's socket lives on exactly one server, so messages for that user must go to that server. Profile requests can go to any instance.
With 150M concurrent users and 100K sockets per server, how many chat servers do you need?
Answer
150,000,000 ÷ 100,000 = 1,500 servers, plus spare capacity for failures and reconnect storms.
The sender's phone times out and resends a message. How does the system avoid storing it twice?
Answer
The phone reuses the same client_msg_id. The server sees it has already stored that id for this sender and returns the original ack instead of writing again.
Why might two Snowflake ids from different servers be out of order in real time?
Answer
The timestamp bits come from each server's own clock, and clocks drift by milliseconds. A per-conversation sequence number from one owner fixes the order inside a chat.
User B is offline when A sends. Walk through what happens.
Answer
The message is stored and acked to A. Discovery finds no socket for B, so a push notification is sent. When B opens the app, the device sends its last seq and syncs everything newer.
Why does a device ack "up to seq 812" instead of acking each message?
Answer
It is cheaper (one small frame for many messages), and it doubles as the device's cursor: on reconnect the server resends everything after 812.
20. Key Terms & Further Reading
WebSocket
A long-lived, two-way connection upgraded from HTTP (RFC 6455).
Stateful service
A server that holds per-client data, here open sockets, so requests must reach a specific instance.
Service discovery
A directory that says which server holds which user's connection.
At-least-once delivery
Resend until acknowledged; duplicates are possible and removed by id.
Idempotency key
A client-generated identity for one logical operation. The server binds it to caller scope, endpoint, request fingerprint, state, and result. A retry uses the same key and reuses the original outcome rather than repeating the effect. Example and limits above.
Heartbeat
A small periodic ping that proves a connection is alive.
Snowflake id
A 64-bit id built from time, machine id and a counter.
Sync cursor
The last sequence number a device has, used to resume after reconnect.
Alex Xu, System Design Interview – An Insider's Guide, Volume 1, "Design a Chat System".
IETF RFC 6455, "The WebSocket Protocol".
Martin Kleppmann, Designing Data-Intensive Applications, Chapter 9: ordering, sequence numbers and why clocks are unreliable for order.
Twitter Engineering blog, "Announcing Snowflake" (2010): the original 64-bit id scheme.
