A system design interview guide to building a service that follows a shipment from the moment a label is printed through to delivery, ingesting a stream of scan events from carriers, keeping a current status and a full history, predicting arrival, and serving fast "where is my package?" lookups to millions of customers.
Student edition · start with the guarantee
Turn messy carrier scans into a trustworthy shipment history and a reproducible current-state projection.
Correctness contract: Events are source-scoped, identity/payload-bound, and durably retained under policy. Projection ordering/ties/corrections are deterministic; late prerequisites can trigger replay.
Basic HTTP, databases, and programming are assumed. Specialized terms are explained below. Named products motivate the problem; this is an educational design, not a claim about their current internals. Capacity numbers are illustrative decimal payload estimates, not benchmarks. Add indexes, protocol overhead, replicas, headroom, and failure capacity before provisioning.
Package tracking is a deceptively good interview problem because it is, at heart, an exercise in turning an unreliable, out-of-order stream of external events into a clean, queryable model of the world. A shipment does not move through your system; it moves through trucks, planes, and sorting hubs you do not control, and all you ever see are scans — discrete observations that someone, somewhere, handled the package. Those scans arrive late, occasionally out of order, sometimes more than once, and you have to fold them into a single authoritative answer to two questions a customer keeps asking: where is my package right now, and when will it arrive? This guide builds up a design that treats the shipment as a state machine, ingests events idempotently, keeps both a current state and an append-only history, recomputes an ETA as new information lands, and serves the read path from a store tuned for speed.
Each definition includes a concrete example and the condition under which the mechanism is useful. You can read this guide without opening another guide.
State machines and projections
A state machine specifies legal states and transitions. An event-sourced projection derives read state from a durable event history under a deterministic ordering and conflict policy.
Example. A shipment can go from transit to exception and back; it is not a simple increasing status integer. A late prerequisite scan may require rebuilding the projection.
Limit. Keeping only the newest legal event can lose an earlier rejected transition forever. Timestamps need deterministic tie handling and source trust; append-only storage still has retention/privacy rules.
Event time and watermarks
Event time is when the producer says an event happened; processing time is when a consumer handles it. A watermark is the processor's policy-driven estimate of event-time progress, used to emit and eventually clean up window state.
Example. An event for 12:00 arrives at 12:07. A 5-minute allowed-lateness policy may update its window, divert it to correction, or reject it according to the API contract.
Limit. Watermarks are not proof all earlier events arrived. Idle inputs, skew, and unbounded delay require explicit policies; later corrections must propagate to rollups and reports.
Deduplication
Recognizing multiple deliveries of one logical event using a stable identity. The same logical event must keep its ID across retries, and claiming the ID must be atomic with the counted or stored effect.
Example. Events e1,e2,e1 should yield a count of 2. An exact seen set plus the increment in one transaction makes the second e1 a no-op.
Limit. A TTL limits the dedup horizon; retries after expiry may count again. Approximate filters may drop new events, and changing the ID or routing a retry to another local seen set defeats dedup.
Optimistic concurrency and compare-and-set
Update state only if the expected version or status still matches, in the same atomic operation that changes it. A failed comparison means a conflicting operation won; reread or return a conflict.
Example. UPDATE seats SET status='held', version=version+1 WHERE id=? AND status='available' allows only one concurrent claimant to update the row.
Limit. Application-level GET then unconditional SET is not equivalent. Include acquisition identity to reject stale callbacks, and couple related invariants in a transaction.
Transactional outbox
A business transaction writes its state change and an event row to the same database. A relay publishes committed event rows later, retrying until the broker durably accepts them.
Example. An order and its OrderCreated outbox row commit together. A relay crash after publish but before marking sent publishes the same event ID again on restart.
Limit. The relay is at-least-once, not magically exactly-once. Consumers dedup transactionally; preserve per-entity sequence when order matters. CDC is change data capture: reading committed database changes.
ETA and uncertainty
An estimated time of arrival combines location, route, historical travel time, and current conditions. A distance-only estimate is not a road-network travel-time estimate; predictions should include uncertainty and freshness.
Example. A nearby driver across a river may take longer than a farther driver already on the correct road. Rank by road ETA after cheap geographic candidate filtering.
Limit. GPS timestamps can be stale or wrong. State the location age, update version, cancellation policy, and fallback when routing or traffic data is unavailable.
1. Requirements and Clarifying Questions
Who sends the data? Many carriers and our own hubs, by webhook or polling [yes].
What does the customer see? Current status, last location, ETA and full scan history [yes].
Live GPS of the van on a map? Nice to have for the last mile; mention only.
Notifications? Push, SMS or email on key transitions [yes].
Scale? 50 million new packages a day, about 10 scans each, 5 days average in transit.
Functional
Non-functional
Ingest scan events from carriers and hubs.
Never lose an accepted scan (durable history).
Show current status, ETA and history for a tracking number.
Reads fast (< 100 ms) and highly available, even during spikes.
Notify the customer on meaningful changes.
Status is eventually consistent; seconds of lag are fine.
Flag packages that are stuck or in an exception.
Correct under duplicate and out-of-order events.
Intuition
Think of a package as a board-game token and each scan as a card someone mails you about where it moved. The cards arrive late, some twice, some out of order. You keep every card in a shoebox (history) and keep one token on the board (current state), moving it only when a card is newer than the last one you acted on.
2. Back-of-the-Envelope Estimation
Quantity
Working
Result
New packages per day
assumption
50 M
Scans per day
50 M × 10 scans
500 M
Average scan writes
500 M / 86,400 s
≈ 5,800/s
Peak scan writes
avg × 3 (evening hub batches)
≈ 17,000/s
Average tracking reads
50 M × 5 lookups / 86,400 s
≈ 2,900/s (spikes far higher)
History growth
500 M × ~300 B
≈ 150 GB/day
History per year
150 GB × 365
≈ 55 TB/yr
Active packages
50 M/day × 5 days in transit
250 M rows
Current-state store
250 M × ~500 B
≈ 125 GB
The current-state store fits on a small sharded cluster and caches well. History is the big one: 55 TB a year is too much for a hot database, so keep recent months hot and move older events to cheap object storage. Reads are modest on average but spike hard (a sale day, a delayed shipment everyone refreshes), which is why the read path is cached and separated from ingestion.
3. API and Data Model
# carriers and hubs (authenticated, idempotent by event_id)
POST /v1/scans [ { "event_id": ..., "tracking_id": ..., "status_code": ..., "scanned_at": ... }, ... ]
-> 202 Accepted
# customers
GET /v1/track/{tracking_id} -> { status, location, eta_window, updated_at, recent_events: [...] }
GET /v1/track/{tracking_id}/history -> { events: [...] } # full journey, paged
PUT /v1/track/{tracking_id}/subscribe { "channel": "sms", "to": "+1..." }
scan_history (append-only, partitioned by tracking_id)
tracking_id TEXT
scanned_at TIMESTAMP
event_id TEXT UNIQUE -- dedup key
status_code TEXT
facility TEXT
received_at TIMESTAMP
PRIMARY KEY (tracking_id, scanned_at, event_id)
shipment_state (one row per package)
tracking_id TEXT PRIMARY KEY
status TEXT -- CREATED, IN_TRANSIT, ...
location TEXT
last_event_at TIMESTAMP -- event time of the scan that set status
eta_start TIMESTAMP
eta_end TIMESTAMP
Tip
Partition both tables by tracking_id. All events for one package then land on the same shard, so the history insert and the state update can share one local transaction, and "show me this package's journey" is a single-partition range scan.
4. The Package Lifecycle
A shipment projection uses named states and legal transitions, but carrier evidence is messy and the real path is not a strictly increasing status code. Distinguish Created (label/intent), Picked up (custody), In transit, Out for delivery, Delivered, Exception, Returning, and Returned (return physically completed).
Exceptions can recover into transit. A return instruction is not proof a package is already returned. A delivery scan can later be superseded by an explicit misdelivery/correction event. Some carriers skip nominal scans or provide repeated hub observations, so the projection policy must interpret evidence rather than discard every nonadjacent transition.
Define event identity, authoritative source/version where available, deterministic time ties, correction linkage, and legal/deferred transition rules. Keep raw normalized history under retention policy and rebuild affected projections when older prerequisites or corrections arrive. This state machine models our accepted evidence, not perfect knowledge of the package's physical state.
5. Ingesting Scan Events
Everything the system knows comes from scan events emitted by carriers and sorting hubs. Each scan is a small record saying "package X was observed at location Y at time Z with status code C." Some carriers push these to you via webhook as they happen; others expect you to poll their API on a schedule. Either way, the ingestion layer has to accept a high-volume, bursty, and fundamentally messy stream.
A representative scan event looks like this:
{
"event_id": "scan-9f2c-0042", # unique per scan, used for dedup
"tracking_id":"1Z-TRACK-7788",
"status_code":"DEPARTED_HUB", # maps to a lifecycle transition
"facility": "Memphis Hub",
"lat_lng": [35.04, -89.98],
"scanned_at": "2026-06-29T14:03:00Z", # event time, NOT receive time
"received_at":"2026-06-29T14:07:11Z"
}
Three properties of this stream shape the rest of the design, and it is worth naming them explicitly:
Out of order. Network delays, batched uploads, and clock skew mean a later scan can arrive before an earlier one. You cannot assume received_at order matches scanned_at order, so transitions must be applied on event time, not arrival time.
Duplicated. Webhooks retry, polls overlap, and hubs occasionally double-scan. The same logical event can show up two or three times, so ingestion must be idempotent.
Untrusted in shape. Status codes vary by carrier, fields go missing, timestamps are malformed. Every event has to be validated and normalized into your own vocabulary before it touches the state machine.
6. End-to-End Architecture
The architecture follows the flow of an event: carriers and hubs emit scans, an ingest service validates and applies them, the result is written to a current-state store and appended to an immutable history, an ETA service recomputes the predicted arrival, and the customer-facing read path is served from a store tuned for fast lookups by tracking number.
Carriers and hubs send scan and GPS events to the ingest service, which validates them. Event history and current state live in one Tracking DB, partitioned by tracking number, and each event writes both in a single transaction. After commit the ETA service recomputes arrival, the notifier pushes status changes to customers, and a fast read store serves the customer app's lookups by tracking number.
Walking the components in the order an event touches them:
Carriers / hubs. The external sources of truth about physical reality. They push webhooks or expose polling APIs. You treat them as untrusted and unreliable, exactly as a notification system treats its gateways.
Ingest service. The single entry point for all events. It validates the payload, normalizes the carrier-specific status code into your lifecycle vocabulary, checks the event id for duplicates, and decides whether this event advances the state machine. It is the only component allowed to mutate the state store.
Event history. An append-only log of every accepted scan. Nothing here is ever updated or deleted. This is the audit trail and the source from which current state can always be rebuilt.
State store. The current state of every shipment — one row per tracking number with its present status, last known location, and current ETA. This is small, hot, and frequently read.
ETA service. Given the latest location and the route, it predicts the arrival window and writes it back into the state store.
Notifier. When a transition is meaningful to the customer (picked up, out for delivery, delivered, exception), it pushes a status update — reusing the kind of fan-out notification pipeline covered elsewhere.
Read store. A denormalized, read-optimized view that the customer app hits. Keeping reads off the write path means a traffic spike of "where is my package?" lookups cannot slow down event ingestion.
7. Current State vs. History
Keep a durable event history plus a versioned current-state projection. Each event has tracking/source identity, event ID, event time, receive time, source sequence where available, normalized meaning, and correction linkage. Current state includes projection/rule version and last incorporated evidence.
Rebuild from deterministic ordered evidence when late scans/corrections change interpretation. A strict state machine may need deferred transitions or a normalized-observation policy; simply requiring a legal transition and timestamp>last_event is not equivalent to full ordered replay. Keep unresolved conflicts visible for operator review. Use an outbox for notifications, and discard stale derived callbacks by projection version.
8. Idempotent, Out-of-Order Ingestion
Dedup on a source-scoped event identity and bind the payload fingerprint. The same ID with a different body is a conflict/correction to handle explicitly, not an interchangeable duplicate. Store history and projection/outbox effects atomically where co-located, or derive projections from a durable replayable stream with transactional checkpoints.
Timestamp comparison alone is insufficient. Source clocks can skew; equal timestamps need a deterministic tie-breaker (carrier sequence/version where available, then a documented fallback). A legal-transition gate can reject a newer successor that arrived before its older prerequisite; when that prerequisite arrives, replay the affected history or release deferred successors. Otherwise the projection never catches up.
Model repeated hub visits, exceptions, returns, misdelivery corrections, and scans that skip nominal steps. A returned-to-sender request is not necessarily physically returned; Delivered can be superseded by a correction. State is a deterministic business interpretation of retained evidence, not always the maximum timestamp/status number. Retention, access control, and privacy may require lifecycle-aware history deletion.
9. Computing the ETA
Customers care less about the current location than about the answer to "when will it get here?" The ETA service turns the stream of location scans into a predicted arrival window. It is recomputed whenever a new, applied scan moves the package, because each scan is fresh evidence about how the shipment is progressing relative to plan.
A first-cut ETA can be simple — the carrier's committed delivery date for the service level — and then refined as scans arrive:
Baseline. At creation, the ETA is the service-level commitment (e.g. "ground, 3–5 days"). This gives the customer an answer before a single scan exists.
Progress-based refinement. As the package hits expected waypoints on time, confidence narrows the window. If it sits at a hub longer than the historical average for that leg, the ETA slips.
Exception handling. An exception scan widens or invalidates the ETA — you do not pretend to know the arrival of a package that is currently lost.
The recomputed ETA is written back into the state store so it travels with the current status and is served on the next read. A more sophisticated version feeds historical leg times and current network load into a model, but the interview point is the shape: ETA is a function of the event history and the route, recomputed on each meaningful transition.
Note that every branch returns a time (now plus a duration), not a bare duration. Mixing the two is an easy bug: "2 days" stored where "Friday 14:00" was expected.
function recompute_eta(tracking_id):
state = state_store.get(tracking_id)
legs = route.remaining_legs(state.location, state.destination)
base = sum(historical_leg_time(leg) for leg in legs)
if state.status == EXCEPTION:
return widen(now() + base) # a wide window, not a point
return now() + base * congestion_factor() # adjust for load
10. Notifying the Customer
A status update is only useful if the customer hears about it. When an applied scan produces a transition the customer cares about, the ingest service emits an event to a notifier, which pushes a notification through the appropriate channel — push, SMS, or email — based on the customer's preferences.
Two design points keep this clean. First, not every scan is notification-worthy: a customer wants to know when the package is picked up, out for delivery, delivered, or stuck in an exception, but they do not need a ping for every intermediate hub scan. The notifier subscribes only to the meaningful transitions. Second, notifications should fire off the transition, not off the raw event — because of out-of-order and duplicate scans, firing on every event would mean duplicate or even contradictory alerts ("delivered" followed by "in transit"). Driving notifications from the deduplicated, ordered state machine guarantees the customer sees a coherent story.
NOTIFY_ON = {PICKED_UP, OUT_FOR_DELIVERY, DELIVERED, EXCEPTION}
function on_transition(tracking_id, old, new):
if new in NOTIFY_ON and new != old: # meaningful + actually changed
notifier.send(customer_of(tracking_id),
template=new,
params={tracking_id, eta_of(tracking_id)})
Notifying off the state machine rather than off raw events is the same discipline as deduping by event id: both ensure the customer experiences one coherent, monotonic narrative even though the underlying event stream is messy.
11. The Customer Read Path
The most common operation in the whole system is a customer typing a tracking number into a page and asking "where is my package?" These reads vastly outnumber writes, they spike unpredictably (a delivery-day surge, a marketing email, a delayed shipment everyone refreshes), and they must be fast. So the read path is deliberately separated from the write path.
The read store holds a denormalized view per tracking number: current status, last known location, ETA, and a compact recent history — everything the tracking page needs in a single point lookup, with no joins. It is updated asynchronously whenever the state store changes, and it is the only thing the customer-facing API touches. This buys several properties:
Isolation. A flood of read traffic hits the read store, not the ingest pipeline, so a viral "everyone is refreshing" moment cannot stall event processing.
Caching. Because a tracking page is read far more often than the package moves, results cache well; most lookups never reach a database.
Independent scaling. Reads and writes scale on separate hardware tuned for their own pattern — point reads on one side, append-heavy ingestion on the other.
The flow for one carrier scan: validate the event, then in one transaction insert the history row if its event id is new, lock the package's state row, skip if the event is older than the current state, and otherwise upsert a legal transition. After commit, recompute the ETA, notify the customer if the status changed, and refresh the read store so the next query is instant.
This is a textbook command-query separation: writes flow through ingestion into the authoritative state and history, while reads are served from a projection optimized purely for lookups. The slight asynchrony — the read store lags the state store by a moment — is an acceptable trade for the isolation and speed it buys, because a tracking page being a second stale is invisible to the customer.
12. Failure Modes
What breaks
Effect
How to handle it
Ingest crashes mid-event
Scan half-applied, package looks stuck.
One transaction per event, and a state update that is safe to repeat; the carrier's retry finishes the job.
A carrier's webhooks stop
Packages from that carrier go silent.
Monitor per-carrier scan rate; fall back to polling; show "last update 2 days ago" honestly.
Carrier sends a burst after an outage
Ingest overload.
Put a queue (for example Kafka, partitioned by tracking_id) in front of ingest and let it drain.
Clock skew in a hub scanner
Scans ordered wrongly by scanned_at.
Reject timestamps far in the future; the state machine refuses illegal jumps and flags them for review.
Read store lags or is down
Stale or missing tracking page.
Serve from cache with an "updated at" time; fall back to the state store for a single lookup.
13. Tradeoffs and Alternatives
Choice
Option A
Option B
Rule of thumb
Getting data
Webhook push
Polling
Push is fresher and cheaper. Keep polling as a backup for carriers that do not push or go quiet.
Source of truth
Mutable state row
Event log, with state as a projection (event sourcing)
Keep both. The log is truth and lets you replay after a bug; the state row makes reads cheap.
Ordering
By arrival time
By event time (scanned_at)
Event time, because arrival order lies. Guard against skew with sanity checks.
ETA
Fixed service-level date
Model using leg times and load
Start with the fixed date; refine with a model when you have history.
Read path
Read the state store directly
Separate cached read store
A separate store adds a little lag but isolates ingestion from read spikes.
Worked example: a prerequisite arrives after its successor
In-transit scan E2 arrives before pickup scan E1.
Read top to bottom. The table below explains the state and guarantee at each step.
Step
State, calculation, and reasoning
1. Store E2 first
Keep its event-time/source/version evidence. If a strict transition rejects it from Created, retaining history alone does not automatically apply it later.
2. Receive E1 later
E1 explains the prerequisite. Rebuild the affected shipment projection in deterministic event order rather than permanently discarding E2's effect.
3. Apply a carrier correction
A delivered scan may be corrected as misdelivery. Store a new correction event, not silent mutation; the projection policy can supersede prior state.
4. Notify from the committed projection
Emit versioned status-change intent through an outbox. Delayed notifications check current relevance so 'in transit' does not arrive after a newer correction as if current.
Check your understanding: Why is 'apply only newer legal scans' incomplete?
A successor can arrive first and be rejected, then an older prerequisite can arrive later. Without replay/deferred processing, the successor never affects state even though the history contains it.
Run a small, testable model
Download concepts.py. It uses only the Python 3.10+ standard library. From the directory where you saved it, run:
python3 concepts.py projection
python3 concepts.py test
The model sorts late arrivals deterministically and rebuilds state, demonstrating the rejected-successor trap.
The program contains deterministic assertions and failure traces. It is a learning model, not a production service: it does not implement network consensus, distributed durability, authentication, or third-party delivery. Read its contracts and try changing an input before extending it.
Many carriers, status + ETA + history, notifications, 50 M packages/day.
5–10
Estimate
~6K scans/s (17K peak), 150 GB/day of history, 125 GB of current state.
10–15
States and API
Draw the lifecycle state machine; scans in, track and subscribe out.
15–25
High-level design
Carriers → queue → ingest → history + state → ETA, notifier, read store.
25–38
Deep dive
Duplicates and out-of-order events: event-id dedup, event-time guard, one transaction per package.
38–45
Wrap up
Failures, history tiering, what you would monitor (scan lag per carrier, stuck packages, ETA accuracy).
15. Common Mistakes
Mistake 1: ordering by arrival time
Webhooks arrive late and out of order. Ordering by received_at can move a delivered package back to "in transit".
Mistake 2: dedup check that is not atomic with the update
A crash between the two steps turns every retry into a "duplicate" and the package freezes.
Mistake 3: notifying on raw events
Duplicates and late scans become duplicate or contradictory alerts. Notify on state transitions instead.
Mistake 4: keeping all history in the hot database
55 TB a year belongs mostly in cheap storage; keep recent events hot.
Mistake 5: trusting carrier data
Status codes, fields and timestamps vary. Validate and map to your own vocabulary first.
16. Summary
Delivery tracking is the discipline of turning an unreliable external event stream into a clean, queryable model. The design is a handful of decisions that reinforce each other:
Concern
Mechanism
How do we model a shipment?
A state machine: Created → Picked up → In transit → Out for delivery → Delivered, plus Exception and Returned.
Where does information come from?
Scan events from carriers and hubs, via webhook or polling — out of order, duplicated, and untrusted.
How do we accept events safely?
An ingest service that validates, normalizes, and is the sole writer of state.
How do we keep an audit trail?
An append-only event history; current state is a projection that can be replayed from it.
How do we handle duplicates and reordering?
Idempotency by event id; advance the state machine only on newer, legal transitions.
When will it arrive?
An ETA service that recomputes the arrival window on each meaningful scan and writes it into state.
How does the customer find out?
A notifier driven off state transitions (not raw events) for meaningful status changes.
How do we serve "where is my package?"
A fast, denormalized read store, separated from the write path for isolation and scale.
The recurring theme: separate the immutable facts (the event history) from the derived views (current state, ETA, read store), and make ingestion idempotent and event-time ordered. Everything else — fast reads, coherent notifications, a trustworthy audit trail — falls out of those two commitments.
17. Check Yourself
A "delivered" scan arrives, then a delayed "in transit" scan from the day before. What should the system do?
Answer
Append the late scan to history (it is a true fact) but do not change the state, because its scanned_at is older than last_event_at.
The ingest service crashes after writing a scan to history but before updating state. The carrier retries. What goes wrong with a check-then-append design, and how do you fix it?
Answer
The retry is seen as a duplicate and dropped, so state never advances. Fix it by doing both writes in one transaction and by making the state update idempotent (guarded by the event-time check) instead of returning early on duplicates.
Why partition history and state by tracking_id?
Answer
Every event for one package lands on the same shard. That allows a local transaction across both tables and makes a package's history a single range scan.
Roughly how much history do 500 M scans a day at 300 B each produce in a year?
Answer
150 GB/day × 365 ≈ 55 TB/year, which is why old history is moved to cheaper storage.
Why drive notifications from state transitions rather than from each incoming event?
Answer
The event stream has duplicates and out-of-order scans. Transitions are deduplicated and ordered, so the customer gets one clean story with no repeated or backwards alerts.
What is the bug in returning widen(base) from an ETA function whose other branch returns now() + base?
Answer
One branch returns a duration and the other a point in time. The caller would store "2 days" as if it were a date. Both branches must return a time, for example widen(now() + base).
18. Key Terms
Scan event
A record that a package was seen at a place and time with a status code.
Event time vs. arrival time
When the scan happened (scanned_at) vs. when we received it.
Idempotent
Applying the same event twice has the same effect as once.
Event sourcing
Storing every change as an immutable event and deriving current state by replay.
Projection
A state machine specifies legal states and transitions. An event-sourced projection derives read state from a durable event history under a deterministic ordering and conflict policy. Example and limits above.
CQRS
Command-query responsibility segregation: separate models for writes and reads.
ETA window
A predicted arrival range, wider when confidence is low.
Exception
A legitimate non-happy state such as damaged, delayed or address problem.
19. Further Reading
Martin Kleppmann, Designing Data-Intensive Applications, chapter 11 "Stream Processing" (event time, idempotence, event sourcing).
Martin Fowler, "Event Sourcing" and "CQRS" articles on martinfowler.com.
Tyler Akidau et al., "The Dataflow Model," VLDB 2015, on event time vs. processing time.
Apache Kafka documentation on partitioning and idempotent producers.
Carrier tracking APIs such as UPS and FedEx developer docs, to see real status-code vocabularies.