Reviewing for Performance, Scalability & Reliability
The code is correct. Now: what does it cost per event, and what breaks when traffic doubles?
Student edition · understand, apply, verify
Review costs and failure amplification against the actual workload rather than optimizing by intuition.
What you should be able to do: Bound batch size, queues, retries, and resources while preserving behavior and measuring the bottleneck.
A context-dependent practice guide, not a universal policy or verified personal history. Examples and numbers are illustrative. Use genuine, authorized evidence; personnel/privacy/legal decisions follow applicable organization policies and qualified review.
Concepts explained in this guide
Read these definitions first, or use the linked terms as you go. The examples and limitations are local to this guide; companion pages are optional.
Complexity and workload assumptions
Big-O describes growth with input size under a cost model. Worst-case, expected, amortized, and output-sensitive bounds are distinct promises.
Example. Binary search is O(log n) comparisons on sorted random-access input; printing every subset needs exponential output work.
Boundary. Expected hash-table O(1) is not worst-case O(1). Include output size, recursion space, input assumptions, integer costs, and library operations.
Backpressure and load shedding
Backpressure tells an upstream producer that the downstream stage cannot accept more work. Buffering delays that signal; shedding deliberately rejects or drops lower-priority work when a bound is reached.
Example. A sink accepts 100 events/s while input is 150/s. A 1000-event queue fills in 20 seconds. The system must then block, reject, or apply its documented drop policy.
Boundary. A finite queue cannot absorb an infinite outage. Prioritized messages still need hard quotas, deadlines, and bounded client queues; otherwise an exemption can exhaust the system.
Outputs, outcomes, and measurement
Outputs are produced things; outcomes are changes for users/business/team. Useful metrics state definition, denominator, population, period, and attribution limits.
Example. 'Shipped a tool' is output. 'Median onboarding fell ten to six days in this cohort' is an outcome with sample/context caveats.
Boundary. Before/after does not prove sole causation. Avoid gaming, selection bias, hidden costs, and unlike-team comparisons.
Event loops and nonblocking I/O
One loop handles ready sockets/callbacks instead of one waiting thread per connection. Blocking or long CPU work delays other clients.
Example. Ten thousand sockets share a loop, but their file descriptors, state, and buffers still consume memory.
Boundary. Readiness is not a whole request. Preserve partial read/write parser state; background threads and defaults depend on release/configuration.
Preconditions and postconditions
A precondition states required inputs/state before an operation. A postcondition states the result promised when those requirements hold and execution terminates normally. Together they make requirements testable.
Example. For sum_first(b,n), require integer 0≤n≤len(b) and numeric elements; promise the sum of b[0:n].
Boundary. State exceptions, mutation, numeric domains, and empty cases. A mathematical array range does not automatically validate a runtime input.
For an engineer coming from high-throughput ad-events infrastructure, this is home turf and the interviewer knows it. The question is whether you can spot the performance and reliability footguns that don't show up in a functional test — the ones that pass in review, pass in staging at 1x load, and then take down a service at peak. A correct algorithm with an accidental N+1, an allocation in a per-event loop, an unbounded cache, or a missing timeout is a latent incident. Your job as a reviewer is to read a diff for its behavior at scale and under failure, not just its behavior on the happy path with one request.
Read the diff at scale: find the hot path, then check for N+1s, per-iteration allocations, unbounded memory/cardinality, blocking I/O, big-O regressions, missing backpressure, caching, and data-pipeline cost.
What this question is really testing
Whether you carry a cost model in your head while you read code. Anyone can say “check for performance.” The signal is specificity: you locate the hot path first (what executes per request or per event), then you know the concrete anti-patterns to look for and why each one bites at scale. It also tests whether you treat reliability as part of performance — timeouts, retries, backpressure, and pool sizing are what keep a fast system from becoming a cascading-failure amplifier.
How to answer
Find the hot path first. Ask what runs per request or per event versus once at startup. A wasteful line in initialization is fine; the same line inside a per-event loop processing a million events a second is a fire. Frame everything relative to call frequency.
N+1 queries and work inside loops. The classic: a query or RPC per item in a collection instead of one batched call. Same shape for allocations, regex compilation, logging, and lock acquisition moved inside a loop that should be hoisted out.
Unbounded memory and cardinality. Any collection, cache, buffer, or map that grows with input and never evicts is an OOM waiting for the right traffic. Watch metric labels and log keys built from unbounded values (user IDs, URLs) — that's cardinality explosion in your monitoring bill.
Blocking I/O on the wrong thread. A synchronous network or disk call on an event loop, request handler, or a shared thread pool serializes everything behind it. Look for blocking calls that should be async or off-loaded.
Big-O regressions. A nested loop that quietly turned an O(n) pass into O(n²), a sort inside a loop, a linear scan where a map lookup belongs. These are invisible at test scale and quadratic at production scale.
Backpressure, timeouts, and retries. Every remote call needs a deadline. Every retry needs a bound and backoff with jitter, or you build a retry storm that DDoSes your own dependency. Ask where backpressure lives when a downstream slows down — unbounded queues just move the OOM.
Connection-pool and resource misuse. A connection or client created per call instead of pooled, a pool too small (serialization) or too large (thundering herd on the DB), a resource leaked because it's not closed on the error path.
Caching. Is the cache actually helping — realistic hit rate, sane key, bounded size, correct invalidation? A cache with a bad invalidation story trades a performance problem for a correctness one.
Data-pipeline performance. For batch/stream work: data skew (one hot key stalls a reducer), unnecessary shuffles, serialization cost (a chatty format or per-record serde), and small-file / partition problems. These dominate pipeline runtime far more than the map logic does.
What the interviewer is looking for
You anchor on the hot path and call frequency before critiquing anything.
Named, specific anti-patterns — N+1, cardinality, retry storm, skew — not vague “make it faster.”
Reliability folded into performance: timeouts, bounded retries, backpressure.
Awareness that observability cost (cardinality) is a real production cost.
“Measure before optimizing” discipline — you ask for a profile/benchmark rather than guessing, and you don't demand premature optimization off the hot path.
A concrete before/after that captures the most common catch — an N+1 plus an allocation inside a per-event loop:
# Conceptual bounded batching, not a complete RPC client:
# Require an input-size limit, batch-size limit, deadline, and missing-key policy.
# Reuse an injected client; batch unique IDs in supported chunks.
# Preserve output order and define errors/partial responses before returning.
# A single unbounded batch_lookup(ids) does not by itself bound fanout.
Common follow-ups
You suspect a perf problem but there's no benchmark. What do you do?
How to answer
Ask for the measurement, don't block on a guess. Request a profile, a micro-benchmark, or the expected call frequency — “measure before optimizing” cuts both ways.
A likely hot-path complexity/network amplification issue can justify a review concern before full production measurement, but quantify assumptions and request a representative benchmark.
Distinguish latent from realized. A slow path that runs once at deploy is a nit; the same on the ingestion path is blocking.
How do you review retry logic?
How to answer
Bounded attempts, exponential backoff, jitter. Unbounded or synchronized retries are a self-inflicted DDoS on a struggling dependency.
Only retry the retryable. Retrying a non-idempotent write or a 4xx just amplifies harm; pair retries with idempotency keys.
Budget and shed. Look for a retry budget / circuit breaker so retries stop when the downstream is clearly down.
What performance issues are specific to data pipelines?
How to answer
Skew. One hot key (a null, a whale advertiser) sends most rows to one reducer; look for salting or a skew-join strategy.
Shuffle and serialization. Unnecessary repartitions and a chatty record format dominate runtime; prefer columnar formats and fewer wide shuffles.
Partitioning and small files. Too many tiny partitions or output files crush the scheduler and downstream readers.
How do you avoid over-optimizing in review?
How to answer
Weight by the hot path. Off-path code doesn't need micro-tuning; demanding it is premature optimization that hurts readability.
Correctness and clarity first, then measured perf. Don't trade a clear design for a speculative speedup with no benchmark.
Block only on real, scaled risk. Everything else is a “nit / consider later.”
Evidence to collect with your approved tools
A decision/review record with the problem, constraints, options, and your contribution.
Relevant tests, dashboards, or measurements with definitions, denominators, dates, and uncertainty.
Incident or delivery records showing actual outcomes and unresolved limitations.
Stakeholder agreements and follow-up actions with owners and review dates.
Only authorized, minimized, safe-to-share evidence. Do not copy confidential personnel notes into a public story.
Worked example: batch enrichment without an unbounded request
This is an illustrative situation. The point is the reasoning and evidence, not copying the numbers or first-person story.
Sequence for this example. The table explains the evidence and decision at each step.
Step
Evidence, decision, and expected result
Define scale and output policy
A request may contain 10,000 events; missing dimensions and duplicate IDs have explicit behavior.
Reuse a client
Avoid opening a connection for every event, but do not assume an unlimited batch is safe.
Bound batch size and time
Split dimension lookups into supported bounded batches with deadlines, retries, and concurrency limits.
Measure and preserve results
Compare latency, allocations, throughput, missing-key handling, and tail behavior under representative load.
Check your understanding: Does one batch call automatically mean bounded fanout?
No. Input cardinality, backend limits, timeout, and concurrency still need a bound.
Run and change the example
Download guide_labs.py. Python 3.10+, standard library only. In its directory, run:
python3 guide_labs.py algorithms
python3 guide_labs.py test
The lab validates algorithmic boundary/cost assumptions. The batch/RPC path still needs its own provider contract and load test.
The program tests its stated finite contracts. It is an educational model, not an implementation of the named production product or a proof of unmodeled distributed faults.
