← Back to the Applied Science guide

Part I Bank 2 Math

Statistics and Probability

Every model is a guess about a distribution. Every experiment is a bet on the noise. This bank covers the math under both.

The breadth round almost always has a stats block. The questions look basic. The grade is not whether you know Bayes. It is whether you can work the numbers out loud, state the conditions, and say where the method breaks.

This page has ten questions. Each one has what they test, a strong answer, the follow-ups, the traps, and one line to say out loud. Work each one with a pen first. Then read the answer.

Contents

  1. Bayes and the rare disease test
  2. Central limit theorem
  3. What a p-value is not
  4. MLE vs MAP
  5. Expected value puzzles
  6. Reservoir sampling
  7. Type I, Type II and power
  8. Simpson's paradox
  9. Standard error and the bootstrap
  10. Which distribution, when
How to answer a stats question. Define the thing in one line. Write the formula. Plug in numbers and get a result. State the assumptions. Then name one case where it breaks and what you would do instead. Five steps, about two minutes. Numbers beat words every time.

Part A — Probability

1. A disease affects 1 in 1,000 people. A test catches 99% of cases and has a 5% false positive rate. You test positive. What is the chance you have it? Easy

What they are testing

Strong answer

Start with the terms. Let D mean "has the disease" and + mean "tests positive".

Bayes' rule flips the condition:

P(D | +) = P(+ | D) P(D) / P(+)

P(+) = P(+ | D) P(D) + P(+ | not D) P(not D)
     = 0.99 * 0.001 + 0.05 * 0.999
     = 0.00099 + 0.04995
     = 0.05094

P(D | +) = 0.00099 / 0.05094 = 0.0194   (about 2%)

So a positive test means about a 2% chance of disease. Most people guess 95% or 99%. That gap is the whole point of the question.

Say it with counts. Counts are easier to follow than fractions. Picture 100,000 people.

100,000 people 100 sick 99,900 healthy 99 test + 1 test − 4,995 test + 94,905 test − Positives: 99 + 4,995 = 5,094. True share: 99 / 5,094 = 1.9%
The yellow boxes are all the positives. The healthy group is so large that its 5% error wins.

The odds form is faster. Posterior odds equal prior odds times the likelihood ratio.

LR+ = P(+ | D) / P(+ | not D) = 0.99 / 0.05 = 19.8

prior odds      = 0.001 / 0.999 = 0.001
after 1 test +  = 0.001 * 19.8  = 0.0198  -> p = 1.9%
after 2 tests + = 0.0198 * 19.8 = 0.392   -> p = 28.2%
after 3 tests + = 0.392 * 19.8  = 7.77    -> p = 88.6%

This form makes repeat tests easy. Each positive multiplies the odds by 19.8. But that only holds if the test errors are independent given the true state.

The ML link. This is precision under class imbalance. Swap "sick" for "fraud" and "test" for "classifier".

Follow-ups they will ask

Common traps

Say this out loud: “Out of 100,000 people, 99 sick people test positive and about 5,000 healthy people do. So a positive means about 2%. The base rate dominates. In ML terms, high recall does not give high precision when positives are rare.”

2. State the central limit theorem. When does it hold, and why do ML and A/B tests rely on it? Medium

What they are testing

Strong answer

Central limit theorem. Take X1, ..., Xn independent and identically distributed, with mean μ and finite variance σ2. Then the standardized sample mean tends to a standard normal as n grows: √n (X̄ − μ) / σ → N(0, 1) in distribution.

Read it as: the sample mean is about N(μ, σ2/n) for large n. The data itself can be any shape. Clicks are 0 or 1. Revenue is skewed. The mean of many of them is still close to normal.

Three separate facts often get mixed up.

The conditions.

How fast? The Berry-Esseen bound gives the rate. The max gap between the true CDF and the normal CDF is at most C ρ / (σ3 √n). Here ρ = E|X − μ|3 and C < 0.48. So skewed, heavy-tailed data converges slowly. The "n > 30" rule is a myth for skewed data. Revenue per user may need tens of thousands of users before the mean looks normal.

Why A/B tests rely on it.

Why ML relies on it.

Follow-ups they will ask

Common traps

Say this out loud: “For iid data with finite variance, the sample mean is about normal with variance sigma squared over n. That is why a z-test works on clicks or revenue. But heavy tails slow it down, and correlated units shrink the real sample size.”

3. What is a p-value, and what is it not? How do you read a confidence interval? Medium

What they are testing

Strong answer

p-value. The probability of seeing a test statistic at least as extreme as the one you saw, if the null hypothesis were true. In symbols: p = P(T ≥ tobs | H0).

It measures how surprising the data is under the null. It conditions on H0. It says nothing direct about how likely H0 is.

What a p-value is not.

Why the flip matters. Work the numbers. Say 10% of the ideas you test truly work. Tests use α = 0.05 and 80% power. Run 1,000 tests.

So "p < 0.05" does not mean "95% sure it works". With a low hit rate, a third of your wins can be noise.

95% confidence interval. A procedure that, over many repeated samples, gives an interval that covers the true value 95% of the time.

The 95% is a property of the method, not of one interval. Once you compute [1.2%, 3.4%], the true lift is either in it or not. In strict frequentist terms you cannot say "95% chance the truth is in here".

Correct readings.

Why the interval beats the p-value. It shows the size and the doubt together. "Lift is 0.1%, CI [0.05%, 0.15%]" is significant and maybe too small to matter. "Lift is 3%, CI [−1%, 7%]" is not significant but may be worth a bigger test.

Follow-ups they will ask

Common traps

Say this out loud: “A p-value is the chance of data this extreme if the null were true. It is not the chance the null is true. I would rather report the effect with its confidence interval, because that shows size and doubt together.”

4. Explain MLE vs MAP. Derive the MLE for a Bernoulli and a Gaussian. Show MAP with a Beta prior. How does MAP relate to L2 regularization? Medium

What they are testing

Strong answer

MLE. Pick the parameter that makes the data most likely: θ̂MLE = argmaxθ P(D | θ).
MAP. Pick the parameter that is most likely given the data: θ̂MAP = argmaxθ P(D | θ) P(θ). That is the mode of the posterior.

We always maximize the log. Logs turn products into sums and do not move the argmax.

MLE for a Bernoulli. We see n flips with k heads. The parameter is p.

L(p)     = p^k (1 - p)^(n - k)
log L(p) = k log p + (n - k) log(1 - p)

d/dp     = k / p - (n - k) / (1 - p) = 0
=> k (1 - p) = (n - k) p
=> p_hat = k / n

The second derivative is negative, so this is a max. The negative log-likelihood here is binary cross-entropy. So training a classifier with log loss is MLE.

MLE for a Gaussian. We see x1 ... xn from N(μ, σ2).

log L = -(n/2) log(2 pi sigma^2) - (1 / (2 sigma^2)) * sum_i (x_i - mu)^2

d/d mu      = (1 / sigma^2) * sum_i (x_i - mu) = 0
=> mu_hat   = (1/n) * sum_i x_i

d/d sigma^2 = -n / (2 sigma^2) + (1 / (2 sigma^4)) * sum_i (x_i - mu)^2 = 0
=> sigma2_hat = (1/n) * sum_i (x_i - mu_hat)^2

Two points to mention. First, the variance MLE divides by n, so it is biased low. Dividing by n − 1 fixes the bias. Second, the μ part is just least squares. So mean squared error is the Gaussian negative log-likelihood with fixed variance.

MAP with a Beta prior. Put a Beta(α, β) prior on p. Its density is proportional to pα−1 (1−p)β−1.

posterior  ~ p^(k + alpha - 1) (1 - p)^(n - k + beta - 1)
           = Beta(alpha + k, beta + n - k)

MAP (mode) = (k + alpha - 1) / (n + alpha + beta - 2)
mean       = (k + alpha) / (n + alpha + beta)

Product example. A new ad has 1 click in 3 views. MLE says 33% CTR, which is silly. With a prior Beta(2, 98) (about 2% CTR), the posterior mean is (1 + 2) / (3 + 100) = 2.9%. That shrinkage is how you rank new items fairly.

MAP and L2 regularization. Take linear regression with Gaussian noise: y = Xw + ε, ε ~ N(0, σ2). Put a prior w ~ N(0, τ2 I) on the weights.

-log posterior = (1 / (2 sigma^2)) * ||y - Xw||^2  +  (1 / (2 tau^2)) * ||w||^2  + const

Multiply by 2 sigma^2:
argmin_w  ||y - Xw||^2 + lambda * ||w||^2,    lambda = sigma^2 / tau^2

That is ridge regression. A tight prior (small τ) means a big λ. More noise also means a big λ. A Laplace prior p(w) ∝ exp(−|w|/b) gives L1, the lasso. Its sharp peak at zero is why L1 sets weights to exactly zero.

Follow-ups they will ask

Common traps

Say this out loud: “MLE maximizes the likelihood. MAP adds a log prior. For coin flips, MLE is k over n, and a Beta prior just adds fake counts. A Gaussian prior on weights gives exactly L2, with lambda equal to sigma squared over tau squared.”

5. Expected value puzzles. How many fair coin flips until you see HH? Until HT? Then coupon collector, and one more. Hard

What they are testing

Strong answer

Puzzle A: HH vs HT. Most people guess both are 4. They are not. HT takes 4 flips on average. HH takes 6.

Model it as a Markov chain. A state is how much of the target you have matched so far. Let Es be the expected flips left from state s.

Target HH start H HH H H T (back to start) T Target HT start H HT H T H (stay) T The only difference: a miss from state H resets HH but keeps HT in state H.
Both chains need one H to leave start. They differ in what a miss costs.

HH. Every equation reads "spend one flip, then move".

E_start = 1 + 0.5 * E_H + 0.5 * E_start
E_H     = 1 + 0.5 * 0   + 0.5 * E_start

From the first:  E_start = 2 + E_H
Substitute:      E_start = 2 + 1 + 0.5 * E_start
=>               E_start = 6

HT.

E_start = 1 + 0.5 * E_H + 0.5 * E_start
E_H     = 1 + 0.5 * 0   + 0.5 * E_H      # an H keeps you in state H

From the second: E_H = 2
From the first:  E_start = 2 + E_H = 4

The intuition. Both need an H first. After that, HT waits for a T, and an extra H costs nothing. HH waits for a second H. But a T throws away the H you had, and you start over. The per-position chance is the same, 1/4. But HH matches clump together (HHH holds two), so the gaps between them are longer.

Shortcut for any pattern. Sum 2j over every length j where the prefix of length j equals the suffix of length j. For HH, both j = 1 and j = 2 match, so 2 + 4 = 6. For HT, only j = 2 matches, so 4. For HHH it is 2 + 4 + 8 = 14. This comes from a martingale argument (the gambler's casino trick).

Puzzle B: coupon collector. There are n coupon types, each equally likely per draw. How many draws until you have all n?

Split the wait into stages. Once you hold i types, a new type appears with chance (n − i)/n. The wait for it is geometric, with mean n/(n − i). By linearity, add the stages:

E[T] = sum_{i=0}^{n-1} n / (n - i) = n * (1 + 1/2 + ... + 1/n) = n * H_n
     ~ n ln n + 0.577 n

n = 6  (die faces):  6 * 2.45  = 14.7 rolls
n = 50:              50 * 4.50 = 225 draws

Most of the time goes into the last few types. The final one alone takes n draws on average.

ML link. How many random samples until every class or every shard is seen? How many random negatives before you cover a catalog? It is about n ln n, not n.

Puzzle C: hat check. n people throw their hats in a pile. Each takes one at random. What is the expected number who get their own hat?

Let Ij = 1 if person j gets their own hat. Then P(Ij = 1) = 1/n. The total is ∑ Ij. By linearity, the mean is n · 1/n = 1. That holds for any n. The Ij are not independent, and it does not matter. Linearity never needs independence.

A bonus fact: the chance that nobody gets their own hat tends to 1/e ≈ 0.368.

Follow-ups they will ask

Common traps

Say this out loud: “I will define states by how much of the pattern I have matched. Then each state's expected time is one flip plus the average of where I go next. For HT a miss keeps my H, so it is 4. For HH a miss resets me, so it is 6.”

6. Sample k items uniformly from a stream of unknown length, in one pass and O(k) memory. Prove it works. Medium

What they are testing

Strong answer

The algorithm (Algorithm R). Number the items from 1.

  1. Put the first k items in the reservoir.
  2. For item i > k, keep it with chance k/i.
  3. If you keep it, replace a slot chosen uniformly at random.

At any point, the reservoir is a uniform sample of size k from the items seen so far. You never need to know the stream length.

import random
from typing import Iterable, TypeVar

T = TypeVar("T")


def reservoir_sample(stream: Iterable[T], k: int, seed: int | None = None) -> list[T]:
    """Return k items chosen uniformly at random from a stream.

    Each item ends up in the result with probability k / n, where n is the
    stream length. One pass, O(k) memory, O(1) work per item.

    Args:
        stream: Any iterable. Its length need not be known.
        k: Sample size. Must be at least 1.
        seed: Optional seed for reproducible tests.

    Returns:
        A list of min(k, n) items.
    """
    if k < 1:
        raise ValueError("k must be at least 1")
    rng = random.Random(seed)
    reservoir: list[T] = []

    for i, item in enumerate(stream, start=1):   # i is the 1-based position
        if i <= k:
            reservoir.append(item)               # fill the first k slots
        else:
            j = rng.randint(1, i)                # uniform on 1..i
            if j <= k:                           # true with chance k / i
                reservoir[j - 1] = item          # j is also a uniform slot
    return reservoir

A neat detail: one random draw does both jobs. j is uniform on 1..i. So j ≤ k has chance k/i. Given that, j is uniform over the k slots.

Proof by induction. Claim: after n ≥ k items, each item is in the reservoir with chance k/n.

You can also show it directly. Item i enters with chance k/i. It then survives each later step j with chance (j − 1)/j. The product telescopes:

(k / i) * (i / (i+1)) * ((i+1) / (i+2)) * ... * ((n-1) / n) = k / n

Per-item uniformity is not the full claim. A strong answer notes that every k-subset is equally likely, too. The same induction works on subsets.

Extensions.

Follow-ups they will ask

Common traps

Say this out loud: “Keep the first k. For item i, keep it with probability k over i and evict a random slot. Each old item survives step i with probability i minus 1 over i. The product telescopes to k over n, so every item is equally likely.”

Part B — Inference

7. Define Type I error, Type II error and power. How do you pick a sample size? Easy

What they are testing

Strong answer

Power is always tied to an effect size. "The test has 80% power" means nothing alone. Say "80% power to detect a 2% relative lift".

The sample size formula. For a two-sided test comparing two means, equal groups:

n per group = 2 * (z_{1 - alpha/2} + z_{1 - beta})^2 * sigma^2 / delta^2

alpha = 0.05, power = 0.8:  (1.96 + 0.84)^2 = 7.84
=> n per group ~ 16 * sigma^2 / delta^2

The "16 rule" is worth memorizing. δ is the minimum detectable effect (MDE). σ2 is the metric's variance per unit.

Worked example. Baseline conversion is 10%. You want to detect a lift to 10.5%, a 5% relative lift.

sigma^2 = p (1 - p) = 0.1 * 0.9 = 0.09
delta   = 0.005
n       = 16 * 0.09 / 0.005^2 = 16 * 0.09 / 0.000025 = 57,600 per group

So you need about 115,000 users in total. If you get 20,000 eligible users a day, that is six days. Round up to a full week or two to cover weekday effects.

What moves power.

H0: no effect H1: true lift critical value α β power = area of H1 right of line
Move the line right and α shrinks but β grows. More data narrows both curves and shrinks both errors.

Follow-ups they will ask

Common traps

Say this out loud: “Type I is shipping a dud, Type II is killing a winner. At alpha 0.05 and 80% power, n per group is about 16 sigma squared over delta squared. For a 10% baseline and a half-point lift, that is about 58,000 per group.”

8. What is Simpson's paradox? Give a numeric example and tie it to confounding. Medium

What they are testing

Strong answer

Simpson's paradox. A trend that holds in every subgroup reverses, or vanishes, when you pool the groups. It happens when a third variable affects both the grouping and the outcome, and the groups have very different sizes.

The classic numbers: kidney stone treatments.

Why it flips. Doctors gave A to the hard cases. 263 of A's 350 patients had large stones. Only 80 of B's did. Large stones have lower success under any treatment. So A's pooled rate is dragged down by its hard case mix. The pooled rate is a weighted average. A and B use very different weights.

The link to confounding. Stone size is a confounder. It causes both the treatment choice and the outcome.

   stone size
    /      \
   v        v
treatment -> success

The pooled comparison mixes the treatment effect with the case mix effect. Stratifying by stone size blocks that back-door path. Then the within-group comparison estimates the causal effect. So here, A is the better treatment.

But do not always split. The data alone cannot tell you which view is right. The causal graph does.

Product examples.

Follow-ups they will ask

Common traps

Say this out loud: “A wins in both subgroups but loses pooled, because A got most of the hard cases. Stone size is a confounder. Whether to split depends on the causal graph. Split on a confounder, never on a mediator or a collider.”

9. Derive the standard error of the mean. Then explain the bootstrap, code it, and say when it fails. Medium

What they are testing

Strong answer

Standard error is the standard deviation of an estimator across repeated samples. It tells you how much the estimate would move if you re-ran the study.

Derivation for the mean. Take X1 ... Xn iid with variance σ2.

Var(X_bar) = Var((1/n) * sum_i X_i)
           = (1/n^2) * Var(sum_i X_i)       # Var(aY) = a^2 Var(Y)
           = (1/n^2) * sum_i Var(X_i)       # independence: covariances are 0
           = (1/n^2) * n * sigma^2
           = sigma^2 / n

SE(X_bar)  = sigma / sqrt(n)   ~  s / sqrt(n)   (plug in the sample SD)

Two lessons. First, precision grows like √n. To halve the SE you need four times the data. Second, the independence step is the weak point. If the Xi share a correlation ρ inside clusters of size m, the variance grows by the design effect 1 + (m − 1)ρ.

The bootstrap. Often there is no formula. Think of the SE of a median, a ratio, a percentile, or an AUC. The bootstrap estimates it by simulation. It treats the sample as a stand-in for the population.

  1. From your n data points, draw n with replacement.
  2. Compute the statistic on that resample.
  3. Repeat B times (1,000 to 10,000).
  4. The SD of the B values estimates the SE.
  5. For a 95% CI, take the 2.5th and 97.5th percentiles (the percentile method).
import numpy as np
from typing import Callable


def bootstrap(
    data: np.ndarray,
    stat: Callable[[np.ndarray], float],
    n_boot: int = 5000,
    alpha: float = 0.05,
    seed: int = 0,
) -> tuple[float, float, tuple[float, float]]:
    """Bootstrap the standard error and a percentile CI of any statistic.

    Args:
        data: 1-D array of iid observations.
        stat: Function that maps an array to a number, such as np.median.
        n_boot: Number of resamples.
        alpha: 1 - confidence level. 0.05 gives a 95% interval.
        seed: Random seed.

    Returns:
        (point estimate, standard error, (ci_low, ci_high)).
    """
    rng = np.random.default_rng(seed)
    n = len(data)
    # Each row is one resample: n indices drawn with replacement.
    idx = rng.integers(0, n, size=(n_boot, n))
    stats = np.array([stat(data[row]) for row in idx])
    se = stats.std(ddof=1)
    lo, hi = np.quantile(stats, [alpha / 2, 1 - alpha / 2])
    return stat(data), se, (lo, hi)


# Example: SE of the median of skewed revenue data.
rng = np.random.default_rng(1)
revenue = rng.lognormal(mean=3.0, sigma=1.2, size=2000)
est, se, ci = bootstrap(revenue, np.median)
print(f"median={est:.2f}  se={se:.2f}  95% CI=({ci[0]:.2f}, {ci[1]:.2f})")

# Sanity check: for the mean, the bootstrap SE should match s / sqrt(n).
_, se_mean, _ = bootstrap(revenue, np.mean)
print(se_mean, revenue.std(ddof=1) / np.sqrt(len(revenue)))

For an A/B test, resample each arm on its own, then take the difference of the statistics. For a ratio metric, resample users, not events. That is the cluster bootstrap.

When the bootstrap fails.

Follow-ups they will ask

Common traps

Say this out loud: “Variance of the mean is sigma squared over n, because the covariances vanish under independence. When there is no formula, I bootstrap: resample with replacement, recompute, take the spread. I resample at the randomization unit, and I do not trust it for extremes or tiny samples.”

10. Which distribution would you use, and when? Give real ML and product examples. Easy

What they are testing

Strong answer

Pick a distribution by its story, not its shape. Ask what process made the data. Then check the fit.

Bernoulli(p). One trial, success or failure.

Binomial(n, p). Count of successes in n independent Bernoulli trials with the same p.

Poisson(λ). Count of rare, independent events in a fixed window.

Exponential(λ). Time between events in a Poisson process.

Normal(μ, σ2). Sums and averages of many small effects.

Beta(α, β). A distribution over a probability, on [0, 1].

Log-normal. The log of the value is normal. It comes from many multiplicative effects.

Power law (Pareto, Zipf). P(X > x) ∝ x−a. A few items take most of the mass.

How to check a fit.

Follow-ups they will ask

Common traps

Say this out loud: “I pick by the story. A yes or no is Bernoulli. Counts in a window are Poisson, unless the variance is too high, then negative binomial. Waits are exponential. Rates get a Beta prior. Multiplicative quantities like revenue are log-normal. Popularity is a power law, so I watch for infinite variance.”

Recap

← 1 — ML Fundamentals 3 — Deep Learning and LLMs →