ML Fundamentals

1. Explain the bias-variance trade-off. Easy

What they are testing

Can you write the decomposition, not just name it? Do you know what each term means for a real model? Do you know where the classic story breaks down with deep nets?

Strong answer

Take a target y = f(x) + ε with noise of mean 0 and variance σ². We fit a model f̂ on a random training set D. At a fixed point x, the expected squared error over draws of D and noise splits into three parts.

E[(y - f̂(x))²] = (E[f̂(x)] - f(x))²  +  E[(f̂(x) - E[f̂(x)])²]  +  σ²
                  =        Bias²           +          Variance            + Irreducible noise

• Bias is the error of the average model. It comes from wrong assumptions. A line fit to a curve has high bias.
• Variance is how much the model moves when the training set changes. A deep tree fit to 100 points has high variance.
• Noise is the floor. No model can beat it with these features. New features can lower it. More data cannot.

The trade-off comes from capacity. More capacity lowers bias but raises variance. The test error curve is U-shaped. The best model sits at the bottom of the U.

How to tell which one you have. Look at train and validation error side by side.

• High train error and validation error close to it means high bias. Add capacity, add features, or weaken the regularizer.
• Low train error and a large gap to validation means high variance. Add data, regularize, simplify, or average models.

The modern twist. Very large networks can fit the training data to zero error and still generalize. Test error can rise near the point where the model just barely fits the data. Then it falls again as capacity keeps growing. This is called double descent. The decomposition still holds at every point. What changes is that variance does not grow forever with parameter count. Implicit regularization from SGD and the shape of the minimum it finds keep it in check.

Follow-ups they will ask

• Does bagging reduce bias or variance? Variance. Averaging B models with pairwise correlation ρ gives variance ρσ² + (1-ρ)σ²/B. Bias stays the same.
• Does boosting reduce bias or variance? Mostly bias. It adds weak learners that fix what earlier ones missed. Too many rounds can then raise variance.
• Does this decomposition work for 0-1 loss? Not cleanly. There are versions for classification, but the terms do not add up the same way. Say that squared loss is the clean case.
• Where does regularization sit? It adds a little bias to remove a lot of variance. Ridge regression is the textbook example.
• What is k's effect in k-NN? Small k gives low bias and high variance. Large k smooths the boundary and raises bias.

Common traps

• Saying “more data lowers bias.” It lowers variance. Bias is a property of the model family.
• Forgetting the noise term. Then you cannot explain why the error floor will not move.
• Saying deep nets “break” the trade-off. The math still holds. The shape of the curve changes.

2. How do you detect and fix overfitting? Easy

What they are testing

Do you have a real toolbox, ranked by cost? Do you check the data before you blame the model? Can you tell overfitting apart from a broken validation set?

Strong answer

Detect it. Overfitting means the model learned noise in the training set. The signal is a gap between training and held-out performance.

• Learning curves over epochs. Train loss keeps falling while validation loss turns up. That turn is the start of overfitting.
• Learning curves over data size. Plot both errors as the training set grows. A large gap that shrinks with more data means more data will help. Two curves that meet high means bias, not variance.
• Cross-validation spread. High variance across folds means the model is unstable.
• Too-good results. A validation AUC of 0.99 on a hard task is a red flag. Check for leakage before you celebrate.

Fix it. Go from cheap and safe to expensive.

1. Check the split first. Duplicates across train and test, time leakage, or user leakage can fake a gap or hide one.
2. More data or augmentation. The most reliable fix. For images, use crops, flips and color jitter. For text, use back-translation or token dropout.
3. Regularize the weights. Add L2 (weight decay) or L1. Tune the strength on validation.
4. Cut capacity. Fewer layers, smaller trees, larger min_samples_leaf, fewer features.
5. Early stopping. Stop at the best validation epoch. For gradient descent on a linear model, it acts much like L2.
6. Dropout and noise. Dropout trains an implicit ensemble of thinned networks. Label smoothing stops the net from pushing logits to infinity.
7. Ensembles. Bagging and averaging seeds lower variance directly.

Follow-ups they will ask

• Train and validation loss are both high. What now? That is underfitting. Add capacity, train longer, add features, or lower the regularizer.
• Validation loss rises but validation accuracy also rises. Why? The model gets more confident on the examples it gets wrong. Loss punishes that. Accuracy does not see it. It is a calibration problem as much as an overfitting one.
• Why does dropout help? It stops units from co-adapting. Each step trains a random subnetwork, so the final net acts like an average of many.
• How is early stopping like L2? For a quadratic loss, gradient descent from zero for t steps shrinks small-eigenvalue directions, much as ridge does. Fewer steps means a stronger penalty.

Common traps

• Jumping to dropout before checking the split for leakage.
• Choosing hyperparameters on the test set, then reporting the test score.
• Thinking a small gap means no overfitting. A shifted validation set can hide it.

3. Compare L1 and L2 regularization. Medium

What they are testing

Three views of the same idea. Can you show why L1 gives sparse weights, with geometry and with the gradient? Do you know the Bayesian prior behind each one? Do you know when to use elastic net?

Strong answer

Both add a penalty on the weights to the loss.

L1 (Lasso):  J(w) = Loss(w) + λ ∑ |w_j|
L2 (Ridge):  J(w) = Loss(w) + λ ∑ w_j²

View 1: geometry. Write the penalty as a constraint. Minimize the loss with ∑|w_j| ≤ t or ∑w_j² ≤ t. The L1 ball is a diamond with corners on the axes. The L2 ball is a circle. The loss contours are ellipses that grow until they touch the ball. They usually touch the diamond at a corner. At a corner, some weights are exactly zero. A circle has no corners, so the touch point is almost never on an axis.

View 2: the gradient. The L2 penalty has gradient 2λw. It pulls hard on big weights and gently on small ones. It shrinks weights toward zero but never quite gets there. The L1 penalty has a subgradient of λ·sign(w). It pulls with the same force no matter how small the weight is. If the loss gradient on a weight is smaller than λ, the weight goes to exactly zero and stays there. In one dimension with a squared loss, the closed forms show it.

Ridge:  w* = z / (1 + λ)                    # scale down, never zero
Lasso:  w* = sign(z) · max(|z| - λ, 0)        # soft threshold, exact zeros

Here z is the unpenalized least squares solution, with the penalty scaled to match. Solvers use coordinate descent or proximal gradient (ISTA) to apply this soft threshold.

View 3: Bayesian priors. Regularized least squares is MAP estimation.

• L2 means a Gaussian prior on each weight, w ~ N(0, τ²). The negative log prior is w²/2τ².
• L1 means a Laplace prior, p(w) ∝ exp(-|w|/b). The negative log prior is |w|/b. The Laplace has a sharp peak at zero, so it favors exact zeros.
• λ is set by the ratio of noise variance to prior scale. A tight prior means a strong penalty.

One subtle point: the full Bayesian posterior under a Laplace prior is not sparse. Only the MAP point is. Draws from the posterior almost never land on exactly zero.

Practical differences.

• Feature selection. L1 picks features. L2 keeps them all but small.
• Correlated features. L1 picks one of a group, often at random, and is unstable across samples. L2 spreads weight evenly across the group.
• Optimization. L2 is smooth and strongly convex. Ridge has a closed form, w = (XᵀX + λI)⁻¹Xᵀy, which also fixes a singular XᵀX. L1 is not differentiable at zero.
• Elastic net mixes both: λ₁||w||₁ + λ₂||w||₂². It keeps sparsity but selects correlated groups together. Use it when p is much larger than n.

Follow-ups they will ask

• Do you need to scale features first? Yes. The penalty treats all weights the same. A feature in meters gets a different penalty than the same feature in millimeters. Standardize first and do not penalize the bias term.
• Is weight decay the same as L2? For plain SGD, yes. For Adam, no. Adam divides the L2 gradient by the adaptive scale, so heavy-gradient weights get less decay. AdamW applies decay directly to the weights to fix this.
• How many features can lasso select when p > n? At most n before it saturates. Elastic net removes that limit.
• How do you pick λ? Cross-validation along a path of values. Many people use the “one standard error” rule to pick a simpler model.
• What about L0? It counts nonzero weights, which is the true sparsity goal. It is NP-hard to optimize. L1 is its tightest convex relaxation.

Common traps

• Saying L2 “makes weights zero.” It makes them small.
• Trusting lasso’s chosen features as causal or stable. Bootstrap the selection to check.
• Forgetting to standardize. Then the penalty depends on units.

4. Precision, recall, F1, and when does ROC-AUC lie? Medium

What they are testing

Do you know the formulas cold? Can you pick a metric from the business cost? Do you know why ROC-AUC looks great on a rare-event problem while the model is useless?

Strong answer

Start from the confusion matrix counts: TP, FP, TN, FN.

Precision = TP / (TP + FP)        # of the ones I flagged, how many were right
Recall    = TP / (TP + FN)        # of the real positives, how many I caught  (= TPR)
FPR       = FP / (FP + TN)        # of the real negatives, how many I flagged
F1        = 2 · P · R / (P + R)   # harmonic mean of P and R
Fβ        = (1 + β²) · P · R / (β² · P + R)   # β > 1 weights recall more

Pick by the cost of each error.

• Precision first when a false alarm is costly. A spam filter that hides a real email. An auto-ban on a real user.
• Recall first when a miss is costly. Cancer screening. Fraud that a human will review anyway.
• F1 when you need one number and both errors matter. It uses the harmonic mean, so a model cannot hide a bad recall behind a great precision.
• Best of all, put a dollar cost on FP and FN. Then pick the threshold that minimizes expected cost.

ROC-AUC. The ROC curve plots TPR against FPR as the threshold moves. AUC is the area under it. It has a clean meaning. AUC is the chance that a random positive scores higher than a random negative. It ignores the threshold and the class ratio.

When ROC-AUC lies. Its strength is also its flaw. FPR divides by the number of negatives. With heavy imbalance, negatives are huge. A large count of false positives is still a tiny FPR.

The PR curve plots precision against recall. Precision divides by what you flagged, not by all negatives. So it shows the false-positive cost directly. Report PR-AUC, also called average precision, for rare positive classes.

• A random classifier has ROC-AUC 0.5 at any class ratio.
• A random classifier has PR-AUC equal to the positive rate. With 0.1% positives, the baseline is 0.001. Always state the baseline next to PR-AUC.
• ROC-AUC can also hide where the gain lives. Two models can share an AUC but differ a lot in the low-FPR region you actually use. Look at partial AUC or TPR at a fixed FPR.

Follow-ups they will ask

• Is ROC-AUC ever better than PR-AUC? Yes, when the class ratio will change in production. ROC is invariant to the ratio. PR is not. It is also better when you care about ranking both classes equally.
• Does AUC measure calibration? No. Any monotone change to the scores keeps AUC the same. A model can have AUC 0.95 and badly wrong probabilities.
• Macro vs micro F1 for multiclass? Macro averages per-class F1, so rare classes count equally. Micro pools all counts, so it tracks the big classes. For single-label multiclass, micro F1 equals accuracy.
• Why not interpolate PR curves linearly? Precision does not move linearly between points. Linear interpolation overstates the area. Use step-wise average precision.

Common traps

• Reporting accuracy on a 99-to-1 problem. Predicting all negative scores 99%.
• Calling ROC-AUC “robust to imbalance” as praise. It is blind to imbalance, which is a different thing.
• Quoting PR-AUC without the positive rate baseline.

5. How do you handle class imbalance? Medium

What they are testing

Do you know that imbalance is often not the problem? Can you list fixes at the data, loss and decision levels? Do you know what each fix does to calibration?

Strong answer

First, ask if imbalance is really hurting you. A well-specified model trained with log loss handles a 1% positive rate fine. It learns the right probabilities. The real problems are three. The metric hides poor performance. The default 0.5 threshold is wrong. And there are too few positives to learn from. Fix those directly.

Level 1: the metric. Drop accuracy. Use PR-AUC, recall at a fixed precision, or expected cost. Use stratified splits so every fold has positives.

Level 2: the decision threshold. Often this is the only fix you need. Train normally, then choose the threshold on validation. With costs C_FP and C_FN and calibrated probabilities, the cost-optimal rule is simple.

Predict positive when  p(y=1|x) > C_FP / (C_FP + C_FN)

If a miss costs 9 times a false alarm, the threshold is 0.1, not 0.5.

Level 3: the loss.

• Class weights. Multiply each example’s loss by a class weight. A common choice is inverse frequency, w_c = n / (K · n_c). This has the same expected effect as oversampling, but without copying rows.
• Focal loss. From object detection, where easy negatives swamp the loss. It down-weights examples the model already gets right.

FL(p_t) = -α_t (1 - p_t)^γ log(p_t)       # p_t = prob of the true class, γ ~ 2

When p_t is 0.9, the factor (1-p_t)² is 0.01. So easy examples barely count. γ = 0 gives plain weighted cross-entropy.

Level 4: the data.

• Random undersampling of negatives. Fast and often fine at scale. Ads and fraud teams do this all the time. You throw away data, so use it when negatives are plentiful.
• Random oversampling of positives. Risk: the model memorizes the copies.
• SMOTE makes new positives by interpolating between a positive and its neighbors. It can help simple models on tabular data. It often adds noise in high dimensions or with categorical features. Results on modern boosted trees are mixed.
• Get more positives. Active learning, better labeling, or weak labels often beat any resampling trick.

The calibration cost. Resampling and class weights shift the predicted probabilities. If you undersample negatives at rate s (keep fraction s), you can correct the output.

p_true = p / (p + (1 - p) / s)

Equivalently, subtract log(1/s) from the logit. Anything downstream that needs real probabilities, like bidding or expected value, needs this fix.

import numpy as np

def focal_loss(p, y, gamma=2.0, alpha=0.25, eps=1e-7):
    """Binary focal loss. p: predicted P(y=1). y: 0/1 labels."""
    p = np.clip(p, eps, 1 - eps)
    p_t = np.where(y == 1, p, 1 - p)              # prob given to the true class
    a_t = np.where(y == 1, alpha, 1 - alpha)      # class balance weight
    return np.mean(-a_t * (1 - p_t) ** gamma * np.log(p_t))

def best_threshold(p, y, c_fp=1.0, c_fn=9.0):
    """Pick the threshold that minimises total cost on a validation set."""
    grid = np.unique(p)
    costs = [c_fp * np.sum((p >= t) & (y == 0)) + c_fn * np.sum((p < t) & (y == 1))
             for t in grid]
    return grid[int(np.argmin(costs))]

Follow-ups they will ask

• Should you resample the validation set? Never. Keep the real class ratio there, or your metrics will not match production.
• Resample before or after the split? After, and only on the training fold. Oversampling before the split puts copies of the same row in train and test.
• Class weights or threshold moving? For a linear model with enough data, they end in similar places. Threshold moving keeps probabilities calibrated, so I start there.
• What about extreme imbalance, 1 in a million? Treat it as anomaly detection or ranking. Use two stages: a cheap high-recall filter, then a precise model on the survivors.

Common traps

• Running SMOTE before the train-test split.
• Using class weights, then trusting the output as a real probability.
• Fixing the data when the only real problem was the 0.5 threshold.

6. Derive the logistic regression loss and gradient. Why not MSE? Medium

What they are testing

Can you derive log loss from maximum likelihood on a whiteboard? Can you get the clean gradient Xᵀ(p - y)? Can you give two separate reasons MSE is a bad fit?

Strong answer

The model. Logistic regression models the log odds as linear in x.

z = wᵀx + b
p = σ(z) = 1 / (1 + e^(-z))          # P(y = 1 | x)
log(p / (1 - p)) = z                    # log odds are linear

The loss from likelihood. Each label is a Bernoulli draw with success chance p. The chance of one label is p^y (1-p)^(1-y). Take the product over the data, then the negative log. That gives binary cross-entropy, also called log loss.

L(w) = -(1/n) ∑_i [ y_i log p_i + (1 - y_i) log(1 - p_i) ]

The gradient. One fact makes this short. The sigmoid has derivative σ'(z) = σ(z)(1 - σ(z)) = p(1-p). For one example:

∂L/∂p = -y/p + (1-y)/(1-p) = (p - y) / (p(1-p))
∂p/∂z = p(1-p)
∂L/∂z = (p - y)                    # the p(1-p) terms cancel
∂L/∂w = (p - y) x

Batch:  ∇_w L = (1/n) Xᵀ(p - y)      ∇_b L = (1/n) ∑(p_i - y_i)

The gradient is the error times the input. It has the same form as linear regression. That is no accident. Both are generalized linear models with a canonical link.

The Hessian. H = (1/n) Xᵀ S X, where S is diagonal with entries p_i(1-p_i). All entries of S are positive, so H is positive semi-definite. The loss is convex. Gradient descent finds the global minimum. Newton’s method on this loss is IRLS (iteratively reweighted least squares).

import numpy as np

def sigmoid(z):
    # stable form: avoid overflow in exp for large |z|
    return np.where(z >= 0, 1 / (1 + np.exp(-z)), np.exp(z) / (1 + np.exp(z)))

def fit_logreg(X, y, lr=0.1, epochs=1000, l2=0.0):
    """Batch gradient descent on L2-regularized log loss."""
    n, d = X.shape
    w, b = np.zeros(d), 0.0
    for _ in range(epochs):
        p = sigmoid(X @ w + b)
        err = p - y                          # dL/dz for every row
        w -= lr * (X.T @ err / n + l2 * w)   # do not regularize b
        b -= lr * err.mean()
    return w, b

def log_loss(y, z):
    """Log loss from logits, stable: log(1 + e^z) - y*z."""
    return np.mean(np.logaddexp(0, z) - y * z)

Why not MSE? Give both reasons.

1. Non-convex. MSE on top of a sigmoid, (σ(wᵀx) - y)², is not convex in w. Gradient descent can stall in flat regions.
2. Vanishing gradient when badly wrong. The MSE gradient with respect to z is 2(p - y) · p(1-p). Say y = 1 and the model says p = 0.001. Then p(1-p) is about 0.001, so the gradient is tiny. The model is confidently wrong and learns slowly. Log loss cancels that factor. Its gradient is p - y ≈ -1, full strength.
3. Wrong likelihood. MSE is the MLE under Gaussian noise. Labels in {0, 1} are Bernoulli, not Gaussian. Log loss is the matching likelihood, and it is a proper scoring rule, so it rewards honest probabilities.

Follow-ups they will ask

• What happens on linearly separable data? The weights grow without bound. The loss keeps shrinking toward zero as ||w|| goes to infinity. Add L2 or stop early. Plain gradient descent converges in direction to the max-margin solution.
• Extend to K classes. Softmax with cross-entropy. The gradient on logits is softmax(z) - onehot(y), the same clean form.
• How do you read a coefficient? A one-unit rise in x_j multiplies the odds by e^(w_j), holding other features fixed.
• Why compute loss from logits? log(σ(z)) underflows for very negative z. Use logaddexp or a fused “with logits” loss.
• With labels in {-1, +1}? The loss becomes log(1 + e^(-y·z)). This is the logistic loss in margin form. Compare it with hinge loss max(0, 1 - y·z) from SVMs.

Common traps

• Losing a sign in the gradient. Check: if p > y, the step must lower z.
• Saying logistic regression is “not linear.” The decision boundary wᵀx + b = 0 is a hyperplane.
• Saying only “MSE is non-convex.” The vanishing-gradient reason is the one they want.

7. Random forests vs gradient boosting. Medium

What they are testing

Do you know bagging and boosting as mechanisms, not just names? Can you say which error each one attacks? Do you know what makes XGBoost and LightGBM fast and well-regularized?

Strong answer

Random forest: bagging plus feature sampling

• Train B deep trees in parallel. Each tree sees a bootstrap sample of rows.
• At each split, each tree only looks at a random subset of features. A common default is √p for classification and p/3 for regression.
• Average the trees for regression. Take a vote or average probabilities for classification.

Why it works. Deep trees have low bias and high variance. Averaging B trees with pairwise correlation ρ gives variance ρσ² + (1-ρ)σ²/B. More trees kill the second term. Feature sampling lowers ρ, which shrinks the first term. Adding more trees never overfits. It only stops helping.

Bonus. Each row is left out of about 37% of bootstraps, since (1 - 1/n)^n ≈ 1/e. Score each row with the trees that did not see it. That gives a free out-of-bag error estimate.

Gradient boosting: sequential fits to the gradient

• Start with a constant, F_0 = argmin_c ∑ L(y_i, c).
• At round m, compute the negative gradient of the loss at each row, r_i = -∂L(y_i, F)/∂F. For squared loss, that is just the residual.
• Fit a small tree h_m to those pseudo-residuals.
• Update F_m = F_(m-1) + η · h_m with learning rate η, often 0.01 to 0.1.

Why it works. It is gradient descent in function space. Shallow trees have high bias and low variance. Each round removes bias. Too many rounds will overfit, so use early stopping on validation.

XGBoost details

• Second-order step. It uses a Taylor expansion with gradient g_i and Hessian h_i per row. The best leaf weight is w* = -G / (H + λ), where G and H sum over rows in the leaf.
• Regularized objective. It penalizes tree size: Ω = γT + ½λ||w||², with T leaves.
• Split gain. Gain = ½[G_L²/(H_L+λ) + G_R²/(H_R+λ) - (G_L+G_R)²/(H_L+H_R+λ)] - γ. A split happens only if the gain is positive. So γ acts as built-in pruning.
• Sparsity-aware splits. Each split learns a default direction for missing values.
• Row and column subsampling, borrowed from forests, to lower variance.

LightGBM details

• Histogram binning. Bucket each feature into about 255 bins. Split search costs O(bins), not O(rows).
• Leaf-wise growth. Split the leaf with the biggest gain anywhere in the tree. XGBoost’s classic mode grows level by level. Leaf-wise reaches lower loss with fewer leaves but overfits small data. Limit it with num_leaves and min_data_in_leaf.
• GOSS (gradient-based one-side sampling). Keep rows with big gradients. Sample the small-gradient rows and up-weight them.
• EFB (exclusive feature bundling). Merge sparse features that are rarely nonzero together.
• Native categoricals. It sorts categories by gradient statistics to find the best split. CatBoost goes further with ordered target statistics to avoid target leakage.

When to pick which

• Random forest. Little tuning. Hard to badly overfit. Good first baseline. Parallel training. Robust to noisy labels.
• Gradient boosting. Usually the most accurate on tabular data. Needs tuning of rounds, learning rate, depth and subsampling. More sensitive to label noise because it chases hard rows.

Follow-ups they will ask

• Why shallow trees for boosting and deep for forests? Boosting reduces bias, so it wants weak, low-variance learners. Bagging reduces variance, so it wants strong, low-bias learners.
• How do learning rate and rounds trade off? A smaller η needs more rounds but usually generalizes better. This is shrinkage. Tune rounds with early stopping for a fixed η.
• Is impurity-based feature importance trustworthy? No. It is biased toward high-cardinality and continuous features. Use permutation importance on held-out data, or SHAP values.
• Can trees extrapolate? No. Predictions are piecewise constant. Outside the training range, they flatline. A linear model or a trend term handles that better.
• Do trees need feature scaling? No. Splits only depend on order. Any monotone transform gives the same tree.

Common traps

• Saying more trees overfit a random forest. They do not. More boosting rounds do.
• Saying boosting fits “residuals” for every loss. It fits negative gradients. They only equal residuals for squared loss.
• Mixing up level-wise and leaf-wise growth.

8. How do you set up cross-validation and avoid data leakage? Hard

What they are testing

This is the most common real-world failure. Can you match the split to how the model will be used? Can you name concrete leakage cases from experience? Do you know that preprocessing must sit inside the fold?

Strong answer

The rule. The validation split must copy the production setting. In production, the model predicts on future data, for new users, with only the features known at that moment. Every gap between your split and that reality is a leak.

Pick the right split

• k-fold. For i.i.d. rows. Stratify on the label for classification so each fold keeps the class ratio. Typical k is 5 or 10.
• Group k-fold. When rows share an entity. Many rows per user, patient, session or document. All rows of a group go to one fold. Otherwise the model memorizes the user and the score inflates.
• Time-series split. Train on the past and validate on the future. Use an expanding or sliding window. Never shuffle. Add a gap (embargo) between train and validation when labels take time to mature or features use rolling windows.
• Nested CV. Use an inner loop to tune and an outer loop to score. Tuning and scoring on the same folds is optimistic.

The leakage catalog

• Target leakage. A feature that is caused by the label or only known after it. Examples: “number of refund calls” in a churn model. “Was given antibiotics” when predicting infection. “Account closed date” when predicting default.
• Time-travel features. Aggregates computed over the full table, including the future. A user’s “lifetime purchase count” as of today, used to predict a purchase last March. Fix with point-in-time joins.
• Preprocessing leakage. Fitting a scaler, imputer, PCA, feature selector, or target encoder on all data before splitting. The test fold’s statistics leak into training. Put every fitted step inside a pipeline that is refit per fold.
• Duplicate leakage. Near-duplicate rows or images across train and test. Common in scraped data and in LLM eval sets that leak into pretraining data.
• Group leakage. The same user or patient in train and test.
• Proxy IDs. Row order, file names, or ID numbers that correlate with the label. A famous case: a model detected the hospital scanner, not the disease.
• Tuning leakage. Picking features or hyperparameters by looking at test results.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GroupKFold, TimeSeriesSplit, cross_val_score

# Every fitted step lives inside the pipeline, so it is refit on each training fold.
pipe = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(),
                     LogisticRegression(max_iter=1000))

# One user never appears in both train and validation.
scores = cross_val_score(pipe, X, y, groups=user_id, cv=GroupKFold(n_splits=5),
                         scoring="average_precision")

# Rows sorted by time. gap= leaves an embargo between train and validation.
ts_scores = cross_val_score(pipe, X, y, cv=TimeSeriesSplit(n_splits=5, gap=7))

How to catch leaks.

• Look at feature importance. One feature with huge importance is suspect.
• Ask “when is this value known?” for every feature. Write the timestamp next to it.
• Compare offline lift to online lift. A big drop in the A/B test often means a leak.
• Train an “adversarial validation” classifier to tell train from test. If it wins easily, the splits differ.

Follow-ups they will ask

• Why not shuffle a time series? Shuffling lets the model see the future. Autocorrelation makes neighbors in time near-copies, so the score inflates.
• What k should you use? Larger k means less bias in the estimate but more compute and more correlated folds. 5 or 10 is the usual answer. Leave-one-out has high variance and is costly.
• Your offline AUC is 0.92 but online lift is zero. Why? Check leakage first. Then train-serve skew in feature code. Then feedback loops, where the old model chose which data got labels.
• How do you split a recommender dataset? By time, so you predict future interactions. Hold out users too if cold start matters.

Common traps

• Scaling or selecting features on the full dataset before CV.
• Random row splits when users have many rows.
• Reporting the CV score that also chose the hyperparameters.

9. Walk me through feature engineering for a tabular model. Medium

What they are testing

Practical judgment. Do you know which models need scaling? Can you handle a category with a million values? Do you know how target encoding leaks? Do you treat missing values as signal?

Strong answer

Numeric features

• Scaling. Models that use distances or gradients need it. That means linear models with regularization, SVMs, k-NN, k-means, PCA and neural nets. Trees do not need it.
• Standardize with (x - μ)/σ for roughly normal features. Min-max to [0, 1] for bounded ones. Robust scaling with median and IQR when outliers exist.
• Skewed features like income or counts: apply log1p or a Box-Cox / Yeo-Johnson transform. This helps linear models and neural nets.
• Clip outliers at a high percentile so one bad row does not dominate a gradient.
• Bucketing lets a linear model learn a non-linear shape. Quantile bins work well.

Categorical features

• One-hot for low cardinality, under about 50 values. Drop one level for unregularized linear models to avoid perfect collinearity.
• Ordinal encoding only when order is real, like size S/M/L. For trees, an arbitrary integer code often works anyway.
• Frequency encoding. Replace the category with its count. Cheap and leak-free.
• Feature hashing for very high cardinality. Map each value to hash(value) mod D. Memory is fixed and new values need no vocabulary. Collisions add noise. A second sign hash makes collisions cancel in expectation. Ads and search models use this at billions of features.
• Learned embeddings for neural nets. A rule of thumb is a dimension near the fourth root of cardinality, tuned from there.

Target encoding and its pitfalls

Target encoding replaces a category with the mean label for that category. It is powerful for high-cardinality data like zip code or merchant ID. It is also the easiest way to leak the label.

• The leak. If a row’s own label is in its category mean, the feature contains the answer. A category with one row gets an encoding equal to its label. The model looks perfect on train and fails on test.
• Fix 1: out-of-fold encoding. Compute each row’s encoding from the other folds only.
• Fix 2: smoothing. Shrink rare categories to the global mean.

enc(c) = (n_c · mean_c + m · global_mean) / (n_c + m)      # m = prior strength

• Fix 3: ordered statistics. CatBoost encodes each row using only rows before it in a random order.
• Time. For temporal data, compute the mean only from the past.

import numpy as np
import pandas as pd
from sklearn.model_selection import KFold

def target_encode_oof(cat: pd.Series, y: pd.Series, m: float = 20.0, k: int = 5, seed: int = 0):
    """Out-of-fold, smoothed target encoding. Each row never sees its own label."""
    out = np.zeros(len(cat))
    prior = y.mean()
    for tr, va in KFold(k, shuffle=True, random_state=seed).split(cat):
        stats = y.iloc[tr].groupby(cat.iloc[tr]).agg(["sum", "count"])
        enc = (stats["sum"] + m * prior) / (stats["count"] + m)
        out[va] = cat.iloc[va].map(enc).fillna(prior).to_numpy()   # unseen -> prior
    return out

Missing values

• Ask why it is missing. Missing completely at random (MCAR) is harmless. Missing at random (MAR) depends on other features. Missing not at random (MNAR) depends on the value itself. A blank “income” field often means low or high income. That is signal.
• Add a missing flag. Impute with median or mode and add a binary “was missing” column. The model can then learn from the pattern.
• Model-based imputation (k-NN, iterative) can help but costs time and can leak if fit outside the fold.
• Trees in XGBoost and LightGBM handle NaN natively. They learn which branch missing rows take.

Follow-ups they will ask

• Hashing or embeddings for a 10M-value ID? Hash into a fixed table of embeddings. That bounds memory and handles new IDs. Use multiple hash functions to cut collisions.
• How do you handle a category unseen in training? Map it to an “unknown” bucket, the prior mean, or its hash bucket. Train with rare values folded into “unknown” so the bucket learns something.
• Do features matter for deep models? Less for images and text. A lot for tabular data, where boosted trees with good features still often beat neural nets.
• How do you avoid train-serve skew? Compute features with one shared code path or a feature store. Log the served features and train on those logs.

Common traps

• In-sample target encoding. It is a direct label leak.
• Filling missing values with 0 when 0 is a real value.
• Fitting scalers or encoders on the full dataset.

10. What is calibration and how do you fix it? Hard

What they are testing

Do you know the difference between ranking and calibrated probabilities? Can you read a reliability diagram? Do you know Platt, isotonic and temperature scaling, and when each fits? Do you know where calibration matters in a product?

Strong answer

Definition. A model is calibrated when its scores match real frequencies. Of all cases where it says 0.7, about 70% should be positive. Formally, P(Y = 1 | p̂ = p) = p for all p.

Why it matters. Ranking only needs order. Many decisions need the actual number.

• Ad auctions bid pCTR × value. A model that says 2% when the truth is 1% overbids by 2x.
• Expected-cost thresholds like C_FP / (C_FP + C_FN) assume real probabilities.
• Combining models, or showing a risk score to a doctor, needs honest numbers.

Reliability diagram. Bin predictions by score, often 10 or 15 bins. For each bin, plot mean predicted score against observed positive rate. A calibrated model sits on the diagonal. Points below the diagonal mean overconfidence. Points above mean underconfidence.

Measure it.

ECE = ∑_b (n_b / n) · | acc(b) - conf(b) |        # weighted gap across bins
MCE = max_b | acc(b) - conf(b) |                  # worst bin

ECE depends on the binning. Equal-mass bins are more stable than equal-width ones. Also report a proper scoring rule like log loss or the Brier score, mean((p - y)²). Those catch miscalibration and poor ranking together.

Who is miscalibrated, and which way.

• Logistic regression is usually well calibrated, since it optimizes log loss directly.
• Naive Bayes pushes scores toward 0 and 1, because its independence assumption double-counts evidence.
• Random forests push scores away from 0 and 1, since averaging rarely gives extreme votes.
• SVM margins and boosted trees with many rounds are not probabilities at all without a fix.
• Modern deep nets tend to be overconfident. Long training and low log loss on train push logits large.
• Resampling and class weights shift every score. See question 5.

Fix it. Fit a small map from score to probability on a held-out calibration set. Never on the training set.

• Platt scaling. Fit p = σ(a·s + b) with logistic regression on the scores. Two parameters, so it needs little data. Assumes a sigmoid-shaped distortion.
• Isotonic regression. Fit a free-form non-decreasing step function (pool adjacent violators). Fixes any monotone distortion. Needs more data, roughly thousands of points, or it overfits. Creates ties, which can slightly change AUC.
• Temperature scaling. For neural nets. Divide all logits by one scalar T, fit on validation log loss: softmax(z / T). T > 1 softens. It does not change the argmax, so accuracy stays the same. One parameter, hard to overfit.
• Beyond these: vector or matrix scaling for multiclass, histogram binning, and beta calibration.

import numpy as np

def ece(probs, labels, n_bins=15):
    """Expected calibration error for binary probabilities, equal-width bins."""
    probs, labels = np.asarray(probs), np.asarray(labels)
    edges = np.linspace(0, 1, n_bins + 1)
    idx = np.clip(np.digitize(probs, edges[1:-1]), 0, n_bins - 1)
    total = 0.0
    for b in range(n_bins):
        mask = idx == b
        if mask.any():
            total += mask.mean() * abs(labels[mask].mean() - probs[mask].mean())
    return total

def fit_temperature(logits, labels, grid=np.linspace(0.5, 5, 91)):
    """Pick T that minimises multiclass NLL on a validation set."""
    def nll(T):
        z = logits / T
        z = z - z.max(axis=1, keepdims=True)
        logp = z - np.log(np.exp(z).sum(axis=1, keepdims=True))
        return -logp[np.arange(len(labels)), labels].mean()
    return min(grid, key=nll)

Follow-ups they will ask

• Does calibration change AUC? Platt and temperature scaling are strictly monotone, so AUC stays the same. Isotonic can create ties and shift AUC a little.
• Platt or isotonic? Platt for small calibration sets or sigmoid-shaped errors. Isotonic when you have plenty of data and an odd distortion.
• Can a model be calibrated but useless? Yes. Always predicting the base rate is perfectly calibrated with zero ranking power. Calibration and discrimination are separate.
• Calibration drifts in production. What do you do? Monitor predicted mean versus observed rate per segment. Refit the calibrator often on fresh labels, since it is cheap. Watch for label delay.
• Does a calibrated model stay calibrated per segment? Not guaranteed. Check calibration within key slices like country or new users. Multicalibration targets this.

Common traps

• Fitting the calibrator on training data. The scores there are already overfit.
• Treating ECE as a full metric. A trivial model can score low ECE.
• Assuming a high-AUC model gives good probabilities.

11. Generative vs discriminative models. Medium

What they are testing

Can you state what each one models? Do you know the classic Naive Bayes vs logistic regression result? Can you say when each one wins?

Strong answer

• Generative models learn the joint p(x, y) = p(x | y) p(y). They predict with Bayes’ rule: p(y | x) ∝ p(x | y) p(y). They can also sample new x. Examples: Naive Bayes, Gaussian discriminant analysis (LDA, QDA), HMMs, and modern deep generative models.
• Discriminative models learn p(y | x) directly, or just a decision boundary. Examples: logistic regression, SVMs, CRFs, most neural classifiers.

The Naive Bayes and logistic regression pair

Naive Bayes assumes features are independent given the class.

p(y | x) ∝ p(y) ∏_j p(x_j | y)

Take the log odds for binary y. With Gaussian features that share a variance per feature, or with Bernoulli features, the log odds are linear in x.

log p(y=1|x)/p(y=0|x) = log p(y=1)/p(y=0) + ∑_j log p(x_j|y=1)/p(x_j|y=0) = wᵀx + b

So both models have the same functional form for the posterior. They differ in how they fit w.

• Naive Bayes fits each feature’s class-conditional on its own, by counting. Closed form. Very fast.
• Logistic regression fits w jointly to maximize p(y | x). It makes no claim about how x is distributed.

Ng and Jordan (2001). Naive Bayes reaches its (higher) best error with O(log d) examples. Logistic regression needs O(d) examples but reaches a lower best error when the independence assumption is wrong. So Naive Bayes often wins with little data. Logistic regression wins as data grows.

When to pick which

• Generative wins with small data, when the model assumptions roughly hold, with missing features (you can marginalize them out), with unlabeled data in semi-supervised setups, and when you need to sample or detect outliers by low p(x).
• Discriminative wins with lots of labeled data and correlated features. It spends all its capacity on the boundary, not on modeling x. “Do not solve a harder problem than you need” (Vapnik).

Follow-ups they will ask

• Why is Naive Bayes poorly calibrated? Correlated features get counted as independent evidence. Ten copies of one feature multiply its effect by ten. Scores get pushed to 0 and 1. The ranking can still be good.
• Why Laplace smoothing? A word never seen with a class gives p = 0, which zeros out the whole product. Adding α to every count is MAP with a Dirichlet prior.
• How do LDA and QDA relate? Both model p(x|y) as Gaussian. LDA shares one covariance across classes, so the boundary is linear. QDA gives each class its own, so the boundary is quadratic.
• Is a GPT-style LLM generative or discriminative? Generative. It models p(x) as a product of next-token conditionals. When you use it as a classifier, you score p(label | prompt), which is a discriminative use of a generative model.

Common traps

• Saying generative models are always worse. With little data they often win.
• Forgetting that Naive Bayes and logistic regression share the same linear form.
• Saying “generative means it makes images.” It means it models x.

12. Explain the curse of dimensionality. Derive PCA. Hard

What they are testing

Can you make the curse concrete with numbers? Can you derive PCA from variance maximization to the eigenvectors of the covariance? Do you know the SVD link and the cases where PCA fails?

Strong answer

The curse of dimensionality

• Space grows fast. To cover [0,1]^d with a grid of spacing 0.1 takes 10^d cells. With d = 20, that is 10²⁰. Data cannot fill it.
• Neighbors are not local. To capture 1% of uniform data in a sub-cube, the edge must be 0.01^(1/d). At d = 10, that edge is 0.63. Your “neighborhood” spans most of each axis.
• Distances concentrate. For random points, the ratio of farthest to nearest distance goes to 1 as d grows. So “nearest” neighbor loses meaning. k-NN, kernel methods and clustering suffer.
• Volume moves to the shell. The fraction of a unit ball within ε of the surface is 1 - (1-ε)^d, which goes to 1. Gaussian samples in high d sit on a thin shell of radius about √d.

Why ML still works. Real data is not uniform. It lies near a low-dimensional manifold with structure. Models that exploit smoothness, sparsity or invariance beat the curse. Regularization, feature selection, and dimension reduction all help.

PCA derivation

Center the data so each column has mean 0. Call it X, with shape n by d. The sample covariance is Σ = XᵀX / (n-1).

Goal: find a unit vector u so that the projected data Xu has the most variance.

maximize   Var(Xu) = uᵀ Σ u
subject to uᵀu = 1

Lagrangian:  ℒ(u, λ) = uᵀΣu - λ(uᵀu - 1)
∂ℒ/∂u = 2Σu - 2λu = 0    ⇒    Σu = λu

So u must be an eigenvector of Σ. Plug back in: the variance is uᵀΣu = λuᵀu = λ. The best u is the eigenvector with the largest eigenvalue. The next component maximizes variance while staying orthogonal to the first. That gives the second eigenvector, and so on. Σ is symmetric, so its eigenvectors are orthogonal and the eigenvalues are real and non-negative.

The same answer comes from a second view. PCA finds the k-dimensional subspace with the smallest squared reconstruction error. Max variance and min reconstruction error are the same problem, since total variance is fixed.

The SVD link. Write X = U S Vᵀ. Then XᵀX = V S² Vᵀ. So the columns of V are the principal directions. The eigenvalues are s_i² / (n-1). The projected data is XV_k = U_k S_k. In practice use SVD on X, not an eigendecomposition of XᵀX. Forming XᵀX squares the condition number and loses precision. For huge data, use randomized SVD.

import numpy as np

def pca(X, k):
    """PCA by SVD. Returns projected data, components, and explained variance ratio."""
    Xc = X - X.mean(axis=0)                       # centering is required
    U, S, Vt = np.linalg.svd(Xc, full_matrices=False)
    var = S**2 / (len(X) - 1)                     # eigenvalues of the covariance
    components = Vt[:k]                           # rows are principal directions
    Z = Xc @ components.T                         # same as U[:, :k] * S[:k]
    return Z, components, var[:k] / var.sum()

Picking k. Keep enough components to explain 90 to 99% of variance. Or look for an elbow in the scree plot. Or pick k by the downstream model’s validation score, which is what matters.

When PCA fails

• Unscaled features. PCA chases variance, so a feature in large units dominates. Standardize first, which is PCA on the correlation matrix, unless the units are shared and meaningful.
• Non-linear structure. Data on a curved manifold, like a Swiss roll, is not captured by a linear subspace. Use kernel PCA, autoencoders, Isomap, or UMAP and t-SNE for visualization only.
• Variance is not signal. PCA is unsupervised. The direction that separates classes can have low variance and get dropped. For supervised reduction, use LDA or PLS.
• Outliers. Squared error lets one outlier tilt a component. Robust PCA helps.
• Interpretability. Each component mixes all features. Sparse PCA gives components with few nonzero loadings.
• Non-Gaussian sources. PCA only removes correlation. To separate independent signals, like mixed audio, use ICA.

Follow-ups they will ask

• Why must you center? Without centering, the first component points toward the mean, not along the spread. The covariance formula assumes zero mean.
• PCA before a tree model? Usually not. Trees handle many features and rotations hurt their axis-aligned splits. PCA helps distance-based and linear models more.
• Cost? Full SVD is O(n d min(n, d)). Randomized SVD for the top k is about O(n d k).
• What if d > n? At most n-1 nonzero components exist. Compute SVD on X directly, or eigendecompose the n by n Gram matrix XXᵀ.
• Does PCA leak if fit on all data? Yes. Fit it inside each CV fold like any other transform.

Common traps

• Forgetting to center or to scale.
• Saying PCA picks the most predictive features. It picks directions of high variance, with no label.
• Using t-SNE output as model features or reading distances between its clusters.