0.97 on the test set, 0.61 in production

🏢 The situation

Thursday, 11:00. Alarm in #ml-alerts: a neighboring team's scoring model that showed ROC-AUC 0.97 on the test yields 0.61 in production — worse than the old manual rules. CTO Victor demands an investigation. Lena assigns you to the incident group:

"The best way to learn validation is to investigate someone else's failure. Find where exactly they fooled themselves. Look at three things: how they split the data, what got into the features, and when the statistics were computed."

🎯 Your task

1. Understand overfitting and underfitting through bias/variance.
2. Master cross-validation and time-based validation.
3. Build a catalog of data leaks — and find the leak in this incident.

---

📚 Theory

Overfitting and underfitting

• Underfitting (high bias): the model is too simple; bad on both train and test.
• Overfitting (high variance): the model learned the noise; brilliant on train, a flop on new data.

Diagnosis by the gap: train_metric ≫ val_metric → overfitting. Both low → underfitting. Andrew Ng: "Look at train error and val error like a doctor at two lab tests — the combination gives the diagnosis."

Overfitting cures: more data, a simpler model, regularization, early stopping. Underfitting cures: a more complex model, better features, longer training.

Cross-validation

One train/test split is one random estimate. K-fold CV: split the data into K parts, train K times on K−1 parts, validating on the remaining one; average. You get both the estimate and its spread.

• StratifiedKFold — under class imbalance.
• TimeSeriesSplit / time-based split — whenever data has time (almost always in business!). Train on the past, validate on the future. A random split of two years of transactions = the model "saw the future".
• GroupKFold — when one customer has many rows: all of a customer's rows must land in one fold, otherwise the model recognizes the customer, not the pattern.

The data leakage catalog (learn by heart)

1. Target leakage into features: a feature is a consequence of the target ("number of collector calls" for default).
2. Leakage through time: features/aggregates computed using the future (not point-in-time).
3. Leakage through preprocessing: scaler/encoder/imputer fit on all data before the split.
4. Leakage through duplicates/groups: the same customer in both train and test.
5. Leakage through feature selection: selection done on all data, then an "honest" CV.

The symptom of all leaks is the same: a suspiciously high offline metric and a production flop.

---