93% of the alerts are false — and the economics still work

🏢 The situation

You are transferred to the anti-fraud sub-project. Igor brings you up to speed:

"The Datacore wallet processes 400,000 transactions a day. Fraud is 0.1%, but every missed case averages −$220, and every false block of an honest customer means a support call (−$5) and an angry user. The manual review team handles at most 500 alerts a day. Your model must live within these constraints. A 1:1000 imbalance is where all the textbook habits break."

🎯 Your task

1. Understand what breaks under heavy imbalance and how to fix it.
2. Master class weights, oversampling/undersampling, SMOTE — and their limits.
3. Choose the threshold from economics and operational limits.

---

📚 Theory

What breaks at 1:1000

1. Accuracy is meaningless (module 6): the constant "not fraud" gives 99.9%.
2. The loss function barely sees the rare class: 0.1% of examples contribute 0.1% of the gradient — the model learns to ignore fraud.
3. ROC-AUC is deceptively high; look at PR-AUC and precision/recall at the operating point.

Ways to fight it

Method	Essence	Pros / cons
Class weights (class_weight="balanced", scale_pos_weight)	errors on the rare class cost more	simple, no data distortion; the first choice
Undersampling	drop part of the majority class	faster training; loses information
Oversampling	duplicate the rare class	overfitting risk on duplicates
SMOTE	synthetic examples between rare-class neighbors	helps weak models; often useless for boosting, creates unrealistic points
Threshold tuning	don't touch the data, shift the decision	mandatory always, whatever you do above

Industry practice: for boosting — class weights + threshold tuning. Sampling — when the data doesn't fit in memory or the model is very weak.

⚠️ Golden rules:

• Sampling is applied to train only, never to validation/test — otherwise the metrics are measured in a "fantasy" world.
• After resampling/weights the predicted probabilities are biased — they cannot be read as real frequencies without calibration (CalibratedClassifierCV).

The threshold from economics

The expected value of an alert at threshold t: V(t) = TP(t)·220 − FP(t)·5, subject to TP(t)+FP(t) ≤ 500/day. The threshold is the solution of an optimization problem, not the constant 0.5.

---