🏢 The situation
An intern from a neighboring team brags in the general chat: "Trained an anti-fraud model, 99.2% accuracy!" Igor sends you a DM:
"Look at his notebook and find the catch. Hint: fraud is 0.8% of the data. Then present it at the demo — it's the best lesson on metrics you can get."
You open the notebook: the model always predicts "not fraud". Accuracy = 99.2%, because honest transactions are 99.2%. The model is useless — it didn't catch a single fraudster.
🎯 Your task
- Master the confusion matrix and the metrics: precision, recall, F1.
- Understand ROC-AUC and PR-AUC.
- Learn to choose the metric based on the business cost of errors.
📚 Theory
The confusion matrix
For binary classification (fraud = the positive class):
| Predicted: fraud | Predicted: not fraud | |
|---|---|---|
| Actually fraud | TP (caught) | FN (missed a fraudster) |
| Actually not fraud | FP (blocked an honest user) | TN |
Two errors — two different damages:
- FN — missed fraud: the company loses money directly.
- FP — a blocked honest customer: ruined experience, support calls, churn.
The core metrics
| Metric | Formula | Answers the question |
|---|---|---|
| Accuracy | (TP+TN)/all | "What share of predictions is correct?" — misleading under imbalance |
| Precision | TP/(TP+FP) | "Of those flagged as fraud — how many really are?" |
| Recall | TP/(TP+FN) | "Of all fraud — how much did we catch?" |
| F1 | harmonic mean of P and R | a compromise when both matter |
Precision and recall are always in tension: lower the threshold → catch more fraud (recall ↑), but block honest users more often (precision ↓).
ROC-AUC and PR-AUC
- ROC curve: TPR (recall) against FPR across all thresholds. AUC is the area under it: the probability that a random fraud case gets a higher score than a random honest one. 0.5 — a coin flip, 1.0 — ideal.
- ROC-AUC is threshold-independent and handy for comparing models, but under heavy imbalance it can look optimistic.
- PR-AUC (precision-recall) is more honest with a rare positive class — use it for fraud, defects, rare diseases.
How to choose a metric: from the cost of errors
The mature engineer's algorithm:
- Write down the cost of an FP and an FN in money/consequences.
- Understand the operational constraint (how many cases can the manual review team handle?).
- Choose the metric: e.g. "recall at precision ≥ 90%" or "precision within the top-1000 daily alerts".