Half an hour spent reading errors beats a week of tuning

🏢 The situation

The SuperMegaRetail design doc is approved. Lena:

"Now we fill it with engineering substance. Four sections the book calls the 'early stage': metrics and losses, the dataset, the validation schema, the baselines. Each with a justification of why exactly this way. Remember: a mistake at this stage costs months, because everything downstream is built on these decisions."

🎯 Your task

1. Build a hierarchy of metrics: from the loss to the business metric.
2. Design the dataset and answer "how much data is enough".
3. Choose the validation schema (including the non-trivial ones: nested, adversarial).
4. Build the baseline ladder and understand why the constant baseline matters.

---

📚 Theory

The hierarchy of metrics

One metric does not describe a system. In a mature design doc the metrics form a pyramid — from what the gradient optimizes to what the CFO sees:

The pyramid's rules:

• The loss ≠ the metric: the loss must be differentiable and "friendly" to the metric. If under-forecasting hurts the business more — take a quantile loss, not MSE.
• Consistency metrics: the stability of the forecast between runs and versions — the business hates it when Wednesday's order changes by 40% after the nightly recompute.
• The offline ↔ online link is verified by experiments: if an offline gain doesn't move the online metric — the pyramid is torn, and fixing it matters more than improving the model.

The dataset: how much data is enough?

The doc's data sections: sources, ETL, filtering, labeling, metadata.

• The sample-wise learning curve is the main tool for answering "is the data enough": train the model on 10%, 25%, 50%, 100% of the sample and look at the quality trend. Reaching a plateau → enough data; still climbing → data is worth more than new features.
• The chicken-and-egg problem: recommendations need logs, but logs appear after launching recommendations. The solution — launch a simple version (rules/popularity) to start collecting data.
• A healthy data pipeline: idempotency, schema checks, volume alerts, reproducible snapshots.

Validation schemas: standard and non-trivial

Schema	When
Holdout	lots of data, fast iterations
K-fold CV	little data, need the estimate's spread
Time-series (rolling)	anything with time — like our demand forecast
Nested CV	tuning hyperparameters and estimating quality honestly at the same time: the inner loop tunes, the outer measures
Adversarial validation	a "does train resemble test?" check: train a classifier to tell train from test. AUC ≈ 0.5 — same world; AUC → 1 — the distributions differ and your validation lies

Adversarial validation is an underrated trick: it also shows which features distinguish train from prod (the classifier's top features) — those are the future sources of drift.

The baseline ladder

• The constant baseline (mean/median for regression, the majority class for classification, random order for ranking) is a sanity check and the reference point. Spending two weeks on a model and losing to a constant built in 5 minutes is a classic — from which a day-one baseline saves you.
• When a baseline isn't needed (per the book): the cost of error is extreme (medicine — fallback is a human there), the solution is already battle-proven, or you're rebuilding a working system (the old version is the baseline).
• Complexity bias — the cognitive trap "complex = good". The baseline ladder is the organizational cure.