Who fixes the model at three in the morning

🏢 The situation

The SuperMegaRetail demand model beat every baseline. Victor:

"Congratulations, the easy part is over. Now comes what separates a system from a model: training pipelines, an honest experiment, integration into procurement processes, monitoring, and the answer to 'who fixes it at 3 am'. Finish the design doc — this is your exam for a true middle with a senior trajectory."

🎯 Your task

1. Separate the training pipeline from the inference pipeline; understand the feature store's role.
2. Design the A/B experiment and the results report (debrief).
3. Build four-level monitoring and distinguish the drift types.
4. Design the fallbacks, the release cycle and the system's ownership.

---

📚 Theory

Training pipeline ≠ inference pipeline

Training and inference are different systems with different requirements:

	Training	Inference
Runs	on schedule/trigger	24/7 or in batches before the order deadline
Requirements	reproducibility, cheap replays	latency, reliability, fallbacks
Danger	silent data degradation	a service outage at peak hour

Their common point is the features: they must be computed identically in both (otherwise training/serving skew). Hence a feature store or, at minimum, a shared feature library + logging the actual inference features.

The training pipeline must be testable: unit tests on transformations, property-based tests ("forecast ≥ 0", "shares sum to 1"), a smoke training run on a sample in CI.

The system's A/B test and the debrief

Designing the experiment for a demand forecast is non-trivial: the randomization unit is not a user but a store (spillovers within a store). Key decisions from the book:

• decision metrics (write-offs, out-of-stock, margin) and guardrail metrics — fixed before the start;
• the split strategy: stratify stores by size/region; verify with an A/A test;
• when an A/B is impossible (too few stores) — switchback (alternating periods) or synthetic control;
• a debrief document after the experiment: what we expected, what we got, the decision, the lessons. A negative result with a clear debrief is also a company asset.

Monitoring: four levels and the drift types

The book splits ML system monitoring into levels: incoming data → the model → the model's output → decision-making. Plus ordinary service health (latency, errors).

Sudden drift lockdown, process change, release: alert + fallback + rebuild

Recurring drift seasons, holidays, weekdays: design it into the system

The vocabulary interviews ask about:

• Data drift (covariate shift): the input distribution changed, the relationship didn't. The model "hasn't seen such examples".
• Concept drift: the input → output relationship itself changed, even with the same inputs.
• Output drift: a shift in the prediction distribution — the cheap early indicator of both.
• Training-serving skew: production features computed differently than in training — not drift but a bug, with similar symptoms.

Responses: retrain on fresh data, rebuild the model, switch to the fallback. To find the retraining cadence — an "aging test": train the model on data up to T and measure the quality decay as you move away from T.