Business problem first, then the data, and only then the model

🏢 The situation

A year and a half at Datacore is behind you. Lena invites you to the meeting room; Victor and Max are there:

"We're opening a Middle ML Engineer position, and you're the lead candidate. One formality: defend a capstone project before the three of us. A new, real task, entirely yours — from framing to the operations plan. Show everything you've learned."

The project: Datacore's delivery service wants to predict a late order (will we deliver later than promised) in order to: (a) show honest delivery times at checkout, (b) proactively warn customers, (c) reinforce problem routes.

🎯 Your task

Walk through the project by yourself, checking against the model answers. This is a rehearsal of a middle's real work — and of your interview.

---

💼 The defense: the model solution

Step 1. Framing (modules 1, 18)

• Business metric: share of orders delivered within the promised time (SLA) + delivery NPS.
• ML task: binary classification "late / on time" at order checkout. Second iteration — regression of the exact time.
• Constraints: prediction at checkout in < 50 ms; explainability for the operations team.

Step 2. Data and features (modules 2, 7)

• History of 1.2M deliveries: route, warehouse, courier service, product, order time, weather, actual delivery time.
• EDA findings: 2% duplicates from retries; 8% of orders have "negative" delivery time (timezones!); a late-delivery peak on pre-holiday days.
• Features strictly point-in-time: route/courier delay aggregates over 30/90 days, warehouse load at order hour, calendar, distance. No fields filled after delivery!

Step 3. Model and validation (modules 4–10)

• Baseline: "late if the route was late in >30% of cases over the past month" → F1 = 0.41.
• Logreg → ROC-AUC 0.78; LightGBM + Optuna tuning → ROC-AUC 0.86, PR-AUC 0.52.
• Time-based validation (rolling, 3 windows); the test — the last 2 months, untouched until the end.
• Classes 12/88 → class weights; the threshold from the cost matrix: a false alarm = needlessly worried customer (cheap), a miss = a broken promise (expensive) → threshold at recall 0.85.

Step 4. What we show in the A/B (modules 3, 18)

• A test on 10% of traffic: checkout shows the "honest ETA" (forecast + buffer) instead of the standard one.
• North star: share of on-time orders; guardrail: checkout conversion (did honest but longer ETAs hurt it?).
• Pilot result: SLA violations −22%, conversion −0.4% (within tolerance), "where is my order" contacts −18%.

Step 5. Production and operations (module 19)

• FastAPI + Docker, an ONNX model, p99 = 23 ms.
• Monitoring: weekly PSI on features and scores; delayed quality as actual deliveries complete (labels arrive within 1–5 days — fast feedback!).
• Monthly retraining or on a drift alert; shadow mode for new versions; fallback — the old ETA formula.

The committee's questions (think first, then reveal)

Victor: "The model suggests showing longer ETAs — won't we lose orders?"

That's what the A/B guardrail was for: conversion dipped 0.4% while SLA violations fell 22% and contacts fell 18%. The economics are positive: retention and repeat purchases from reliable ETAs outweigh it. Plus the product can show a range "3–5 days" instead of a point.

Max: "Why should I believe ROC-AUC 0.86?"

Believe the process, not the AUC: time-based validation, a test that took no part in tuning, point-in-time features (verified in review), and above all — the pilot A/B confirmed the effect on live traffic. The offline metric is a filter; the decision was made by the experiment.

Lena: "What breaks first and how will you know?"

The courier-service aggregates: a change of contractor instantly stales the history. I'll know from a PSI alert on the service features and the delayed error growth in that segment. Runbook: drop the service features, retrain, temporarily widen the ETA buffer.

---