🏢 The situation
You are now a Middle ML Engineer. Victor brings you to a meeting with a major client: the "SuperMegaRetail" chain (2000 stores, 47 distribution centers) wants a demand forecasting system.
"Datacore is taking this contract, and you're the lead engineer. But we won't write code for another month. First — a design document. Juniors start with the model; middles start with the question 'is there even a problem, and is it worth solving?'. Read Babushkin & Kravchenko — and bring me the first version of the design doc."
🎯 Your task
- Distinguish the problem space from the solution space and find the real problem.
- Assess the risks and the cost of a mistake before the project starts.
- Draft the design document structure and run its review.
📚 Theory
Problem space vs solution space
The engineer's classic mistake is to jump into the solution space ("let's do gradient boosting with lags!") without studying the problem space ("what actually hurts the business and why?").
Problem-space questions for SuperMegaRetail:
- How does the current ordering and supply process work? Who owns the process (logistics? procurement? stores?)
- How much does the company lose on the "delivered − sold" gap? (The client's estimate: ~$800M a year.)
- Which of the two errors hurts more: overstock (easy to compute — write-offs) or out-of-stock (hard: lost sales + the customer leaving for a competitor)?
The cost of a mistake — before the model, not after
Before building the system, write down what happens if the model errs, in both directions:
| Error | Consequence | Estimate |
|---|---|---|
| Forecast too high (overstock) | surplus, perishables written off | easy to compute from data |
| Forecast too low (out-of-stock) | empty shelf, a lost customer | hard: A/B tests or expert judgment |
The asymmetry of error costs will later determine the metric, the loss (quantile!), and the system's SLA.
Preliminary research: build vs buy and the degree of innovation
- Build vs buy: demand forecasting is a mature market with vendors. Build your own when the forecast is the core of competitive advantage, the data is peculiar, or no vendor fits the constraints.
- Decomposition: "demand forecasting" is not one problem but a family: new items vs mature, promo vs regular sales, perishables vs long shelf life.
- Degree of innovation: don't invent architecture when the industry standard solves the task. Save innovation for where the standard hits its limits.
The design document: why and what
A design doc is a document where the solution is designed and criticized before it becomes code. The myths that stop people from writing one:
"It's for big companies"— a startup finds it even cheaper to catch a mistake in a document than in production."Only for complex projects"— a simple project = a short doc."You need a template"— what matters is not the template but covering the key questions."Every doc must end in a deployed system"— a "don't do it" conclusion is also a success of the doc (and a very cheap one).
The skeleton of an ML system design doc:
- The problem and the goal — in business language, with numbers.
- Goals and antigoals — what we are NOT doing matters as much as what we are.
- Relevance and reasons — an analysis of the current flow, how much we lose, who else could use it.
- Previous work — what was already tried and why it didn't fly.
- Metrics and losses (module 22) · Data · Validation · Baselines.
- Integration, monitoring, ownership (module 23).
- Risks and the cost of a mistake.