🏢 The situation
After your EDA report a debate broke out. Marketing: "Discounts increased conversion, we're awesome!" Max: "Or maybe conversion would have grown anyway — it's the season."
Lena:
"A classic. A battle of opinions is settled only by an experiment. Marketing already ran a test: 20,000 users were randomly split in half, group A saw a 15% discount, group B didn't. Conversion A = 6.4%, B = 5.8%. Marketing is shouting 'victory'. Your job is to check whether the difference is significant or just noise. And explain the result to Max in plain words."
🎯 Your task
- Understand basic statistics: distributions, mean vs median, variance.
- Learn how an A/B test works and what statistical significance is.
- Give the business an answer: does the discount work — and at what cost.
📚 Theory
Distributions — the language of data
The normal distribution is a symmetric "bell" (human height, measurement errors). Skewed distributions have a long tail (incomes, order values: many small ones, few huge ones).
Practical consequence: for skewed data the mean lies. An average order of $45 with a median of $12 means the "typical" order is $12, and the mean is pulled up by whales. Report the median and quantiles.
Hypothesis testing in plain words
- H₀ (null hypothesis): there is no difference; the gap between groups is chance.
- H₁ (alternative): there is a difference.
- p-value — the probability of seeing such a difference (or larger) if H₀ is true. Small p-value (usually < 0.05) → the data is poorly compatible with "chance" → reject H₀.
⚠️ Frequent mistakes that get you flagged in review:
- The p-value is not "the probability that H₀ is true".
- "Not significant" ≠ "no effect". Maybe there's just not enough data.
- Statistical significance ≠ practical significance. A 0.01% conversion lift can be significant with millions of users — and useless for the business.
A/B test: the minimal checklist
- Randomization: users are assigned randomly (not "A — Moscow, B — St. Petersburg"!).
- The decision metric is chosen before launch (e.g. profit per user, not just conversion).
- Sample size is computed in advance (power analysis); the test is not stopped the moment it "became significant" — that's peeking and it breaks the statistics.
- For conversions (success proportions) use a z-test for proportions or chi-squared; for means — a t-test.