🏢 The situation
Wednesday. A Slack message from Lena:
"First real task. Max wants to understand what's going on with sales in the 'Electronics' category — revenue is growing but profit isn't. Anya exported
sales.csvfor you covering the last 12 months (50,000 orders). I need an EDA report by Friday: what's in the data, what anomalies, what hypotheses. Not a model — an understanding of the data. And check data quality: Anya had complaints about duplicates caused by pipeline retries."
🎯 Your task
- Load the data and understand its structure.
- Find quality issues: missing values, duplicates, anomalies.
- Answer the business question: why is revenue growing while profit is flat?
📚 Theory
What EDA is and why it matters
EDA (Exploratory Data Analysis) is the first thing any ML engineer does with a new dataset. Goals:
- Understand the structure: which columns, types, how many rows.
- Find problems: missing values, duplicates, outliers, type errors.
- See patterns: distributions, correlations, trends.
- Form hypotheses before building any models.
A rule from practice: a model trained on unverified data is a bug you'll notice too late.
The essential Pandas toolkit
What to look for in describe()
- Implausible min/max? Price = −500 or age = 200 → data errors.
- Mean far from the median (50%)? The distribution is skewed, there are outliers.
- std = 0? A constant column, useless.
Missing values: three strategies
| Strategy | When to use |
|---|---|
| Drop rows | < 1–2% missing and they are random |
| Fill (median/mode/special value) | noticeable share, the column matters |
| Keep as a feature | the very fact of missingness is informative (e.g. "didn't provide a phone number") |
Outliers
The classic method is the interquartile range (IQR): an outlier is a value outside [Q1 − 1.5·IQR, Q3 + 1.5·IQR], where IQR = Q3 − Q1. But remember: not every outlier is an error. A 900,000 order may be a real wholesale purchase. Investigate outliers first, then decide.