A gain of 0.005 against a spread of 0.004 is noise

🏢 The situation

Friday. Lena:

"The LightGBM churn model is good, but you trained it on default parameters. Before rolling out to the whole base, let's squeeze out the maximum. Just no fanaticism: you have one day and one GPU server shared by three teams. Show that you can tune efficiently — not by brute force, but with brains. And don't tune the model into 'perfect on validation, useless in production'."

🎯 Your task

1. Understand which hyperparameters matter and why.
2. Compare grid search, random search and Bayesian optimization (Optuna).
3. Avoid overfitting to the validation set.

---

📚 Theory

Parameters vs hyperparameters

• Parameters the model learns itself (weights, tree split thresholds).
• Hyperparameters the engineer sets before training (tree depth, learning rate, regularization strength).

Three search strategies

Strategy	How it works	When
Grid search	full sweep of a grid	2–3 parameters with few values
Random search	random points from distributions	the default choice: for the same budget it covers the space better (Bergstra & Bengio, 2012)
Bayesian (Optuna, TPE)	builds a "parameters → quality" model and tries promising points	expensive training, many parameters — the industry standard

Why random > grid: if only 1 of 3 parameters matters, a 5×5×5 grid tries just 5 unique values of the important one; random tries all 125.

What to tune in gradient boosting (in order of importance)

1. learning_rate + n_estimators (with early stopping — fix a lowish LR, plenty of trees);
2. num_leaves / max_depth — model capacity;
3. min_child_samples — leaf regularization;
4. feature_fraction, bagging_fraction — stochasticity;
5. lambda_l1, lambda_l2 — if it still overfits.

Overfitting to validation — the silent killer

Hundreds of tuning iterations against a single validation set = you "fit" the model to it. Rules:

• Evaluate candidates with cross-validation, not a single split.
• Measure the final number on a test set that took no part in tuning, exactly once.
• A gain smaller than the CV spread (std) is noise, not improvement.

---