🏢 The situation
The churn model from module 5 has been live for a month. Lena at planning:
"Logreg gave ROC-AUC 0.83 — a good start. Retention asks for more precision: every extra percent of quality is hundreds of retained customers. Time to bring out the heavy artillery of tabular ML: trees and ensembles. Compare Random Forest and gradient boosting with our logreg. Spoiler: on tabular data boosting almost always beats neural networks — understand why."
🎯 Your task
- Understand how a decision tree works and why a single tree is weak.
- Master the two ensemble strategies: bagging (Random Forest) and boosting.
- Beat the baseline and explain the gain.
📚 Theory
The decision tree
A tree splits the data with a series of "feature > threshold?" questions, at each step choosing the split that maximally reduces "impurity" (Gini/entropy). Pros: needs no scaling, captures non-linearities and feature interactions, interpretable. Con: a single tree overfits easily — a deep tree memorizes noise.
Ensembles: the wisdom of the crowd
The idea: many weak models that err differently are together more accurate than one strong model.
Bagging → Random Forest:
- Each tree trains on a random row subsample (bootstrap) and a random feature subset.
- Predictions are averaged → variance drops, overfitting is smoothed out.
- Trees are independent → training parallelizes.
Boosting → XGBoost / LightGBM / CatBoost:
- Trees are built sequentially: each next one corrects the errors (fits the gradient of the loss) of the previous ones.
- Trees are shallow (3–8 levels), and there are hundreds or thousands of them.
- Usually more accurate than a forest, but more sensitive to hyperparameters and easier to overfit.
| Random Forest | Gradient Boosting | |
|---|---|---|
| Strategy | parallel, averaging | sequential, error correction |
| Overfitting | robust | needs control (early stopping) |
| Training speed | faster (parallel) | slower, but LightGBM is very fast |
| Tabular quality | good | usually the best |
Key boosting hyperparameters
n_estimators— number of trees; with early stopping you can set it high.learning_rate— each tree's contribution; lower = more stable, but needs more trees.max_depth/num_leaves— tree complexity, the main overfitting knob.early_stopping_rounds— stop when the validation metric hasn't improved for N iterations.
Why boosting > neural nets on tables
Tabular data is heterogeneous (money, counters, categories) and the dependencies are piecewise-sharp ("delinquency > 30 days → risk jumps"). Trees model such thresholds natively; neural networks struggle. Confirmed by industry and Kaggle: for tables the default is gradient boosting.