If you have never overfitted a network, you cannot train one

🏢 The situation

The team's Friday tech talk. Lena:

"You've already seen that boosting beats neural nets on tables. But ahead of us are images (module 17) and deeper text work — there neural networks have no alternative. Today we dissect how they work, without magic: the neuron, layers, backprop. Homework — train a net in PyTorch and overfit it on purpose to see what that looks like. Whoever hasn't overfitted a network doesn't know how to train one."

🎯 Your task

1. Understand the network's anatomy: neuron, activations, layers, backpropagation.
2. Train a network in PyTorch: write the training loop by hand.
3. See overfitting on the learning curves and defeat it.

---

📚 Theory

The neuron and layers

A single neuron = that same linear model: output = activation(w·x + b). A network is layers of neurons: one layer's outputs become the next layer's inputs. A composition of linear layers with non-linear activations between them can approximate arbitrarily complex functions.

Activations: without them a stack of linear layers collapses into one linear model. The hidden-layer standard is ReLU max(0, x): simple, fast, doesn't "vanish" like sigmoid. At the output: sigmoid (binary classification), softmax (multiclass), nothing (regression).

How the network learns: backpropagation

The same gradient descent from module 4, but the gradients over millions of weights are computed by the chain rule from output to input — that is backprop. One training step:

1. Forward: run the batch, get predictions and the loss.
2. Backward: loss.backward() — PyTorch computes all gradients itself (autograd).
3. Optimizer step: optimizer.step() — update the weights; optimizer.zero_grad() — reset the gradients (they accumulate otherwise — rookie bug #1).

The training vocabulary

• Batch — a chunk of data per step; epoch — a full pass over the data.
• Optimizers: SGD with momentum; Adam/AdamW — adaptive step, the default to start with.
• The learning rate is the most important hyperparameter. Too big → the loss jumps/explodes; too small → forever.
• Regularization: Dropout (randomly switch off neurons during training), weight decay, early stopping, data augmentation.

Learning curves — the training's ECG

• train loss ↓, val loss ↓ — learning;
• train loss ↓, val loss ↑ — overfitting has begun, right at this epoch;
• both flat — underfitting or a too-small LR.

---