Beginner28 min

Generalisation, Overfitting and Underfitting

Diagnosing capacity problems from learning curves, with the classic polynomial-fit demonstration.

Assumes you know

Formulating a Learning Problem

Generalisation, Overfitting and Underfitting

Intuition first

A student who memorises past exam papers word for word will score perfectly on those papers and badly on a new one. A student who barely studied scores badly on both. Neither has learned the subject; they fail in opposite directions.

Overfitting is memorisation — the model has captured detail specific to the training sample, including its noise, and that detail does not recur.

Underfitting is not learning enough — the model is too rigid to capture the pattern that genuinely is there.

The diagnostic is the same in both cases: compare performance on data used for training against performance on data held back. The pattern of the two numbers tells you which failure you have, and the two failures need opposite treatments. Applying the wrong one makes things worse, which is why diagnosing before acting matters.

The two numbers, and what their pattern means

Training error	Validation error	Diagnosis	What to do
High	High (similar)	Underfitting	More capacity, better features, train longer
Low	High (large gap)	Overfitting	Regularise, more data, less capacity
Low	Low	Healthy	Ship it
High	Low	Suspicious	A bug — check for leakage or a broken split

The generalisation gap

\text{gap} = R(\hat h) - \hat{R}(\hat h)

Estimated in practice as validation error minus training error. Overfitting is a large gap. Note that a large gap is not automatically fatal — a model with 0.02 training and 0.08 validation error may still be the best available. What matters is validation error; the gap only tells you which lever will reduce it.

Model complexity: the classical picture

As you increase flexibility, training error falls monotonically. Validation error falls, then rises. The minimum is the sweet spot.

Training error falls monotonically with complexity; validation error is U-shaped. The vertical line marks the capacity that minimises validation error.

"Complexity" is whatever knob controls flexibility: polynomial degree, tree depth, number of parameters, training epochs, or the inverse of a regularisation strength.

Learning curves: error against dataset size

Plotting error as $n$ grows separates the two failures more reliably than a single pair of numbers.

Learning curves. Left: the curves converge at a high error — a bias problem, so more data will not help. Right: a persistent gap — a variance problem, where more data will help.

Reading them:

Curves converge, both high — bias-dominated. Adding data moves nothing. Change the model.
Curves stay apart, training low — variance-dominated. The gap narrows as $n$ grows, so collecting data is worthwhile and you can extrapolate how much you need.
Validation still falling at the right edge — you are data-limited; more will help.
Validation flat at the right edge — you are model-limited; more will not.

Overfitting is not only about parameter count

Common but incomplete: "too many parameters causes overfitting". More precisely, overfitting grows with effective capacity, which is affected by:

Parameter count — but a heavily regularised 10-million-parameter model can have less effective capacity than an unregularised 50-parameter one.
Training duration — the same architecture overfits more at epoch 200 than at epoch 20. This is why early stopping works.
Number of decisions made using the data — every feature you selected by looking at correlations with the target consumed validation integrity.
Noise level — with noisier labels, the same model overfits more, because there is more noise available to memorise.

Solved problem 1 · Diagnosing four training runs

Each row is a completed run on the same dataset. Diagnose each and give the single most useful next action.

Run	Train MSE	Validation MSE
A	0.91	0.94
B	0.04	0.62
C	0.22	0.26
D	0.48	0.31

Run A — 0.91 / 0.94

Both errors high, gap of 0.03 — negligible. The model is not chasing noise; it cannot represent the signal. Underfitting.

Action: increase capacity. Add interaction or polynomial features, use a non-linear model, or reduce regularisation strength. Collecting more data is wasted effort here.

Run B — 0.04 / 0.62

Training error near zero, validation 15× higher. Gap of 0.58. Severe overfitting — the model has essentially memorised the training set.

Action: reduce variance. Strongest levers in order: more training data, stronger regularisation, reduced capacity, early stopping.

Run C — 0.22 / 0.26

Both moderate, small gap. Healthy fit. Whether 0.26 is good depends on the irreducible noise floor. If labels carry variance around 0.20, this model is close to optimal and further effort is wasted.

Action: establish the noise floor before optimising further — e.g. how well can duplicate measurements predict each other.

Run D — 0.48 / 0.31

Validation error below training error. Not a success — a red flag.

Possible causes: dropout or other noise active during training but not evaluation, which inflates training loss legitimately; an easier validation split by chance or by bad stratification; leakage in the opposite direction; or training loss averaged over early epochs while validation is measured at the end.

Action: debug the pipeline before interpreting any number from this run.

Answer

A underfits — add capacity. B overfits badly — regularise or get more data. C is healthy — establish the noise floor. D indicates a pipeline bug — investigate rather than celebrate.

Producing a learning curve

python

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.model_selection import learning_curve
from sklearn.datasets import make_regression

X, y = make_regression(n_samples=800, n_features=20, noise=12.0, random_state=0)

sizes, train_scores, val_scores = learning_curve(
    Ridge(alpha=1.0), X, y,
    train_sizes=np.linspace(0.1, 1.0, 8),
    cv=5,
    scoring="neg_mean_squared_error",
)

# learning_curve returns *negative* MSE because sklearn maximises scores.
train_mse = -train_scores.mean(axis=1)
val_mse = -val_scores.mean(axis=1)

for n, tr, va in zip(sizes, train_mse, val_mse):
    print(f"n={n:4d}  train={tr:8.1f}  val={va:8.1f}  gap={va - tr:8.1f}")

Read the final rows. If gap is still shrinking and val still falling, more data will pay. If both have flattened, the model is the constraint.

Exercise 1

A model has training accuracy 0.99 and validation accuracy 0.72. Rank these four interventions by expected benefit, with reasoning: (a) add more features, (b) collect 10× more data, (c) increase L2 regularisation, (d) train for more epochs.

Show solution

The 27-point gap is variance-dominated, so rank by how much each reduces variance.

(b) collect 10× more data — the most reliable variance reducer, and it costs no bias. Estimation error falls roughly like $1/\sqrt{n}$ .
(c) increase L2 regularisation — directly shrinks effective capacity. Cheap and immediate; costs a little bias, which is affordable when training accuracy is 0.99.
(a) add more features — wrong direction. Enlarges $\mathcal{H}$ and increases variance. Might help only if a genuinely informative feature is missing, but that is a bias fix applied to a variance problem.
(d) train for more epochs — actively harmful. Training accuracy is already 0.99; more epochs increase memorisation. Early stopping is the useful direction.

Exercise 2

Explain why a model can overfit even when it has fewer parameters than training examples.

Show solution

Parameter count is a crude proxy for capacity, and several things break the correspondence.

A single-parameter model can overfit if that parameter is selected from a huge implicit pool: trying 10,000 candidate features and keeping the one best correlated with the target searches an enormous hypothesis space, even though the final model has one coefficient. The effective capacity is set by the search, not by the survivor.

Capacity also depends on the functional form. A decision tree of depth 20 has relatively few stored numbers but can isolate individual training points. A $k$ -nearest-neighbour model with $k = 1$ has no learned parameters at all and interpolates the training set exactly.

Finally, high label noise gives the model more to memorise. The same architecture on clean labels may generalise while on noisy labels it fits the noise.

Next: The Bias–Variance Trade-off, where the expected error is decomposed algebraically into bias, variance and an irreducible term.