Beginner22 min

Train, Validation and Test Splits

The role of each split, why the test set must stay untouched, and stratification.

Assumes you know

Generalisation, Overfitting and Underfitting

Train, Validation and Test Splits

Intuition first

You need to answer three different questions, and each needs its own data.

What should the model's parameters be? — answered by fitting, using the training set.
Which model, and which settings, should I choose? — answered by comparing, using the validation set.
How well will this actually perform? — answered once, at the end, using the test set.

The reason these cannot share data is that each question involves choosing based on the answer, and choosing based on a measurement contaminates that measurement. Fit on the training set and its error becomes optimistic. Tune against the validation set enough times and its error becomes optimistic too. Only data used for no decision whatsoever gives an honest number.

The test set is not a third opinion. It is a single-use instrument.

The three roles

Split	Typical share	Used for	How many times
Training	60–80%	Fitting parameters	Every epoch
Validation	10–20%	Model and hyperparameter selection, early stopping	Many
Test	10–20%	Final unbiased estimate	Once

Data flows one way. Decisions are made using training and validation data; the test set enters only at the final step.

How large should each be?

Not a fixed ratio — it depends on what precision you need from the estimate.

The standard error of an accuracy estimate on $m$ held-out examples is

\text{SE} = \sqrt{\frac{p(1-p)}{m}}

where $p$ is the accuracy. This gives a principled way to size the test set: decide how precise the final number must be, then solve for $m$ .

Solved problem 1 · Sizing a test set

You need the final accuracy reported to within $\pm 1$ percentage point at 95% confidence, and expect accuracy near 0.90. How many test examples are needed?

Step 1 — translate the requirement

A 95% confidence interval is roughly $\hat p \pm 1.96\,\text{SE}$ . Requiring a half-width of $0.01$ :

1.96 \sqrt{\frac{p(1-p)}{m}} \leq 0.01

Step 2 — substitute p = 0.90

p(1-p) = 0.90 \times 0.10 = 0.09

1.96\sqrt{\frac{0.09}{m}} \leq 0.01

Step 3 — solve for m

Divide by 1.96 and square both sides:

\frac{0.09}{m} \leq \left(\frac{0.01}{1.96}\right)^2 = (0.005102)^2 = 2.603 \times 10^{-5}

m \geq \frac{0.09}{2.603 \times 10^{-5}} \approx 3458

Step 4 — interpret

About 3,500 test examples. Note the requirement did not mention the total dataset size at all — precision depends on the absolute number held out, not the percentage.

Answer

$m \approx 3{,}500$ . With 10 million rows, a 20% test split wastes 2 million examples that would be better used for training; with 4,000 rows, no split can give $\pm 1$ point and you must use cross-validation and report wider intervals honestly.

Splitting correctly

A uniformly random split is correct only when rows are independent and identically distributed. Several common situations break that.

Stratify for classification

With a 2% positive class and a random 20% split, the test set's positive count is itself random and could easily be far off 2%. Stratified splitting preserves class proportions in every split, which reduces variance in the estimate at no cost.

Split by group, not by row

If a patient contributes five admissions, or a user has forty sessions, random row splitting puts the same patient in both training and test. The model can then recognise the individual rather than the pattern, and the test score measures memorisation of identities.

Split on the group key — patient id, user id, document id — so all rows for a group land on one side.

Split by time for temporal data

For anything where you will predict the future from the past, a random split lets the model see the future during training. Split chronologically: train on the earliest period, validate on the next, test on the most recent. This also matches deployment, where the model always faces data later than its training data.

Order of operations

Every preprocessing step that learns anything from the data must be fitted on the training split only, then applied unchanged to the others. Standardisation learns a mean and standard deviation. Imputation learns a fill value. Target encoding learns per-category averages. All are leakage if fitted before the split.

python

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=4000, n_features=20, weights=[0.9, 0.1],
                           random_state=0)

# Hold the test set out first, stratified so class balance is preserved.
X_rest, X_test, y_rest, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=0)

# Then carve validation out of what remains.
X_train, X_val, y_train, y_val = train_test_split(
    X_rest, y_rest, test_size=0.25, stratify=y_rest, random_state=0)

print(f"train {len(y_train)}  val {len(y_val)}  test {len(y_test)}")
print(f"positive rate — train {y_train.mean():.3f}  val {y_val.mean():.3f}  test {y_test.mean():.3f}")

# A Pipeline guarantees the scaler is fitted on training folds only.
model = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000)),
])

model.fit(X_train, y_train)
print(f"validation accuracy {model.score(X_val, y_val):.3f}")
# Touch X_test exactly once, when everything is finalised.

How the validation set degrades

Each hyperparameter comparison against the validation set extracts a little information from it. After a few hundred comparisons, the selected configuration is partly fitted to the validation set, and validation error underestimates true error by a margin that grows like $\sqrt{\ln m}$ in the number of configurations tried.

Mitigations, in order of practicality:

Prefer cross-validation over a single split, which averages out split-specific noise.
Keep a count of how many configurations you tried, and treat the winner's margin sceptically if it is smaller than the standard error.
Refresh the validation set periodically on long-running projects.
Keep the test set genuinely sealed. If it must be reused, report that you did.

Solved problem 2 · Why the split ratio surprises people

A dataset has 1,000,000 rows. Compare an 80/10/10 split against a 98/1/1 split for a model whose accuracy is around 0.95.

Step 1 — held-out sizes

\text{80/10/10}: \quad 100{,}000 \text{ validation}, \; 100{,}000 \text{ test}

\text{98/1/1}: \quad 10{,}000 \text{ validation}, \; 10{,}000 \text{ test}

Step 2 — precision of the test estimate in each case

With $p = 0.95$ , $p(1-p) = 0.0475$ .

m = 100{,}000: \quad \text{SE} = \sqrt{\frac{0.0475}{100{,}000}} = \sqrt{4.75\times10^{-7}} \approx 0.00069

m = 10{,}000: \quad \text{SE} = \sqrt{\frac{0.0475}{10{,}000}} = \sqrt{4.75\times10^{-6}} \approx 0.00218

Step 3 — compare to the extra training data

The tighter split gives a 95% interval half-width of $\pm 0.14$ points; the smaller one $\pm 0.43$ points. Both are far more precise than anyone needs.

Meanwhile the 98/1/1 split provides $980{,}000$ training rows instead of $800{,}000$ — 22.5% more data, which will measurably improve the model.

Answer

For a million rows, 98/1/1 is the better choice: both estimates are already precise to well under half a percentage point, and the extra 180,000 training rows do real work. The habitual 80/20 split is calibrated for datasets of a few thousand rows and is simply wasteful at scale.

Exercise 1

You are predicting whether a customer support ticket will be escalated. Some customers have submitted hundreds of tickets. Explain what goes wrong with a random row split and how to fix it.

Show solution

With a random split, tickets from the same customer appear in both training and test. Customers have persistent, idiosyncratic characteristics — writing style, product mix, baseline irritation — so the model can learn "tickets from customer 8842 tend to escalate" rather than any general signal about escalation.

The test score then measures the model's ability to recognise customers it has already seen, which will be high. In deployment on a new customer, none of that memorised information is available and accuracy drops.

The fix is a grouped split: partition on customer_id so every ticket from a given customer lands entirely in training or entirely in test. In scikit-learn, GroupShuffleSplit or StratifiedGroupKFold with groups=customer_id.

Worth checking afterwards: the group split will usually show a lower score than the random split. That drop is not a regression — it is the removal of an illusion, and the grouped number is the one that predicts production behaviour.

Exercise 2

A colleague reports: "I tried 300 hyperparameter combinations and the best validation accuracy was 0.882. Then test accuracy came out at 0.851, so the test set must be harder." Assess this.

Show solution

The explanation is almost certainly wrong. The gap is the expected consequence of selecting a winner from 300 candidates on one validation set.

Each configuration's validation accuracy is a noisy estimate of its true accuracy. Taking the maximum over 300 noisy estimates systematically favours whichever configuration got the most favourable validation split, so the winning score is biased upward. With a validation set of, say, 2,000 examples and accuracy near 0.88, the standard error is

\sqrt{\frac{0.88 \times 0.12}{2000}} \approx 0.0073

and the optimism from maximising over 300 candidates is on the order of $0.0073 \times \sqrt{2\ln 300} \approx 0.026$ — very close to the 0.031 gap observed.

So 0.851 is the honest estimate and 0.882 was inflated by the search. Calling the test set "harder" invites the real mistake, which would be to keep tuning until the test score improves — at which point there is no clean estimate left at all.

The right response: accept 0.851 as the reported figure, and use nested cross-validation next time if an unbiased estimate of the selection procedure is needed.

Next: Cross-Validation, which replaces a single fragile validation split with an average over several.