Advanced26 min

Probability Calibration

Reliability diagrams, Brier score, Platt scaling and isotonic regression.

Assumes you know

ROC and Precision–Recall Curves

Probability Calibration

Intuition first

A model says "70% chance of rain" on a hundred days. If it rained on about seventy of them, the model is calibrated — its numbers mean what they say. If it rained on thirty, the model may still be useful for ranking which days are wettest, but the number 70% is a lie.

Ranking and calibration are separate abilities. AUC measures only the first: it is unchanged if you square every probability, cube it, or pass it through any increasing function. So a model can rank perfectly and still be systematically over- or under-confident.

Calibration matters the moment a probability is used as a quantity rather than a sort key — expected value calculations, cost-sensitive thresholds, combining model output with other evidence, or showing a number to a human who will act on it.

When it matters, and when it does not

Use	Needs calibration?
Rank leads, review top 100	No — ranking suffices
Multiply by transaction value to get expected loss	Yes
Apply a cost-optimal threshold	Yes
Show "23% risk" to a clinician	Yes
Combine several models' outputs	Yes
Pick the argmax class	No

Measuring calibration

Reliability diagram

Bin predictions by confidence, and plot the mean predicted probability in each bin against the observed frequency. Perfect calibration is the diagonal.

Reliability diagram. The diagonal is perfect calibration. The curve below it is over-confident — predicted probabilities exceed observed frequencies, typical of boosted trees and deep networks.

Expected Calibration Error

\text{ECE} = \sum_{b=1}^{B} \frac{n_b}{n}\,\Big\lvert \; \overline{y}_b - \overline{p}_b \;\Big\rvert

where bin $b$ holds $n_b$ predictions, $\overline{p}_b$ is their mean predicted probability and $\overline{y}_b$ the observed positive rate.

Brier score

\text{BS} = \frac{1}{n}\sum_{i=1}^{n}\big(\hat{p}_i - y_i\big)^2

Mean squared error on probabilities. It decomposes usefully:

\text{BS} = \underbrace{\text{calibration error}}_{\text{fixable by post-processing}} - \underbrace{\text{refinement}}_{\text{how informative}} + \underbrace{\text{irreducible}}_{\text{base rate variance}}

Solved problem 1 · Computing ECE and Brier score

A model's predictions are grouped into four bins:

Bin	Count	Mean predicted	Observed positive rate
0.0–0.25	400	0.10	0.05
0.25–0.50	300	0.35	0.20
0.50–0.75	200	0.62	0.45
0.75–1.00	100	0.88	0.70

Compute ECE and describe the miscalibration.

Step 1 — absolute gap per bin

\lvert 0.05 - 0.10 \rvert = 0.05

\lvert 0.20 - 0.35 \rvert = 0.15

\lvert 0.45 - 0.62 \rvert = 0.17

\lvert 0.70 - 0.88 \rvert = 0.18

Step 2 — bin weights

Total $n = 400 + 300 + 200 + 100 = 1{,}000$ .

\frac{400}{1000} = 0.40, \quad \frac{300}{1000} = 0.30, \quad \frac{200}{1000} = 0.20, \quad \frac{100}{1000} = 0.10

Step 3 — weighted sum

\text{ECE} = 0.40(0.05) + 0.30(0.15) + 0.20(0.17) + 0.10(0.18)

= 0.020 + 0.045 + 0.034 + 0.018

= 0.117

Step 4 — read the pattern

Every observed rate is below its predicted probability, and the gap widens with confidence: 0.05 in the lowest bin, 0.18 in the highest.

This is systematic over-confidence. When the model says 88%, the truth is 70%.

Step 5 — what this costs in practice

Suppose these are fraud probabilities used to compute expected loss on £1,000 transactions. In the top bin the model claims expected loss

0.88 \times £1{,}000 = £880

when the truth is

0.70 \times £1{,}000 = £700

A 26% overstatement, which would cause systematic over-blocking of legitimate customers if the block decision compares expected loss against a fixed cost.

Answer

$\text{ECE} = 0.117$ — an average absolute miscalibration of 11.7 percentage points, with consistent over-confidence that worsens at high confidence. Ranking may be fine; the numbers are not usable as probabilities without correction.

Which models are miscalibrated, and how

Model	Typical behaviour	Cause
Logistic regression	Well calibrated	Optimises log-loss directly, which is a proper scoring rule
Naive Bayes	Badly over-confident	Independence assumption multiplies correlated evidence repeatedly
SVM	No probabilities at all	Optimises margin; `decision_function` is not a probability
Random forest	Under-confident at extremes	Averaging votes pulls predictions towards 0.5
Gradient boosting	Over-confident	Optimises a margin-like loss aggressively
Deep networks	Over-confident, worsening with capacity	Trained to near-zero training loss; softmax saturates

Fixing it

Both methods fit a one-dimensional correction on held-out data.

Platt scaling

Fit a logistic regression on the model's scores:

\hat{p}_{\text{cal}} = \frac{1}{1 + \exp(A\,s + B)}

Two parameters, so it works on small calibration sets. Assumes the distortion is sigmoidal — usually right for SVMs and boosted trees.

Isotonic regression

Fit the best non-decreasing step function mapping score to probability. Non-parametric, so it corrects any monotone distortion, but needs more data — roughly 1,000 calibration examples — and will overfit below that.

python

import numpy as np
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import brier_score_loss, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=8000, n_features=20, n_informative=8,
                           random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)

raw = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_tr, y_tr)

# cv=5 fits base model and calibrator on disjoint folds — no leakage.
cal = CalibratedClassifierCV(
    RandomForestClassifier(n_estimators=200, random_state=0),
    method="isotonic", cv=5,
).fit(X_tr, y_tr)

def ece(y_true, p, bins=10):
    edges = np.linspace(0, 1, bins + 1)
    total = 0.0
    for lo, hi in zip(edges[:-1], edges[1:]):
        m = (p > lo) & (p <= hi)
        if m.sum():
            total += m.mean() * abs(y_true[m].mean() - p[m].mean())
    return total

for name, clf in (("raw", raw), ("isotonic", cal)):
    p = clf.predict_proba(X_te)[:, 1]
    print(f"{name:9s} AUC={roc_auc_score(y_te, p):.4f}  "
          f"Brier={brier_score_loss(y_te, p):.4f}  ECE={ece(y_te, p):.4f}")

Run it and the pattern is consistent: AUC barely moves while Brier and ECE improve. Calibration is a monotone transformation, so it cannot change the ranking — it only relabels the scores with honest numbers.

Exercise 1

A model has AUC 0.94 and ECE 0.21. Should you retrain with a different architecture?

Show solution

No. Those two numbers say the model is an excellent ranker with dishonest probabilities — a post-processing problem, not a modelling one.

AUC 0.94 means the scores separate classes well; whatever the model learned about the signal, it learned. ECE 0.21 means the probability values are off by 21 percentage points on average, which is a monotone distortion of an otherwise good score.

The fix is calibration on held-out data: Platt scaling if the calibration set is small, isotonic if there are at least about a thousand examples. Expect ECE to drop substantially and AUC to stay at roughly 0.94, because a monotone map cannot alter ranking.

Retraining with a different architecture would risk losing the 0.94 while doing nothing that a two-parameter correction achieves for free. The only reason to retrain would be to raise AUC itself — a separate objective.

Exercise 2

Explain why training with class_weight="balanced" damages calibration, and how to recover it.

Show solution

Class weighting changes the effective base rate the model is fitted against. Weighting a 2% positive class by $1/0.02 = 50$ makes the weighted training distribution roughly balanced, so the model learns to output probabilities appropriate to a 50% prevalence world, not a 2% one.

The result is systematic over-prediction: a genuinely 2%-risk case may receive a predicted probability near 0.5. Ranking is largely preserved — weighting rescales rather than reorders — so AUC looks fine while the probabilities are badly wrong, sometimes by an order of magnitude.

Two ways to recover:

Calibrate afterwards on an unweighted held-out set that reflects the true prevalence. Isotonic regression will learn the compression back towards the real base rate.
Correct analytically. For a model trained with the positive class up-weighted by factor $c$ , the calibrated odds are recovered by dividing the predicted odds by $c$ : $\text{odds}_{\text{cal}} = \frac{1}{c}\cdot\frac{\hat p}{1 - \hat p}$ then converting back to a probability.

The cleaner alternative is to skip weighting altogether: train unweighted, keep calibrated probabilities, and handle imbalance by moving the decision threshold — which is exactly the argument in the class imbalance lesson.

Next: Feature Engineering.