Beginner30 min

Classification Metrics

Confusion matrix, accuracy, precision, recall, F1, specificity and Cohen's kappa, all computed by hand.

Assumes you know

Train, Validation and Test Splits

Classification Metrics

Intuition first

Accuracy — the fraction you got right — is the obvious metric and often the wrong one.

Suppose 1 in 1,000 transactions is fraudulent. A model that declares every transaction legitimate is 99.9% accurate and catches zero fraud. Accuracy rewards it for ignoring the only thing you cared about.

The fix is to stop collapsing performance into one number too early. Count the four possible outcomes separately — correctly flagged, wrongly flagged, correctly cleared, wrongly cleared — and then build the metric that matches the cost of each mistake in your situation. Missing a tumour and unnecessarily alarming a healthy patient are not equally bad, and no single number knows that unless you tell it.

The confusion matrix

Everything is computed from four counts.

The four outcomes. Rows are what is true, columns are what the model said. Every metric in this lesson is a ratio of these four numbers.

Notation used in this lesson
Symbol	Meaning	Read aloud
TP	True positives — predicted positive, actually positive	T P
FP	False positives — predicted positive, actually negative	F P
FN	False negatives — predicted negative, actually positive	F N
TN	True negatives — predicted negative, actually negative	T N
P	Total actual positives, TP + FN	P
N	Total actual negatives, FP + TN	N

The metrics

\text{Accuracy} = \frac{TP + TN}{TP + FP + FN + TN}

\text{Precision} = \frac{TP}{TP + FP} \qquad \text{(of those flagged, how many were right?)}

\text{Recall} = \frac{TP}{TP + FN} \qquad \text{(of those that mattered, how many did we catch?)}

\text{Specificity} = \frac{TN}{TN + FP} \qquad \text{(of the negatives, how many were cleared?)}

F_1 = 2 \cdot \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}

Which denominator do you care about?

Situation	Costly error	Optimise
Cancer screening	Missing a case (FN)	Recall
Spam filtering	Losing a real email (FP)	Precision
Fraud review team with 50 slots/day	Wasting reviewer time	Precision@50
Search results	Both, balanced	$F_1$ or NDCG
Legal document discovery	Missing evidence (FN)	Recall, often at 95%+

Solved problem 1 · Working the whole confusion matrix

A fraud model is evaluated on 10,000 transactions. 200 are genuinely fraudulent. The model flags 340 transactions, of which 150 are truly fraudulent.

Compute every metric above, and assess the model.

Step 1 — recover all four counts

Given: total $= 10{,}000$ , actual positives $P = 200$ , predicted positives $= 340$ , and $TP = 150$ .

FP = 340 - 150 = 190

FN = P - TP = 200 - 150 = 50

TN = 10{,}000 - TP - FP - FN = 10{,}000 - 150 - 190 - 50 = 9{,}610

Check: $150 + 190 + 50 + 9{,}610 = 10{,}000 \;\checkmark$

Step 2 — accuracy

\text{Accuracy} = \frac{150 + 9{,}610}{10{,}000} = \frac{9{,}760}{10{,}000} = 0.9760

Step 3 — the baseline accuracy, for comparison

A model predicting "never fraud" gets all 9,800 negatives right:

\frac{9{,}800}{10{,}000} = 0.9800

The trained model's 97.60% is worse than the do-nothing baseline's 98.00%. This is why accuracy is useless here.

Step 4 — precision

\text{Precision} = \frac{150}{150 + 190} = \frac{150}{340} \approx 0.4412

Of every 100 transactions flagged, about 44 are genuinely fraudulent.

Step 5 — recall

\text{Recall} = \frac{150}{150 + 50} = \frac{150}{200} = 0.7500

Three quarters of all fraud is caught.

Step 6 — specificity

\text{Specificity} = \frac{9{,}610}{9{,}610 + 190} = \frac{9{,}610}{9{,}800} \approx 0.9806

Step 7 — F₁

F_1 = 2 \times \frac{0.4412 \times 0.7500}{0.4412 + 0.7500} = 2 \times \frac{0.3309}{1.1912}

= 2 \times 0.2778 = 0.5556

Step 8 — assessment

The model is genuinely useful despite accuracy below baseline. It catches 75% of fraud while sending only 340 cases for review instead of 10,000 — a 29-fold reduction in review volume.

Whether 44% precision is acceptable depends on review cost. If a reviewer takes five minutes per case, 340 cases is 28 hours and 190 of those hours-worth are wasted. If the average fraud costs £800, the 150 caught cases save £120,000. The trade is clearly worth it.

Answer

Accuracy $0.9760$ (below the $0.9800$ do-nothing baseline); precision $0.4412$ ; recall $0.7500$ ; specificity $0.9806$ ; $F_1 = 0.5556$ .

The correct summary is "catches 75% of fraud at 44% precision", not "97.6% accurate".

The threshold is a separate decision

Most classifiers output a score, and the confusion matrix depends on where you cut it. A model has one set of scores but many confusion matrices.

\hat{y} = \begin{cases} 1 & \text{if } \hat{p} \geq t \\ 0 & \text{otherwise} \end{cases}

Raising $t$ raises precision and lowers recall. Lowering $t$ does the reverse. The default $t = 0.5$ has no special status — it is a convention, and for imbalanced data it is usually a bad one.

Multi-class: averaging choices matter

With $K$ classes, precision and recall are computed per class and then averaged, and the averaging method changes the answer substantially.

Macro — unweighted mean over classes. Every class counts equally, so rare classes dominate the score's variability. Use when all classes matter equally.
Micro — pool all TP, FP, FN across classes before computing. Equivalent to accuracy for single-label problems. Large classes dominate.
Weighted — mean weighted by class support. A compromise, but can hide terrible performance on rare classes.

Computing these correctly

python

import numpy as np
from sklearn.metrics import (
    confusion_matrix, classification_report, precision_recall_fscore_support,
)

y_true = np.array([1]*200 + [0]*9800)

# Reconstruct the worked example: 150 TP, 50 FN, 190 FP, 9610 TN.
y_pred = np.concatenate([
    np.ones(150), np.zeros(50),          # actual positives
    np.ones(190), np.zeros(9610),        # actual negatives
]).astype(int)

tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
print(f"TP={tp}  FP={fp}  FN={fn}  TN={tn}")

precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)
print(f"precision={precision:.4f}  recall={recall:.4f}  f1={f1:.4f}")

# Always inspect the per-class report rather than one aggregate number.
print(classification_report(y_true, y_pred, target_names=["legit", "fraud"], digits=4))

Note confusion_matrix(...).ravel() returns tn, fp, fn, tp in that order — not the order most people assume, and a frequent source of silently transposed metrics.

Exercise 1

A medical screening model has recall 0.99 and precision 0.08. The disease affects 1 in 500 people. Is this model useless?

Show solution

No — for screening, this is close to the intended design.

Screening exists to decide who gets a second, more expensive, more definitive test. The costly error is a missed case, because a missed cancer is not caught until it is advanced. Recall 0.99 means 99 of every 100 cases proceed to confirmation.

Precision 0.08 means 12 or 13 people are referred for every genuine case. That is the deliberate price: with prevalence of 1/500, low precision is arithmetically unavoidable at high recall, exactly as in the Bayes lesson.

What determines acceptability is the cost of the confirmatory step. If it is a cheap, non-invasive follow-up, 12 unnecessary follow-ups per case caught is a bargain. If it is an invasive biopsy with its own morbidity, 12 is unacceptable and the threshold must be raised, accepting lower recall.

The metric to report here is not $F_1$ — which would be a dismal $0.148$ — but recall with the referral rate alongside it.

Exercise 2

Two models on the same test set of 1,000 examples with 100 positives:

Model X: TP = 90, FP = 300
Model Y: TP = 60, FP = 40

Compute precision, recall and $F_1$ for each, then say which you would deploy for (a) a disease screen, (b) a spam filter.

Show solution

Model X.

\text{Precision} = \frac{90}{90 + 300} = \frac{90}{390} \approx 0.2308

\text{Recall} = \frac{90}{100} = 0.9000

F_1 = 2 \times \frac{0.2308 \times 0.9000}{0.2308 + 0.9000} = 2 \times \frac{0.2077}{1.1308} \approx 0.3673

Model Y.

\text{Precision} = \frac{60}{60 + 40} = 0.6000

\text{Recall} = \frac{60}{100} = 0.6000

F_1 = 2 \times \frac{0.36}{1.20} = 0.6000

(a) Disease screen: Model X. A missed case is far worse than a false referral. X catches 90 of 100 cases against Y's 60 — thirty additional people identified. The cost is 300 unnecessary follow-ups instead of 40.

(b) Spam filter: Model Y. A false positive means a legitimate email is hidden, which users find far worse than seeing occasional spam. X would misfile 300 real emails; Y misfiles 40.

The point: $F_1$ ranks Y far above X ( $0.60$ versus $0.37$ ), yet X is the correct choice for screening. A metric that does not encode your cost ratio will confidently recommend the wrong model.

Next: ROC and Precision–Recall Curves, which evaluate a model across all thresholds at once.