Intermediate28 min

Handling Class Imbalance

Resampling, SMOTE, class weights, threshold tuning, and choosing metrics that survive skew.

Assumes you know

ROC and Precision–Recall Curves

Handling Class Imbalance

Intuition first

When one class is rare — fraud, disease, equipment failure, ad clicks — two things break at once.

The metric breaks first. At a 0.2% positive rate, predicting "negative" always gives 99.8% accuracy, so accuracy stops distinguishing a useful model from a useless one.

Then the optimisation breaks. Cross-entropy averaged over the dataset is dominated by the 99.8% of easy negatives. The gradient contribution from the rare class is a rounding error, so the model has little incentive to get it right.

What does not break, and this is the part people get wrong, is the underlying statistics. A rare class is not a broken dataset needing repair. Often the best fix is to change the metric and the decision threshold and leave the data entirely alone.

First, decide whether it is a problem

Imbalance alone is not pathological. It becomes a problem when the rare class is both rare and hard to separate, or when you have too few positive examples in absolute terms.

Approach 1: fix the metric (always do this)

Accuracy is unusable. Replace it with measures that ignore the abundant true negatives:

Precision and recall at the operating threshold
Precision–recall AUC / average precision, judged against the positive rate baseline
Recall at a fixed precision, or precision at a fixed capacity, whichever matches the decision being made

Approach 2: move the threshold (usually enough)

A probabilistic classifier trained on imbalanced data is often perfectly good at ranking; the default 0.5 cut is simply in the wrong place.

Solved problem 1 · Threshold beats resampling

A model outputs calibrated probabilities on a dataset with a 2% positive rate. At threshold 0.5 it produces $TP = 12$ , $FP = 3$ , $FN = 188$ , $TN = 9{,}797$ . At threshold 0.08 it produces $TP = 150$ , $FP = 900$ , $FN = 50$ , $TN = 8{,}900$ .

Compute precision and recall at both, and decide which to deploy for a fraud team that can review 1,000 cases.

Step 1 — threshold 0.5

\text{Precision} = \frac{12}{12 + 3} = \frac{12}{15} = 0.800

\text{Recall} = \frac{12}{12 + 188} = \frac{12}{200} = 0.060

Excellent precision, catastrophic recall — 94% of fraud is missed.

Step 2 — threshold 0.08

\text{Precision} = \frac{150}{150 + 900} = \frac{150}{1050} \approx 0.143

\text{Recall} = \frac{150}{150 + 50} = \frac{150}{200} = 0.750

Step 3 — check against the capacity constraint

At threshold 0.08 the model flags $150 + 900 = 1{,}050$ cases. The team can review 1,000, so the threshold needs nudging slightly upward — but the operating point is essentially right.

At threshold 0.5 it flags 15 cases, leaving 985 reviewer slots idle while 188 frauds go uninvestigated.

Step 4 — note what did not change

Both rows come from the same model, the same learned weights, the same ranking. No resampling, no reweighting, no retraining. Only the cut point moved.

Recall went from 0.06 to 0.75 by changing one number.

Answer

Deploy at the low threshold: recall $0.750$ at precision $0.143$ , filling the review capacity. Precision $0.800$ at threshold 0.5 is worthless when it finds only 12 of 200 frauds.

Try threshold tuning before touching the data — it is free, reversible, and often sufficient.

Approach 3: reweight the loss

Tell the optimiser that minority errors cost more:

J(w) = -\frac{1}{n}\sum_{i=1}^{n} c_{y_i}\Big[ y_i \log \hat{p}_i + (1 - y_i)\log(1 - \hat{p}_i) \Big]

The usual choice is inverse frequency, $c_k \propto 1/n_k$ , which is what class_weight="balanced" implements.

Approach 4: resample

Random undersampling discards majority examples. Fast, and it throws away real data — acceptable when the majority class is enormous.

Random oversampling duplicates minority examples. Duplicates encourage memorisation of those exact points.

SMOTE creates synthetic minority points by interpolating between a minority example and one of its minority neighbours:

x_{\text{new}} = x_i + u\,(x_{\text{nn}} - x_i), \qquad u \sim \text{Uniform}(0,1)

python

import numpy as np
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline as ImbPipeline   # NOT sklearn's Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=5000, n_features=20, weights=[0.98, 0.02],
                           n_informative=6, random_state=0)
print(f"positive rate: {y.mean():.3%}  positives: {y.sum()}")

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

variants = {
    "baseline":        ImbPipeline([("sc", StandardScaler()),
                                    ("clf", LogisticRegression(max_iter=2000))]),
    "class_weight":    ImbPipeline([("sc", StandardScaler()),
                                    ("clf", LogisticRegression(max_iter=2000,
                                                               class_weight="balanced"))]),
    "SMOTE":           ImbPipeline([("sc", StandardScaler()),
                                    ("smote", SMOTE(random_state=0)),
                                    ("clf", LogisticRegression(max_iter=2000))]),
}

for name, pipe in variants.items():
    ap = cross_val_score(pipe, X, y, cv=cv, scoring="average_precision")
    print(f"{name:14s} AP = {ap.mean():.4f} ± {ap.std(ddof=1)/np.sqrt(5):.4f}")

Scoring on average_precision rather than accuracy is deliberate. Run this and the three variants are usually within noise of each other on AP — which is the honest and frequently surprising result.

What actually helps when the minority class is genuinely too small

If you have 40 positives, no resampling technique creates information that is not there. The things that do help:

Get more positives. Targeted collection, longer history, weaker labels from a related signal.
Simplify the model. With 40 positives, logistic regression on 5 features will beat a gradient-boosted ensemble on 200.
Use anomaly detection instead. Reframe as one-class: model the normal class and flag deviations. Trains on the abundant class only.
Borrow structure. Transfer learning, or a hierarchical model that shares strength across related rare categories.
Change the target. Predict a common precursor event instead of the rare outcome.

Exercise 1

A colleague applies SMOTE, gets recall 0.88 in cross-validation, and sees recall 0.31 in production. Diagnose.

Show solution

Almost certainly SMOTE applied before the split, or inside scikit-learn's Pipeline rather than imblearn's.

Synthetic minority points are interpolations between real minority neighbours. If resampling happens before splitting, a synthetic point in the training set may be an interpolation of a point sitting in the validation set. The model sees a near-copy of validation data, so validation recall is inflated.

A second possibility with the same signature: the threshold was chosen on a resampled validation set. After SMOTE the apparent positive rate is 50%, so a threshold of 0.5 is reasonable there — and badly wrong on real data at 2%.

Diagnosis steps:

Rerun with imblearn.pipeline.Pipeline so resampling touches training folds only. If cross-validated recall drops towards 0.31, that was the leak.
Check whether the deployed threshold was calibrated on resampled or original data.
Compare average precision with and without SMOTE. If AP is unchanged, SMOTE was never adding value and the entire gain was leakage plus threshold shift.

Exercise 2

A dataset has 1,000,000 negatives and 300 positives. Would you undersample the majority to 300? Explain.

Show solution

Not to 300. That discards 999,700 real examples — the overwhelming majority of the information about what normal looks like — and leaves 600 rows total, from which almost nothing can be learned about a decision boundary.

Better options, roughly in order:

Leave the data alone, train with class_weight="balanced", and tune the threshold on a stratified validation set using average precision. Try this first; it is free.

Moderate undersampling, to perhaps 30,000 negatives — a 100:1 ratio rather than 3,333:1. This keeps a rich picture of the majority class while cutting compute substantially. Combine with class weights.

Ensemble over undersampled subsets — BalancedBaggingClassifier or EasyEnsemble. Train many models, each on all 300 positives plus a different majority subsample, then average. This uses all the majority data across the ensemble without any single model being swamped.

Reframe as anomaly detection. With 300 positives and a million negatives, modelling normality with an isolation forest or one-class approach may outperform supervised classification, since it exploits the abundant class rather than discarding it.

The overriding constraint is that 300 positives caps model complexity regardless of ratio. Keep the model simple, the feature count low, and the evaluation intervals wide — recall estimated on 60 held-out positives has a standard error of roughly 6 points.

Next: Probability Calibration, which matters especially here — reweighting and resampling both distort predicted probabilities, even when they improve ranking.