Pipelines and Data Leakage

The three kinds

1. Preprocessing leakage

Any transformation that learns from data must learn from training data only. Standardisation learns a mean and standard deviation. Imputation learns a fill value. PCA learns components. Target encoding learns per-category averages.

Fit any of them on the full dataset and every fold's "held-out" rows have already influenced the transformation.

# WRONG — the scaler has seen the test set
X_scaled = StandardScaler().fit_transform(X)
X_train, X_test = train_test_split(X_scaled)

# RIGHT — split first, fit only on train
X_train, X_test = train_test_split(X)
scaler = StandardScaler().fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)      # transform, never fit

The optimism from this alone is usually small — a point or two — which is why it survives code review. It becomes severe with target encoding or feature selection.

2. Target leakage

A feature contains information derived from the outcome, or recorded only after it.

Real examples:

• Predicting hospital readmission using discharge_medication_count, recorded at discharge — after the stay the model is meant to predict from.
• Predicting churn using days_since_last_login, which is mechanically small for active users and large for churned ones.
• Predicting loan default using total_payments_made, which is low precisely because the loan defaulted.
• Predicting fraud using chargeback_flag, which is a consequence of confirmed fraud.

3. Group and temporal leakage

The same entity appearing on both sides of a split, or training data that postdates validation data. Covered in the splits lesson; the mechanism is that the model recognises the entity or the era rather than the pattern.

Pipelines make it structural

A pipeline bundles preprocessing with the estimator into one object with a single fit. When cross-validation refits that object per fold, every transformation is refitted on that fold's training portion automatically.

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold

numeric = ["age", "income", "tenure"]
categorical = ["region", "plan"]

# Every step here LEARNS something, so every step must sit inside the pipeline.
preprocess = ColumnTransformer([
    ("num", Pipeline([
        ("impute", SimpleImputer(strategy="median")),   # learns medians
        ("scale", StandardScaler()),                    # learns mean and sd
    ]), numeric),
    ("cat", Pipeline([
        ("impute", SimpleImputer(strategy="most_frequent")),
        ("encode", OneHotEncoder(handle_unknown="ignore")),  # learns categories
    ]), categorical),
])

model = Pipeline([
    ("prep", preprocess),
    ("select", SelectKBest(f_classif, k=10)),   # learns which features — leaks badly outside
    ("clf", LogisticRegression(max_iter=1000)),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(model, df[numeric + categorical], y, cv=cv)
print(f"{scores.mean():.4f} ± {scores.std(ddof=1) / np.sqrt(len(scores)):.4f}")

SelectKBest is the important line. Feature selection performed once on the whole dataset is one of the largest sources of optimism there is, because it searches thousands of candidates using the labels. Inside a pipeline it is redone per fold.

A leakage checklist

Run through this before trusting any result.

1. Is any feature recorded at or after the moment of the outcome? Check timestamps, not intuition.
2. Would this column exist at prediction time in production? Trace where it is populated.
3. Is any preprocessing fitted outside the cross-validation loop? Scalers, imputers, encoders, selectors, dimensionality reduction.
4. Are there duplicate or near-duplicate rows across splits?
5. Do rows share an entity — user, patient, device, document? If so, group the split.
6. Is the data temporal? If so, split by time.
7. Was the test set used more than once? If yes, it is a validation set now.
8. Is the result surprisingly good? Treat that as evidence of a bug until proven otherwise.

When the pipeline is not enough

Pipelines protect against preprocessing leakage. They cannot detect target leakage, because they have no way to know that days_since_last_login is downstream of churn. That requires understanding how the data was produced.

Two practices that help:

• Build a data dictionary recording, for each column, when it is populated relative to the prediction moment.
• Simulate the production query. Reconstruct the feature set as it would have existed at a historical timestamp, and train on that. Tedious, and it catches what nothing else does.

Next: Feature Engineering, where the features themselves are constructed — carefully, and inside the pipeline.