Machine Learning Foundations

Learning from data versus explicit programming, the three paradigms, and an honest account of what ML cannot do.

Input and output spaces, hypothesis classes, loss functions, and turning a vague goal into an objective.

Assumes: What Machine Learning Actually Is

True risk versus empirical risk, why we optimise a proxy, and the approximation–estimation decomposition.

Assumes: Formulating a Learning Problem

Diagnosing capacity problems from learning curves, with the classic polynomial-fit demonstration.

Assumes: Formulating a Learning Problem

Full algebraic decomposition of expected squared error into bias, variance and noise, with a simulation.

Assumes: Generalisation, Overfitting and Underfitting

Why no learner dominates across all problems, and what that means for model selection in practice.

Assumes: The Bias–Variance Trade-off

Shattering, VC dimension, sample complexity bounds, and the theory behind how much data is enough.

Assumes: Empirical Risk Minimisation · Probability Inequalities

The role of each split, why the test set must stay untouched, and stratification.

Assumes: Generalisation, Overfitting and Underfitting

k-fold, stratified, leave-one-out and nested CV, with the bias–variance trade-off in choosing k.

Assumes: Train, Validation and Test Splits

Grid, random and Bayesian optimisation, successive halving, and budgeting search honestly.

Assumes: Cross-Validation · Bayesian Optimisation

Confusion matrix, accuracy, precision, recall, F1, specificity and Cohen's kappa, all computed by hand.

Assumes: Train, Validation and Test Splits

Threshold sweeps, AUC interpretation, and why PR curves beat ROC under heavy imbalance.

Assumes: Classification Metrics

MSE, RMSE, MAE, MAPE, R² and adjusted R², and which to report for which audience.

Assumes: Train, Validation and Test Splits

Reliability diagrams, Brier score, Platt scaling and isotonic regression.

Assumes: ROC and Precision–Recall Curves

Resampling, SMOTE, class weights, threshold tuning, and choosing metrics that survive skew.

Assumes: ROC and Precision–Recall Curves

Transformations, interactions, binning, domain features, and why this still outperforms model tinkering.

Assumes: Encoding Categorical Features

Filter, wrapper and embedded methods; mutual information, RFE and stability selection.

Assumes: Feature Engineering

L1 and L2 penalties, elastic net, the constrained-optimisation view, and why L1 induces sparsity.

Assumes: The Bias–Variance Trade-off · Lagrange Multipliers

Volume concentration, distance concentration, sample-density collapse, and its consequences for kNN and kernels.

Assumes: The Bias–Variance Trade-off

The many ways leakage sneaks in — scaling before splitting, target encoding, temporal leaks — and how pipelines prevent it.

Assumes: Cross-Validation