Supervised Learning

Fitting a line by least squares, deriving the closed-form slope and intercept, and interpreting them.

Assumes: Least Squares and the Normal Equations

The matrix formulation, the normal equations, multicollinearity, and interpreting partial coefficients.

Assumes: Simple Linear Regression

Linearity, independence, homoscedasticity and normality; residual plots, and what to do when each fails.

Assumes: Multiple Linear Regression · Regression Inference and Diagnostics

Modelling curvature with linear machinery, basis functions, splines, and the overfitting cliff.

Assumes: Multiple Linear Regression

The L2-penalised objective, its closed-form solution, the effect on the SVD spectrum, and choosing lambda.

Assumes: Regularisation · Multiple Linear Regression

L1 geometry and sparsity, coordinate descent, the soft-thresholding operator, and elastic net as a hybrid.

Assumes: Ridge Regression · Proximal Gradient Methods

The sigmoid, log-odds, the cross-entropy objective derived from MLE, and gradient-based fitting.

Assumes: Maximum Likelihood Estimation · Gradient Descent

Softmax, one-vs-rest versus multinomial formulations, and the gradient of softmax cross-entropy.

Assumes: Logistic Regression

Fisher's criterion, the generative Gaussian view, the shared-covariance assumption, and LDA for dimensionality reduction.

Assumes: The Multivariate Normal Distribution · Eigenvalues and Eigenvectors

Relaxing the equal-covariance assumption, the quadratic boundary, and the parameter-count trade-off.

Assumes: Linear Discriminant Analysis

The conditional independence assumption, Gaussian/multinomial/Bernoulli variants, and Laplace smoothing.

Assumes: Bayes' Theorem

Instance-based learning, distance metrics, choosing k, weighting, and the cost of prediction.

Assumes: The Curse of Dimensionality

Recursive partitioning, information gain, gain ratio and Gini impurity, with a tree built by hand.

Binary splits, cost-complexity pruning, regression trees, and surrogate splits for missing values.

Assumes: Decision Trees: Entropy and Gini

The update rule, the convergence theorem for separable data, and the XOR limitation.

Maximum-margin classification, the geometry of the margin, hard and soft margins, and the hinge loss.

Assumes: The Perceptron · Convex Sets and Convex Functions

Deriving the dual via Lagrangian duality, KKT conditions, and why only support vectors have nonzero multipliers.

Assumes: Support Vector Machines · KKT Conditions · Linear Programming Duality

The kernel trick, Mercer's condition, polynomial and RBF kernels, and implicit feature spaces.

Assumes: The SVM Dual Problem

Stacking layers to defeat XOR, the forward pass, and why nonlinearity is essential — the bridge to deep learning.

Assumes: The Perceptron

Bootstrap aggregation, variance reduction arithmetic, feature subsampling, and out-of-bag error.

Assumes: CART, Pruning and Regression Trees · Resampling: Bootstrap and Permutation

Sequential reweighting, the derivation of the alpha coefficients, and the exponential-loss interpretation.

Assumes: Bagging and Random Forests

Boosting as gradient descent in function space, residual fitting, shrinkage and subsampling.

Assumes: AdaBoost

Second-order objectives, regularised tree growth, histogram binning, leaf-wise growth and tuning strategy.

Assumes: Gradient Boosting

Meta-learners over base-model predictions, out-of-fold construction, and leakage avoidance.

Assumes: Gradient Boosting

Coefficients, permutation importance, partial dependence, LIME and SHAP, with their assumptions and failure modes.

Assumes: XGBoost, LightGBM and CatBoost · The Shapley Value