Neural Networks and Deep Learning

Why stacking linear layers is pointless, what nonlinearity buys, and the anatomy of a deep network.

Assumes: Multi-Layer Perceptrons and Feed-Forward Networks

What the theorem actually promises, what it does not, and why depth beats width in practice.

Assumes: From Perceptron to Deep Networks

Layer-by-layer computation with explicit matrix shapes, batching, and a full numeric forward pass.

Assumes: From Perceptron to Deep Networks

The algorithm derived from the multivariable chain rule, as a computation graph and in matrix form.

Assumes: Forward Propagation · The Multivariable Chain Rule

Every number in a 2-2-1 network for one full training step, forward and backward, verified by finite differences.

Assumes: Backpropagation

Sigmoid, tanh, ReLU, Leaky ReLU, ELU, GELU, SiLU and softmax, with derivatives and saturation behaviour.

Assumes: Forward Propagation

MSE, MAE, Huber, binary and categorical cross-entropy, KL divergence, and matching loss to task.

Assumes: Activation Functions

Why zeros and large randoms both fail; Xavier/Glorot and He initialisation derived from variance analysis.

Assumes: Backpropagation

Batch, stochastic and mini-batch gradient descent, batch-size effects, and the noise-as-regulariser view.

Assumes: Gradient Descent

Accumulating velocity through ravines, and the look-ahead correction of Nesterov momentum.

Assumes: SGD and Mini-Batching

Per-parameter learning rates, AdaGrad's decay problem, RMSProp, Adam's bias correction, and AdamW's decoupled decay.

Assumes: Momentum and Nesterov Acceleration

Step, exponential and cosine decay, warmup, cyclical rates, and the LR-range test.

Assumes: Adaptive Optimisers: AdaGrad to AdamW

Why gradients decay or blow up with depth, diagnosis by norm tracking, and clipping.

Assumes: Weight Initialisation

Normalising activations, learnable scale and shift, train/inference discrepancy, and the running statistics trap.

Assumes: Vanishing and Exploding Gradients

Normalising across features instead of the batch, and why transformers use LayerNorm and RMSNorm.

Assumes: Batch Normalisation

Random unit deactivation, inverted dropout at inference, and the ensemble interpretation.

Assumes: Regularisation

Patience and restoration, plus augmentation strategies for images, text and tabular data.

Assumes: Dropout

Kernels, stride, padding and dilation; output-size arithmetic and a convolution computed by hand.

Assumes: Forward Propagation

Max and average pooling, global pooling, and computing the receptive field of a deep stack.

Assumes: The Convolution Operation

LeNet, AlexNet, VGG, Inception and EfficientNet, and the design principles each introduced.

Assumes: Pooling and Receptive Fields

The degradation problem, residual blocks, and why identity shortcuts make gradients flow.

Assumes: CNN Architectures

Feature extraction vs fine-tuning, layer freezing, discriminative learning rates and domain shift.

Assumes: ResNets and Skip Connections

IoU, anchor boxes, non-maximum suppression, R-CNN family, YOLO, SSD and mAP evaluation.

Assumes: Transfer Learning and Fine-Tuning CNNs

Fully convolutional networks, U-Net, transposed convolution, Mask R-CNN and Dice loss.

Assumes: Object Detection

Hidden state recurrence, weight sharing across time, and the sequence-modelling task types.

Assumes: Backpropagation

Unrolling the graph, gradient accumulation across timesteps, truncated BPTT, and why long dependencies fail.

Assumes: Recurrent Neural Networks

Cell state, forget/input/output gates, the constant error carousel, and a gate-by-gate numeric trace.

Assumes: Backpropagation Through Time

Update and reset gates, the parameter saving over LSTM, and empirical comparisons.

Assumes: Long Short-Term Memory

Encoder–decoder architecture, the fixed-vector bottleneck, teacher forcing and beam search.

Assumes: Gated Recurrent Units

Encoder, bottleneck and decoder; reconstruction loss, and the relationship to PCA.

Assumes: Backpropagation · Principal Component Analysis

Corruption-based training, sparsity penalties, contractive regularisation and representation quality.

Assumes: Autoencoders

Latent variable modelling, the ELBO derived in full, the reparameterisation trick and posterior collapse.

Assumes: Denoising, Sparse and Contractive Autoencoders · Expectation–Maximisation

The minimax game, discriminator and generator objectives, and the optimal-discriminator analysis.

Assumes: Variational Autoencoders · Zero-Sum Games and the Minimax Theorem

Mode collapse, vanishing discriminator gradients, DCGAN, WGAN with gradient penalty, StyleGAN and FID.

Assumes: Generative Adversarial Networks

Forward noising and reverse denoising, the training objective, DDPM vs DDIM, and classifier-free guidance.

Assumes: Variational Autoencoders

Pretext tasks, InfoNCE, SimCLR, MoCo, BYOL and CLIP, and why negatives matter.

Assumes: Denoising, Sparse and Contractive Autoencoders

A systematic checklist: overfit one batch, check shapes and gradients, and read loss curves diagnostically.

Assumes: Learning Rate Schedules

The same network implemented in both frameworks: tensors, autograd, modules, training loops and deployment.

Assumes: Debugging Deep Networks