Natural Language Processing

Why language is hard for machines: ambiguity, compositionality and the classical processing stages.

Case folding, Unicode normalisation, accent stripping, whitespace and punctuation handling.

Assumes: The NLP Pipeline

Character classes, quantifiers, groups, lookarounds, and catastrophic backtracking.

Assumes: Text Normalisation

Porter and Snowball stemmers versus dictionary lemmatisation, and the precision trade-off.

Assumes: Text Normalisation

The Markov assumption over text, maximum likelihood estimates, and generating from an n-gram model.

Assumes: Markov Chains · Stemming and Lemmatisation

Laplace, add-k, Good–Turing, backoff, interpolation and Kneser–Ney, worked on a small corpus.

Assumes: n-Gram Language Models

Cross-entropy and perplexity derived, computed by hand, and its pitfalls as a comparison metric.

Assumes: Smoothing

Document–term matrices, vocabulary construction, sparsity, and the loss of word order.

Assumes: Stemming and Lemmatisation

Term frequency and inverse document frequency variants, the full formula derived and computed by hand.

Assumes: Bag of Words and Count Vectors

Cosine similarity ranking, length normalisation, and precision/recall for retrieval.

Assumes: TF-IDF

Distributional semantics, both architectures, and the embedding geometry that makes analogies work.

Assumes: Vector Space Retrieval · Backpropagation

Why the full softmax is intractable, the negative sampling objective derived, and subsampling frequent words.

Assumes: word2vec: CBOW and Skip-Gram

Global co-occurrence factorisation, and subword embeddings that handle unseen words.

Assumes: Negative Sampling and Hierarchical Softmax

Similarity, analogy arithmetic, anisotropy, and measuring and mitigating social bias in embeddings.

Assumes: GloVe and fastText

Tagsets, ambiguity, rule-based and statistical tagging, and HMM taggers.

Assumes: Hidden Markov Models

The BIO scheme, linear-chain conditional random fields, and why CRFs beat independent classification.

Assumes: Part-of-Speech Tagging

Entity types, span-based evaluation, nested entities, and modern neural NER.

Assumes: Sequence Labelling and CRFs

Constituency and dependency grammars, CKY parsing, transition-based parsing and treebanks.

Assumes: Named Entity Recognition

Encoder–decoder translation, the bottleneck problem, and what motivated attention.

Assumes: Sequence-to-Sequence Models

Queries, keys and values; additive vs dot-product attention, and an attention matrix computed by hand.

Assumes: Neural Machine Translation

The full encoder–decoder stack: attention sublayers, feed-forward blocks, residuals and normalisation.

Assumes: The Attention Mechanism · Layer, Group and RMS Normalisation

Splitting into heads, scaled dot-product attention, causal masking and padding masks.

Assumes: The Transformer Architecture

Why attention is permutation-invariant, sinusoidal encodings derived, and learned alternatives.

Assumes: Multi-Head and Masked Attention

Every tensor shape from token ids to logits for a concrete small model, with parameter counts.

Assumes: Positional Encoding

Bidirectional pretraining, the MLM and NSP objectives, and fine-tuning for downstream tasks.

Assumes: Transformer Shapes: End-to-End Walkthrough

BERT vs GPT vs T5: which architecture suits which task, and why decoder-only won for generation.

Assumes: BERT and Masked Language Modelling

BPE merges traced by hand, WordPiece, Unigram and SentencePiece, plus vocabulary-size trade-offs.

Assumes: Bag of Words and Count Vectors

Classical baselines through fine-tuned transformers, with class imbalance and multi-label handling.

Assumes: BERT and Masked Language Modelling

LSA via SVD, probabilistic LSA, LDA with its generative story, coherence metrics and BERTopic.

Assumes: TF-IDF · PCA via SVD

Extractive versus abstractive approaches, and the faithfulness problem in generated summaries.

Assumes: Encoder, Decoder and Encoder–Decoder Families

Extractive span prediction, open-domain QA, and the retrieval bridge to RAG.

Assumes: BERT and Masked Language Modelling

BLEU and ROUGE computed by hand, plus METEOR, chrF, BERTScore and their known weaknesses.

Assumes: Summarisation