Reinforcement Learning

Agent, environment, reward and the exploration–exploitation dilemma, contrasted with supervised learning.

The simplest RL problem: action values, regret, and incremental mean updates.

Assumes: The Reinforcement Learning Problem

Epsilon-greedy, optimistic initialisation, UCB and Thompson sampling, compared on regret.

Assumes: Multi-Armed Bandits

States, actions, transitions, rewards and the Markov property; the MDP as the formal RL object.

Assumes: The Reinforcement Learning Problem · Markov Chains

Discounted return, stochastic policies, and the state- and action-value functions.

Assumes: Markov Decision Processes

Expectation and optimality equations derived from first principles, in scalar and matrix form.

Assumes: Returns, Policies and Value Functions

Computing v_pi by repeated sweeps, with a full numeric trace on a grid world.

Assumes: The Bellman Equations

Alternating evaluation and greedy improvement, the policy improvement theorem, and convergence.

Assumes: Iterative Policy Evaluation

Folding improvement into the update, the contraction-mapping argument, and stopping criteria.

Assumes: Policy Iteration

Learning from complete episodes, first-visit vs every-visit, and off-policy importance sampling.

Assumes: Value Iteration

TD(0), bootstrapping, the TD error, and the bias–variance comparison with Monte Carlo.

Assumes: Monte Carlo Methods

On-policy TD control, the quintuple update, and the cliff-walking comparison with Q-learning.

Assumes: Temporal Difference Learning

Off-policy control, the max operator, convergence conditions, and a Q-table updated step by step.

Assumes: SARSA

Reducing update variance, maximisation bias, and the double estimator fix.

Assumes: Q-Learning

Interpolating between TD and Monte Carlo, the lambda-return, and TD(lambda) with traces.

Assumes: Q-Learning

Replacing tables with parameterised functions, the deadly triad, and semi-gradient methods.

Assumes: n-Step Methods and Eligibility Traces

Neural Q-functions, experience replay, target networks, and the Atari result.

Assumes: Value Function Approximation · Adaptive Optimisers: AdaGrad to AdamW

Double DQN, duelling architectures, prioritised replay, noisy nets and distributional RL.

Assumes: Deep Q-Networks

Parameterising policies directly, the log-derivative trick, and the full theorem derivation.

Assumes: Value Function Approximation

Monte Carlo policy gradient, baselines for variance reduction, and the full algorithm.

Assumes: The Policy Gradient Theorem

Combining learned value estimates with policy gradients, and the advantage function.

Assumes: REINFORCE

Synchronous and asynchronous parallel actor-critics, and why parallelism stabilises training.

Assumes: Actor–Critic Methods

Trust regions, the surrogate objective, the clipped ratio, and why PPO became the RLHF default.

Assumes: Actor–Critic Methods

Continuous action spaces, deterministic policy gradients, and maximum-entropy RL.

Assumes: TRPO and PPO

Learning environment models, Dyna, planning with learned models, and MCTS.

Assumes: Value Iteration

Learning from fixed datasets, distributional shift, conservative Q-learning and behaviour cloning.

Assumes: TRPO and PPO