Intermediate28 min

Variance, Moments and Generating Functions

Variance and standard deviation, higher moments, and using MGFs to derive distributions of sums.

Assumes you know

Expectation

Variance, Moments and Generating Functions

Intuition first

The mean tells you where a distribution sits. It says nothing about how spread out it is. A company where everyone earns £50,000 and one where half earn £20,000 and half £80,000 have the same mean and completely different realities.

Variance measures that spread as the average squared distance from the mean. Squaring does two things: it makes deviations positive so they cannot cancel, and it penalises large deviations disproportionately — which is why variance is sensitive to outliers, and why the standard deviation, its square root, is what gets reported since it lives in the original units.

Beyond variance there are higher moments — skewness for asymmetry, kurtosis for tail weight — and a remarkable device, the moment generating function, that packages every moment into a single function and turns questions about sums of random variables into questions about products.

Variance and standard deviation

\Var(X) = \E\big[(X - \mu)^2\big], \qquad \sigma = \text{SD}(X) = \sqrt{\Var(X)}

The computational form is almost always easier:

\boxed{\Var(X) = \E[X^2] - \big(\E[X]\big)^2}

Deriving the computational formulaAdvanced

Expand the square inside the expectation, treating $\mu = \E[X]$ as the constant it is:

\Var(X) = \E\big[(X - \mu)^2\big] = \E\big[X^2 - 2\mu X + \mu^2\big]

Apply linearity of expectation:

= \E[X^2] - 2\mu\,\E[X] + \mu^2

Now substitute $\E[X] = \mu$ :

= \E[X^2] - 2\mu^2 + \mu^2 = \E[X^2] - \mu^2

Two things worth noticing. The formula is more convenient because $\E[X^2]$ needs only one pass over the distribution — you do not have to know $\mu$ first. And it immediately proves $\E[X^2] \geq (\E[X])^2$ , since variance cannot be negative.

The catch: numerically, subtracting two large nearly-equal numbers loses precision. For data with large mean and small variance, the two-pass formula $\frac{1}{n}\sum(x_i - \bar{x})^2$ is more accurate, which is why production code uses Welford's algorithm rather than accumulating $\sum x^2$ .

Properties

\Var(aX + b) = a^2 \Var(X)

Adding a constant shifts the distribution without changing its spread, so $b$ vanishes. Scaling by $a$ scales deviations by $a$ and squared deviations by $a^2$ .

\Var(X + Y) = \Var(X) + \Var(Y) + 2\Cov(X, Y)

\Var(X + Y) = \Var(X) + \Var(Y) \qquad \textbf{only if } X \ind Y

Notation used in this lesson
Symbol	Meaning	Read aloud
Var(X) or σ²	Variance — mean squared deviation	variance of X
σ	Standard deviation, in the units of X	sigma
E[Xᵏ]	kth raw moment about the origin	k-th moment
E[(X−μ)ᵏ]	kth central moment	k-th central moment
Mₓ(t)	Moment generating function, E[e^{tX}]	M of t

Solved problem 1 · Variance of a fair die, three ways

Compute $\Var(X)$ for a fair six-sided die.

Step 1 — the two ingredients

From the expectation lesson:

\E[X] = 3.5, \qquad \E[X^2] = \frac{91}{6} \approx 15.1667

Step 2 — computational formula

\Var(X) = \E[X^2] - (\E[X])^2 = \frac{91}{6} - \left(\frac{7}{2}\right)^2 = \frac{91}{6} - \frac{49}{4}

Common denominator 12:

= \frac{182}{12} - \frac{147}{12} = \frac{35}{12} \approx 2.9167

Step 3 — verify by the definition

Deviations from 3.5 and their squares:

(1-3.5)^2 = 6.25, \quad (2-3.5)^2 = 2.25, \quad (3-3.5)^2 = 0.25

(4-3.5)^2 = 0.25, \quad (5-3.5)^2 = 2.25, \quad (6-3.5)^2 = 6.25

\Var(X) = \frac{6.25 + 2.25 + 0.25 + 0.25 + 2.25 + 6.25}{6} = \frac{17.5}{6} \approx 2.9167 \;\checkmark

Step 4 — standard deviation

\sigma = \sqrt{35/12} \approx 1.7078

Interpretable: a typical roll is about 1.7 away from 3.5.

Answer

$\Var(X) = 35/12 \approx 2.917$ , $\sigma \approx 1.708$ .

Solved problem 2 · Variance of a sum, with and without independence

Two fair dice. Let $S = X_1 + X_2$ be the sum and $D = X_1 - X_2$ the difference. Find $\Var(S)$ and $\Var(D)$ . Then find $\Var(2X_1)$ and compare with $\Var(S)$ .

Step 1 — independence and the sum

The dice are independent, so covariance is zero:

\Var(S) = \Var(X_1) + \Var(X_2) = \frac{35}{12} + \frac{35}{12} = \frac{70}{12} = \frac{35}{6} \approx 5.833

Step 2 — the difference

\Var(D) = \Var(X_1) + (-1)^2\Var(X_2) = \frac{35}{12} + \frac{35}{12} = \frac{35}{6} \approx 5.833

Identical to the sum. Subtracting an independent quantity adds variance exactly as adding one does.

Step 3 — doubling a single die

\Var(2X_1) = 2^2 \Var(X_1) = 4 \times \frac{35}{12} = \frac{140}{12} = \frac{35}{3} \approx 11.667

Step 4 — compare, and note why they differ

\Var(2X_1) \approx 11.667 \quad\text{versus}\quad \Var(X_1 + X_2) \approx 5.833

Exactly double. Both have mean 7, but $2X_1$ can only take even values $2, 4, \dots, 12$ with equal probability, while $X_1 + X_2$ concentrates around 7 because there are more ways to make 7 than to make 2.

This is averaging at work: two independent draws partially cancel each other's deviations, whereas doubling one draw doubles its deviation with nothing to offset it.

Answer

$\Var(S) = \Var(D) = 35/6 \approx 5.833$ ; $\Var(2X_1) = 35/3 \approx 11.667$ , twice as large. Independent draws average out; scaling one draw does not.

Higher moments

Skewness — third standardised moment, measuring asymmetry:

\text{Skew}(X) = \E\left[\left(\frac{X - \mu}{\sigma}\right)^3\right]

Positive means a long right tail (income, claim sizes); negative means a long left tail.

Kurtosis — fourth standardised moment, measuring tail weight:

\text{Kurt}(X) = \E\left[\left(\frac{X - \mu}{\sigma}\right)^4\right]

The normal distribution has kurtosis 3, so excess kurtosis subtracts 3 to make normal the zero point. Positive excess kurtosis means heavier tails than normal — more extreme events than a Gaussian model would predict, which is the standard failure of financial risk models.

Moment generating functions

M_X(t) = \E\big[e^{tX}\big]

when the expectation exists for $t$ in a neighbourhood of zero. Its usefulness comes from two properties.

Moments by differentiation:

\E[X^k] = M_X^{(k)}(0) = \left.\frac{d^k M_X}{dt^k}\right|_{t=0}

Sums become products: for independent $X$ and $Y$ ,

M_{X+Y}(t) = M_X(t)\, M_Y(t)

Why both properties holdAdvanced

Moments. Expand the exponential as a power series and take expectations term by term:

M_X(t) = \E\left[1 + tX + \frac{t^2X^2}{2!} + \frac{t^3X^3}{3!} + \dots\right] = 1 + t\,\E[X] + \frac{t^2}{2!}\E[X^2] + \frac{t^3}{3!}\E[X^3] + \dots

So $M_X$ is a power series whose coefficient on $t^k$ is $\E[X^k]/k!$ . Differentiating $k$ times and evaluating at $t = 0$ isolates exactly that term, leaving $\E[X^k]$ . Every moment is encoded in this one function — hence "generating".

Sums. For independent $X$ and $Y$ , the variables $e^{tX}$ and $e^{tY}$ are also independent, so the expectation factorises:

M_{X+Y}(t) = \E\big[e^{t(X+Y)}\big] = \E\big[e^{tX}e^{tY}\big] = \E\big[e^{tX}\big]\E\big[e^{tY}\big] = M_X(t)M_Y(t)

This converts convolution — the genuinely awkward operation of finding the distribution of a sum — into multiplication. Combined with the fact that an MGF determines the distribution uniquely, it gives a clean method: multiply the MGFs, recognise the result, and you have the distribution of the sum.

It is how one proves that a sum of independent Poissons is Poisson, a sum of independent normals is normal, and a sum of independent exponentials is gamma. It is also the engine behind one standard proof of the Central Limit Theorem.

Solved problem 3 · Moments from an MGF

$X$ has MGF $M_X(t) = (1 - 2t)^{-3}$ for $t < 1/2$ . Find $\E[X]$ and $\Var(X)$ .

Step 1 — first derivative

M_X(t) = (1 - 2t)^{-3}

M_X'(t) = -3(1 - 2t)^{-4} \cdot (-2) = 6(1 - 2t)^{-4}

Step 2 — evaluate at zero for the mean

\E[X] = M_X'(0) = 6(1)^{-4} = 6

Step 3 — second derivative

M_X''(t) = 6 \cdot (-4)(1 - 2t)^{-5} \cdot (-2) = 48(1 - 2t)^{-5}

\E[X^2] = M_X''(0) = 48

Step 4 — variance

\Var(X) = \E[X^2] - (\E[X])^2 = 48 - 36 = 12

Step 5 — identify the distribution

The form $(1 - \beta t)^{-\alpha}$ is the gamma MGF with shape $\alpha = 3$ and scale $\beta = 2$ . Gamma has mean $\alpha\beta = 6$ and variance $\alpha\beta^2 = 3 \times 4 = 12$ , matching both results.

Answer

$\E[X] = 6$ , $\Var(X) = 12$ . The MGF identifies the distribution as Gamma(3, 2).

python

import numpy as np

rng = np.random.default_rng(0)

d1 = rng.integers(1, 7, 400_000)
d2 = rng.integers(1, 7, 400_000)

for name, v, theory in (
    ("Var(X)",      d1,        35/12),
    ("Var(X1+X2)",  d1 + d2,   35/6),
    ("Var(X1-X2)",  d1 - d2,   35/6),
    ("Var(2X1)",    2 * d1,    35/3),
):
    print(f"{name:12s} empirical {v.var():8.4f}   theory {theory:8.4f}")

# Standard error of the mean: sigma / sqrt(n).
for n in (10, 100, 1000):
    means = rng.integers(1, 7, size=(20_000, n)).mean(axis=1)
    print(f"n={n:<5} SD of sample mean {means.std():.4f}  "
          f"theory {np.sqrt(35/12)/np.sqrt(n):.4f}")

Exercise 1

$X$ has mean 10 and variance 4. Find the mean and variance of $Y = 3X - 5$ .

Show solution

Mean, by linearity:

\E[Y] = 3\,\E[X] - 5 = 3(10) - 5 = 25

Variance — the additive constant drops out, the multiplier squares:

\Var(Y) = 3^2 \Var(X) = 9 \times 4 = 36

So $\sigma_Y = 6$ , three times $\sigma_X = 2$ . The standard deviation scales by $\lvert a \rvert$ while the variance scales by $a^2$ , which is the practical reason to report standard deviations: they transform the way intuition expects.

Exercise 2

Independent $X$ and $Y$ each have variance 9. A student computes $\Var(X - Y) = 0$ , reasoning that the variances cancel. Correct them.

Show solution

\Var(X - Y) = \Var(X) + (-1)^2\Var(Y) = 9 + 9 = 18

The error is treating variance as linear. The scaling rule is $\Var(aX) = a^2\Var(X)$ , and for $a = -1$ we get $(-1)^2 = +1$ — the minus sign is squared away, so the variance of $Y$ is added.

Intuitively: subtracting an independent random quantity does not cancel randomness, it introduces more. Each of $X$ and $Y$ fluctuates independently, and the difference fluctuates more than either alone.

A concrete check: if $X$ and $Y$ are independent fair-coin values in $\{0, 1\}$ , then $X - Y$ takes values $-1, 0, 1$ with probabilities $\tfrac14, \tfrac12, \tfrac14$ — genuinely more spread out than either input, which lives in $\{0,1\}$ .

Zero variance would require $X$ and $Y$ to be perfectly correlated, $Y = X$ , in which case $\Cov(X,Y) = 9$ and $\Var(X - Y) = 9 + 9 - 2(9) = 0$ .

Next: Covariance and Correlation, which is the missing term in the variance-of-a-sum formula.