Sample Spaces and the Axioms of Probability

Intuition first

Before you can compute a probability, you have to be precise about what could have happened. Almost every probability mistake — including the ones that show up in machine learning — traces back to being sloppy about that list.

Roll a die. The list of things that could happen is 1, 2, 3, 4, 5, 6. Call that list the sample space. A "probability" is then just a way of spreading one unit of belief across that list, and the rules of probability are the rules for spreading it sensibly: no negative amounts, and the total must come to exactly 1.

That is genuinely all probability is. Everything else in this module is bookkeeping on top of those two constraints.

The three ingredients

An experiment is any procedure whose result is not known in advance. It does not need to involve dice or coins — "pick a user at random and record whether they churned" is an experiment.

An outcome is one single result of the experiment. The set of all possible outcomes is the sample space, written $\Omega$ (capital omega).

An event is any set of outcomes — that is, any subset of $\Omega$. Events are the things we assign probabilities to.

For a single die roll:

$$\Omega = \{1, 2, 3, 4, 5, 6\}$$

The event "the roll is even" is the subset $A = \{2, 4, 6\}$. The event "the roll exceeds 4" is $B = \{5, 6\}$. Notice that events are sets, not numbers — that is what lets us combine them with union, intersection and complement.

Why the sample space must be chosen carefully

Consider tossing two coins. A tempting sample space is

$$\Omega_{\text{bad}} = \{\text{zero heads}, \text{one head}, \text{two heads}\}$$

and if you assume all three are equally likely you will conclude $\Prob(\text{one head}) = 1/3$. That is wrong, and experiment will tell you so.

The problem is that those three results are not equally likely, because "one head" can happen two different ways. The honest sample space distinguishes the coins:

$$\Omega = \{HH, HT, TH, TT\}$$

Now all four outcomes really are equally likely, and

$$\Prob(\text{one head}) = \frac{|\{HT, TH\}|}{|\Omega|} = \frac{2}{4} = \frac{1}{2}$$

Kolmogorov's axioms

A probability measure is a function $\Prob$ that assigns a number to each event, subject to exactly three requirements.

Axiom 1 — non-negativity. For every event $A$,

$$\Prob(A) \geq 0$$

Axiom 2 — normalisation. The certain event has probability one:

$$\Prob(\Omega) = 1$$

Axiom 3 — additivity. If $A_1, A_2, \dots$ are mutually exclusive (no two can happen together, so $A_i \cap A_j = \varnothing$ whenever $i \neq j$), then

$$\Prob\!\left(\bigcup_{i} A_i\right) = \sum_{i} \Prob(A_i)$$

That is the entire foundation. Every other rule you have ever seen is a consequence of these three, not an extra assumption.

Consequences you can derive

The complement rule

$$\Prob(A^c) = 1 - \Prob(A)$$

Derivation of the complement ruleAdvanced

$A$ and $A^c$ are mutually exclusive by construction — nothing can be both in $A$ and not in $A$. Together they exhaust the sample space:

$$A \cup A^c = \Omega, \qquad A \cap A^c = \varnothing$$

Apply Axiom 3 to these two disjoint events:

$$\Prob(A) + \Prob(A^c) = \Prob(A \cup A^c) = \Prob(\Omega)$$

By Axiom 2 the right-hand side is $1$, so

$$\Prob(A) + \Prob(A^c) = 1 \quad\Longrightarrow\quad \Prob(A^c) = 1 - \Prob(A)$$

This rule is worth more than it looks. Problems phrased with "at least one" are almost always easier through the complement, because the complement of "at least one" is the single case "none".

The empty event has probability zero

$$\Prob(\varnothing) = 0$$

This follows instantly: $\varnothing = \Omega^c$, so $\Prob(\varnothing) = 1 - \Prob(\Omega) = 1 - 1 = 0$.

Probabilities never exceed one

$$0 \leq \Prob(A) \leq 1$$

Why probability cannot exceed 1Advanced

From the complement rule, $\Prob(A) = 1 - \Prob(A^c)$. Axiom 1 applied to $A^c$ gives $\Prob(A^c) \geq 0$. Subtracting a non-negative quantity from $1$ cannot exceed $1$, so $\Prob(A) \leq 1$. Combined with Axiom 1 on $A$ itself, we get $0 \leq \Prob(A) \leq 1$.

Inclusion–exclusion for two events

For events that may overlap:

$$\Prob(A \cup B) = \Prob(A) + \Prob(B) - \Prob(A \cap B)$$

Derivation of inclusion–exclusionAdvanced

Split $A \cup B$ into three genuinely disjoint pieces:

$$A \cup B = (A \cap B^c) \;\cup\; (A \cap B) \;\cup\; (A^c \cap B)$$

These are "A only", "both", and "B only". No outcome belongs to two of them, so Axiom 3 applies:

$$\Prob(A \cup B) = \Prob(A \cap B^c) + \Prob(A \cap B) + \Prob(A^c \cap B)$$

Now split $A$ and $B$ the same way:

$$\Prob(A) = \Prob(A \cap B^c) + \Prob(A \cap B)$$$$\Prob(B) = \Prob(A^c \cap B) + \Prob(A \cap B)$$

Add these two:

$$\Prob(A) + \Prob(B) = \Prob(A \cap B^c) + \Prob(A^c \cap B) + 2\Prob(A \cap B)$$

The right-hand side is the three-piece sum plus one extra copy of $\Prob(A \cap B)$. Therefore

$$\Prob(A) + \Prob(B) = \Prob(A \cup B) + \Prob(A \cap B)$$

Rearranging gives the result. The overlap is counted twice when you add $\Prob(A)$ and $\Prob(B)$, so it must be subtracted once.

Equally likely outcomes

When $\Omega$ is finite and every outcome is equally likely, the axioms force a simple counting formula:

$$\Prob(A) = \frac{|A|}{|\Omega|}$$

Why the counting formula follows from the axiomsAdvanced

Let $|\Omega| = n$ and let each single outcome have probability $p$. The singletons $\{\omega_1\}, \dots, \{\omega_n\}$ are mutually exclusive and their union is $\Omega$, so by Axioms 3 and 2:

$$\sum_{i=1}^{n} p = np = \Prob(\Omega) = 1 \quad\Longrightarrow\quad p = \frac{1}{n}$$

An event $A$ is the union of its $|A|$ singletons, each of probability $1/n$, so $\Prob(A) = |A|/n = |A|/|\Omega|$.

This is why counting techniques matter: for equally likely outcomes, computing a probability is a counting problem.

Solved problem 1 · A single die roll

A fair six-sided die is rolled once. Let $A$ be "the result is even" and $B$ be "the result is greater than 3". Find $\Prob(A)$, $\Prob(B)$, $\Prob(A \cap B)$, $\Prob(A \cup B)$ and $\Prob(A^c)$.

Step 1 — write down the sample space and the events

$$\Omega = \{1,2,3,4,5,6\}, \qquad |\Omega| = 6$$$$A = \{2,4,6\}, \qquad B = \{4,5,6\}$$

Step 2 — individual probabilities by counting

$$\Prob(A) = \frac{|A|}{|\Omega|} = \frac{3}{6} = \frac{1}{2}$$$$\Prob(B) = \frac{|B|}{|\Omega|} = \frac{3}{6} = \frac{1}{2}$$

Step 3 — the intersection

Outcomes that are both even and greater than 3:

$$A \cap B = \{4, 6\} \quad\Longrightarrow\quad \Prob(A \cap B) = \frac{2}{6} = \frac{1}{3}$$

Step 4 — the union, two ways

By inclusion–exclusion:

$$\Prob(A \cup B) = \tfrac{1}{2} + \tfrac{1}{2} - \tfrac{1}{3} = 1 - \tfrac{1}{3} = \tfrac{2}{3}$$

Check by direct counting: $A \cup B = \{2,4,5,6\}$, which has 4 elements, so $4/6 = 2/3$. The two agree.

Note what would have happened without the correction term: $1/2 + 1/2 = 1$, claiming the union is certain. But rolling a $1$ satisfies neither event, so that is plainly wrong.

Step 5 — the complement

$$\Prob(A^c) = 1 - \Prob(A) = 1 - \tfrac{1}{2} = \tfrac{1}{2}$$

Indeed $A^c = \{1,3,5\}$, the odd numbers.

Answer

$\Prob(A) = \tfrac12$, $\Prob(B) = \tfrac12$, $\Prob(A \cap B) = \tfrac13$, $\Prob(A \cup B) = \tfrac23$, $\Prob(A^c) = \tfrac12$.

Solved problem 2 · Two dice, and the power of the complement

Two fair dice are rolled. What is the probability that at least one shows a six?

Step 1 — build the sample space

Treat the dice as distinguishable, so an outcome is an ordered pair $(d_1, d_2)$:

$$\Omega = \{(1,1), (1,2), \dots, (6,6)\}, \qquad |\Omega| = 6 \times 6 = 36$$

All 36 pairs are equally likely.

Step 2 — try it directly, and see the difficulty

Let $A$ = "at least one six". Directly, $A$ contains: six outcomes where the first die is a six, six where the second is, minus the double-counted $(6,6)$. That is inclusion–exclusion again:

$$|A| = 6 + 6 - 1 = 11$$

Step 3 — now via the complement, which is easier

The complement of "at least one six" is "no sixes at all". Each die then has 5 allowed faces:

$$|A^c| = 5 \times 5 = 25$$$$\Prob(A^c) = \frac{25}{36}$$

Step 4 — apply the complement rule

$$\Prob(A) = 1 - \Prob(A^c) = 1 - \frac{25}{36} = \frac{11}{36} \approx 0.3056$$

This matches Step 2, as it must.

Answer

$\Prob(\text{at least one six}) = \dfrac{11}{36} \approx 0.306$.

Why this matters later

These definitions are not ceremony. They return directly:

• Bayes' theorem partitions $\Omega$ into disjoint cases and applies Axiom 3.
• Classification metrics are counts over a sample space of predictions; a confusion matrix is a partition of $\Omega$ into four disjoint events.
• The union bound $\Prob(\bigcup A_i) \leq \sum \Prob(A_i)$ — the workhorse of generalisation bounds — is inclusion–exclusion with the overlap terms dropped.

Exercise 1

Given $\Prob(A) = 0.6$, $\Prob(B) = 0.5$ and $\Prob(A \cap B) = 0.2$, find $\Prob(A \cup B)$ and the probability that neither event occurs.

Show solutionHide solution

By inclusion–exclusion:

$$\Prob(A \cup B) = 0.6 + 0.5 - 0.2 = 0.9$$

"Neither occurs" is the complement of the union:

$$\Prob\big((A \cup B)^c\big) = 1 - 0.9 = 0.1$$

Sanity check: forgetting the overlap term gives $0.6 + 0.5 = 1.1$, an impossible probability — a useful signal that the correction is missing.

Exercise 2

A fair coin is tossed three times. Use the complement rule to find the probability of getting at least one head.

Show solutionHide solution

The complement of "at least one head" is "no heads", which is the single outcome $TTT$ out of $2^3 = 8$ equally likely sequences:

$$\Prob(\text{no heads}) = \frac{1}{8}$$$$\Prob(\text{at least one head}) = 1 - \frac{1}{8} = \frac{7}{8} = 0.875$$

Exercise 3

A card is drawn from a standard 52-card deck. A student computes $\Prob(\text{king or heart}) = \tfrac{4}{52} + \tfrac{13}{52} = \tfrac{17}{52}$. Is this correct? Justify your answer using the axioms.

Show solutionHide solution

No. Axiom 3 licenses plain addition only for mutually exclusive events, and these two overlap — the king of hearts is both.

With $\Prob(K) = \tfrac{4}{52}$, $\Prob(H) = \tfrac{13}{52}$ and $\Prob(K \cap H) = \tfrac{1}{52}$:

$$\Prob(K \cup H) = \frac{4}{52} + \frac{13}{52} - \frac{1}{52} = \frac{16}{52} = \frac{4}{13}$$

The student's $17/52$ counts the king of hearts twice.

---

Next: Events, Independence and Mutual Exclusivity, where we pin down the distinction that Exercise 3 depends on — and which is probably the most commonly confused pair of terms in the subject.