Bayes' theorem

Statement

Let \( (\Omega, \mathcal{F}, \mathbb{P}) \) be a probability space and let \( A, B \in \mathcal{F} \) with \( \mathbb{P}(A) \gt 0 \) and \( \mathbb{P}(B) \gt 0 \). Then \[ \mathbb{P}(A \mid B) \;=\; \frac{\mathbb{P}(B \mid A)\,\mathbb{P}(A)}{\mathbb{P}(B)}. \] More generally (the partition form): let \( \{A_i\}_{i \in I} \) be a finite or countable partition of \( \Omega \) into events of \( \mathcal{F} \) — that is, the \( A_i \) are pairwise disjoint, \( \bigcup_{i \in I} A_i = \Omega \), and \( I \subseteq \mathbb{N} \) — with \( \mathbb{P}(A_i) \gt 0 \) for every \( i \in I \). Then for any \( B \in \mathcal{F} \) with \( \mathbb{P}(B) \gt 0 \) and any \( j \in I \), \[ \mathbb{P}(A_j \mid B) \;=\; \frac{\mathbb{P}(B \mid A_j)\,\mathbb{P}(A_j)}{\sum_{i \in I} \mathbb{P}(B \mid A_i)\,\mathbb{P}(A_i)}. \]

Why it matters

Conditional probability is directional: \( \mathbb{P}(A \mid B) \) and \( \mathbb{P}(B \mid A) \) are different objects, often wildly different in magnitude. Bayes' theorem is the exact exchange rate between them. It converts a forward model — how likely is the observed data \( B \) under each hypothesis \( A_i \) — into an inverse statement: how likely is each hypothesis given the data. Every inference problem of the shape "we know how causes produce effects; we observed an effect; which cause?" runs through this identity.

It is the founding equation of Bayesian statistics: with \( \mathbb{P}(A_j) \) read as a prior, \( \mathbb{P}(B \mid A_j) \) as a likelihood, and \( \mathbb{P}(A_j \mid B) \) as a posterior, the theorem says exactly how evidence must update belief if belief is to obey the probability axioms. It also underlies diagnostic testing, spam filtering, forensic reasoning, signal decoding, and hidden-state estimation.

Hypotheses

Proof

We prove the simple form first, then the law of total probability as an explicit lemma, then the partition form. Throughout, \( (\Omega, \mathcal{F}, \mathbb{P}) \) is a probability space.

Result

Corollaries & converses

• Odds form. For two hypotheses \( A \) and \( A^{c} \) with \( 0 \lt \mathbb{P}(A) \lt 1 \): \( \dfrac{\mathbb{P}(A \mid B)}{\mathbb{P}(A^{c} \mid B)} = \dfrac{\mathbb{P}(B \mid A)}{\mathbb{P}(B \mid A^{c})} \cdot \dfrac{\mathbb{P}(A)}{\mathbb{P}(A^{c})} \) — posterior odds \( = \) likelihood ratio \( \times \) prior odds. Follows by dividing two instances of the theorem; the denominators cancel.
• Symmetry of confirmation. If \( \mathbb{P}(A), \mathbb{P}(B) \gt 0 \) then \( \mathbb{P}(A \mid B) \gt \mathbb{P}(A) \iff \mathbb{P}(B \mid A) \gt \mathbb{P}(B) \iff \mathbb{P}(A \cap B) \gt \mathbb{P}(A)\mathbb{P}(B) \): evidence raises a hypothesis exactly when the hypothesis raises the evidence.
• Sequential updating. If \( B_1, B_2 \) are conditionally independent given each \( A_i \), then updating on \( B_1 \) and then on \( B_2 \) (using the posterior as the new prior) equals updating once on \( B_1 \cap B_2 \). Coherence of Bayesian learning.
• \( \mathbb{P}(A \mid B) = \mathbb{P}(B \mid A) \iff \mathbb{P}(A) = \mathbb{P}(B) \) (both positive): the two conditionals coincide precisely when the marginals do — quantifying exactly when "transposing the conditional" happens to be harmless.
• Converse. Bayes' theorem is an identity, so "the converse" in the implication sense is vacuous. The meaningful converse-type question — do the four quantities \( \mathbb{P}(A), \mathbb{P}(B), \mathbb{P}(A \mid B), \mathbb{P}(B \mid A) \) determine each other? — has answer: any three positive ones satisfying \( \mathbb{P}(A \mid B)\mathbb{P}(B) = \mathbb{P}(B \mid A)\mathbb{P}(A) \) force the fourth, but no two of them determine the rest (e.g. \( \mathbb{P}(A) = \mathbb{P}(B) = \tfrac12 \) is consistent with \( \mathbb{P}(A \mid B) \) being any value in \( [0,1] \): tune the overlap).

Fails without

Common errors

• Transposing the conditional (the prosecutor's fallacy). Reporting \( \mathbb{P}(\text{evidence} \mid \text{innocent}) = 10^{-6} \) as if it were \( \mathbb{P}(\text{innocent} \mid \text{evidence}) \). The two differ by the factor \( \mathbb{P}(\text{innocent})/\mathbb{P}(\text{evidence}) \), which can be enormous.
• Base-rate neglect. Quoting a test's sensitivity as the posterior and ignoring the prior \( \mathbb{P}(A_j) \). A 99%-sensitive test for a 1-in-10,000 condition still yields a tiny posterior (see Worked example 1's mechanism).
• Wrong denominator. Writing \( \mathbb{P}(B \mid A_j)\mathbb{P}(A_j) \) alone as the answer — forgetting to normalise over all partition cells, or normalising over a non-exhaustive list of hypotheses.
• Treating repeat evidence as independent unconditionally. Sequential updating needs conditional independence given each hypothesis; two positives from the same flawed test on the same sample are not independent pieces of evidence.
• Conditioning on the wrong event. In the Monty Hall problem, conditioning on "door 3 has a goat" instead of "the host, following his protocol, opened door 3" — the observed event is the host's action, and its likelihood under each hypothesis is what the theorem consumes.
• Dividing by zero silently. Applying the formula when \( \mathbb{P}(B) = 0 \) or some \( \mathbb{P}(A_i) = 0 \) without noticing the conditional probabilities involved are undefined.

Discussion

The result is named for the Reverend Thomas Bayes, whose "An Essay towards solving a Problem in the Doctrine of Chances" was read to the Royal Society in 1763, two years after his death, by his friend Richard Price. Bayes treated a specific inverse problem — inferring the unknown success probability of a Bernoulli process from observed outcomes — rather than the abstract identity above. The general formulation, its systematic exploitation, and the name "inverse probability" are due chiefly to Laplace (1774 onwards), who used it on problems from demography to celestial mechanics. The modern measure-theoretic setting, in which the theorem becomes a one-line consequence of Kolmogorov's 1933 axioms, stripped it of controversy as a theorem while leaving intact the philosophical dispute about its use: whether prior probabilities over hypotheses are legitimate objects.

That dispute is the frequentist–Bayesian divide. Everyone accepts the identity for events in a probability space; the argument is over whether a parameter \( \theta \) may be treated as a random variable so that \( \mathbb{P}(\theta \in \cdot \mid \text{data}) \) makes sense. The Bayesian answer — yes, with the prior encoding pre-experimental information — turns the theorem into a complete engine of statistical inference: all information about \( \theta \) after the experiment is carried by the posterior. Decision theory adds a striking endorsement: under mild conditions (complete class theorems of Wald), every admissible decision rule is (a limit of) Bayes rules, so even a committed frequentist ignores Bayesian machinery at the cost of admissibility.

Structurally, Bayes' theorem is a statement about reversing a factorisation. The joint probability \( \mathbb{P}(A \cap B) \) admits two decompositions, \( \mathbb{P}(A \mid B)\mathbb{P}(B) \) and \( \mathbb{P}(B \mid A)\mathbb{P}(A) \), and the theorem merely equates them. This is why it generalises so smoothly: to random variables with densities \( \bigl( f_{X \mid Y}(x \mid y) = f_{Y \mid X}(y \mid x) f_X(x) / f_Y(y) \bigr) \), to Markov chains (time reversal), and to graphical models, where belief propagation is Bayes' theorem organised along the edges of a graph. In information-theoretic dress it is the statement that \( \log \) posterior odds \( = \log \) prior odds \( + \log \) likelihood ratio, making evidence additive.

At full rigour the elementary theorem is the tip of a measure-theoretic construction. For a sub-\( \sigma \)-algebra \( \mathcal{G} \subseteq \mathcal{F} \), conditional probability \( \mathbb{P}(A \mid \mathcal{G}) \) is defined as a \( \mathcal{G} \)-measurable random variable via the Radon–Nikodym theorem, unique only up to null sets; conditioning on a null event is then meaningful only through a regular conditional probability, which exists on standard Borel spaces (a nontrivial theorem) and yields the abstract Bayes formula \( \mathbb{P}(A \mid X = x) \) for \( \mathbb{P}_X \)-almost every \( x \). The Borel–Kolmogorov paradox — where conditioning a uniform point on a sphere on lying on a great circle gives different distributions depending on whether the circle is presented as an equator or a meridian — shows the almost-everywhere caveat is not pedantry: null-event conditioning depends on the limiting family, i.e. on the \( \sigma \)-algebra, not on the null event alone.

Common misconceptions. Bayes' theorem is not a "subjective" or "Bayesian-school" axiom — it is a theorem of pure probability, provable in three lines, and frequentists use it constantly for genuinely random hypotheses (as in every example below). It does not require the partition cells to be equally likely, nor \( B \) to be "caused by" the \( A_i \): no causal structure enters the proof, only the algebra of events. And a large likelihood ratio does not by itself make a hypothesis probable — the prior always has a vote.

Worked examples

Example 1 (diagnostic screening — the base-rate effect, exactly). A disease has prevalence \( 1\% \) in the screened population. A test has sensitivity \( \mathbb{P}(+ \mid D) = 0.99 \) and specificity \( \mathbb{P}(- \mid D^{c}) = 0.95 \). A randomly screened person tests positive. Find \( \mathbb{P}(D \mid +) \).

Example 2 (source attribution over a three-cell partition). A factory's output comes from machines \( M_1, M_2, M_3 \) producing \( 50\% \), \( 30\% \), \( 20\% \) of all items, with defect rates \( 1\% \), \( 2\% \), \( 4\% \) respectively. An item drawn uniformly at random from total output is defective. Which machine most likely produced it, and with what probability was it \( M_3 \)?

Problems

1. A box holds two coins: one fair, one double-headed. A coin is chosen uniformly at random and flipped once, landing heads. Find the probability it is the double-headed coin.
   Solution

   Partition on the coin: \( \mathbb{P}(\text{DH}) = \mathbb{P}(\text{fair}) = \tfrac12 \); likelihoods \( \mathbb{P}(H \mid \text{DH}) = 1 \), \( \mathbb{P}(H \mid \text{fair}) = \tfrac12 \). Total probability: \( \mathbb{P}(H) = 1 \cdot \tfrac12 + \tfrac12 \cdot \tfrac12 = \tfrac34 \). Bayes: \[ \mathbb{P}(\text{DH} \mid H) = \frac{1 \cdot \tfrac12}{\tfrac34} = \frac{2}{3}. \] Sanity check by outcome counting: of the four equally likely (coin, face) outcomes, three show heads — DH-head, DH-head (its two identical faces), fair-head — and two of those three come from the double-headed coin.
2. Urn I contains 3 red and 2 blue balls; urn II contains 1 red and 4 blue. An urn is chosen uniformly at random and one ball is drawn, which is red. Compute the posterior probability that urn I was chosen, and the posterior probability that a second ball drawn from the same urn (without replacement) is red.
   Solution

   Priors \( \tfrac12, \tfrac12 \); likelihoods \( \mathbb{P}(R \mid \mathrm{I}) = \tfrac35 \), \( \mathbb{P}(R \mid \mathrm{II}) = \tfrac15 \). Then \( \mathbb{P}(R) = \tfrac12 \cdot \tfrac35 + \tfrac12 \cdot \tfrac15 = \tfrac25 \), so \[ \mathbb{P}(\mathrm{I} \mid R) = \frac{\tfrac12 \cdot \tfrac35}{\tfrac25} = \frac{3}{4}, \qquad \mathbb{P}(\mathrm{II} \mid R) = \frac14. \] For the second draw, condition on the urn and the first draw: given urn I and one red removed, \( \mathbb{P}(R_2 \mid \mathrm{I}, R_1) = \tfrac24 = \tfrac12 \); given urn II, \( \mathbb{P}(R_2 \mid \mathrm{II}, R_1) = \tfrac04 = 0 \). By total probability with the posterior as the mixing law: \[ \mathbb{P}(R_2 \mid R_1) = \tfrac34 \cdot \tfrac12 + \tfrac14 \cdot 0 = \frac{3}{8}. \] Note the structural point: after conditioning on \( R_1 \), all further predictions must use the updated weights \( (\tfrac34, \tfrac14) \), not the priors.
3. Forty percent of incoming email is spam. The token "prize" occurs in \( 25\% \) of spam messages and \( 5\% \) of legitimate messages. (a) Find the probability a message containing "prize" is spam. (b) In odds form, identify the prior odds and likelihood ratio, and verify their product gives the same answer.
   Solution

   (a) \( \mathbb{P}(S) = 0.4 \), \( \mathbb{P}(W \mid S) = 0.25 \), \( \mathbb{P}(W \mid S^{c}) = 0.05 \). Total probability: \( \mathbb{P}(W) = 0.25 \cdot 0.4 + 0.05 \cdot 0.6 = 0.10 + 0.03 = 0.13 \). Bayes: \[ \mathbb{P}(S \mid W) = \frac{0.10}{0.13} = \frac{10}{13} \approx 0.769. \] (b) Prior odds \( \mathbb{P}(S)/\mathbb{P}(S^{c}) = 0.4/0.6 = \tfrac23 \); likelihood ratio \( 0.25/0.05 = 5 \). Posterior odds \( = \tfrac23 \cdot 5 = \tfrac{10}{3} \), and converting odds \( o \) to probability via \( o/(1+o) \): \[ \frac{10/3}{1 + 10/3} = \frac{10}{13}, \] in agreement with (a).
4. A disease has prevalence \( 1\% \). A test has sensitivity \( 0.95 \) and specificity \( 0.95 \), and repeated applications are conditionally independent given the true disease status. A patient tests positive twice. (a) Compute the posterior probability of disease. (b) Show that updating sequentially — using the posterior after the first positive as the prior for the second — gives the same answer. (c) Explain in one sentence why conditional independence given disease status, rather than unconditional independence, is the needed hypothesis.
   Solution

   (a) With \( \mathbb{P}(D) = 0.01 \) and \( \mathbb{P}(+ \mid D) = 0.95 \), \( \mathbb{P}(+ \mid D^{c}) = 0.05 \), conditional independence gives \( \mathbb{P}({+}{+} \mid D) = 0.95^2 = 0.9025 \) and \( \mathbb{P}({+}{+} \mid D^{c}) = 0.05^2 = 0.0025 \). Then \[ \mathbb{P}(D \mid {+}{+}) = \frac{0.9025 \cdot 0.01}{0.9025 \cdot 0.01 + 0.0025 \cdot 0.99} = \frac{0.009025}{0.009025 + 0.002475} = \frac{0.009025}{0.0115} = \frac{361}{460} \approx 0.785. \] (b) First update: \( \mathbb{P}(D \mid +) = \dfrac{0.95 \cdot 0.01}{0.95 \cdot 0.01 + 0.05 \cdot 0.99} = \dfrac{0.0095}{0.0590} = \dfrac{19}{118} \). Using \( p_1 = \tfrac{19}{118} \) as the new prior for the second positive: \[ \frac{0.95\,p_1}{0.95\,p_1 + 0.05(1 - p_1)} = \frac{0.95 \cdot 19}{0.95 \cdot 19 + 0.05 \cdot 99} = \frac{18.05}{18.05 + 4.95} = \frac{18.05}{23} = \frac{361}{460}, \] the same. (Cleanest in odds: prior odds \( \tfrac{1}{99} \) times likelihood ratio \( 19 \) twice gives \( \tfrac{361}{99} \); \( \tfrac{361}{361 + 99} = \tfrac{361}{460} \).) (c) The factorisation \( \mathbb{P}({+}{+} \mid D) = \mathbb{P}(+ \mid D)^2 \) used in (a) is exactly conditional independence given \( D \) (and given \( D^{c} \)); unconditional independence of the two test results is actually false here — the first positive raises the probability of the second, precisely because both are informative about \( D \).
5. (Theory.) Let \( A, B \) be events with \( \mathbb{P}(A) \gt 0 \) and \( \mathbb{P}(B) \gt 0 \). (a) Prove that \( \mathbb{P}(A \mid B) \gt \mathbb{P}(A) \) if and only if \( \mathbb{P}(B \mid A) \gt \mathbb{P}(B) \). (b) Prove that \( \mathbb{P}(A \mid B) = \mathbb{P}(B \mid A) \) if and only if \( \mathbb{P}(A) = \mathbb{P}(B) \). (c) Give events on a fair die with \( \mathbb{P}(A \mid B) = \mathbb{P}(B \mid A) \) but \( A \ne B \).
   Solution

   (a) By the definition of conditional probability, \( \mathbb{P}(A \mid B) \gt \mathbb{P}(A) \iff \mathbb{P}(A \cap B) \gt \mathbb{P}(A)\mathbb{P}(B) \) (multiply both sides by \( \mathbb{P}(B) \gt 0 \); direction of the inequality is preserved). By the identical manipulation with \( A \) and \( B \) exchanged, \( \mathbb{P}(B \mid A) \gt \mathbb{P}(B) \iff \mathbb{P}(A \cap B) \gt \mathbb{P}(A)\mathbb{P}(B) \). Both statements are equivalent to the same symmetric inequality, hence to each other. (b) By Bayes' theorem, \( \mathbb{P}(A \mid B) = \mathbb{P}(B \mid A)\,\dfrac{\mathbb{P}(A)}{\mathbb{P}(B)} \). If \( \mathbb{P}(A) = \mathbb{P}(B) \) the ratio is \( 1 \) and the conditionals agree. Conversely, suppose \( \mathbb{P}(A \mid B) = \mathbb{P}(B \mid A) \). If this common value is nonzero, cancel it: \( 1 = \mathbb{P}(A)/\mathbb{P}(B) \), so \( \mathbb{P}(A) = \mathbb{P}(B) \). If the common value is zero, both sides of the claimed equivalence can fail: \( \mathbb{P}(A \cap B) = 0 \) forces both conditionals to be \( 0 \) regardless of the marginals — so the "only if" direction needs the extra hypothesis \( \mathbb{P}(A \cap B) \gt 0 \), and the honest statement of (b) is: for \( \mathbb{P}(A \cap B) \gt 0 \), \( \mathbb{P}(A \mid B) = \mathbb{P}(B \mid A) \iff \mathbb{P}(A) = \mathbb{P}(B) \). (Noticing this necessity is the point of the problem.) (c) On a fair die take \( A = \{1, 2\} \), \( B = \{3, 4\} \): then \( \mathbb{P}(A \mid B) = \mathbb{P}(B \mid A) = 0 \) with \( A \ne B \) (here \( \mathbb{P}(A) = \mathbb{P}(B) \), consistent with (b)). For a nonzero example: \( A = \{1, 2\} \), \( B = \{2, 3\} \) gives \( \mathbb{P}(A \mid B) = \mathbb{P}(B \mid A) = \tfrac12 \) with \( A \ne B \).