The Central Limit Theorem

The Intuitive Picture

Imagine you are trying to estimate the height of a mountain by throwing a dart at a map. If you throw the dart once, the error is random and unpredictable. However, if you throw the dart 100 times and take the average of the 100 positions, the error of that average is much smaller and behaves in a very specific, predictable way. The CLT states that if you sum up a large number of independent, random influences—each with a finite average and spread—the resulting sum will look like a bell curve. It does not matter if the individual influences are biased, skewed, or irregular; as long as they are independent and have finite variance, their sum converges to a Gaussian shape. This is why the "bell curve" is the default model for measurement uncertainty in physics.

Formal Statement and Standardisation

Let \( X_1, X_2, \ldots, X_N \) be independent and identically distributed (i.i.d.) random variables. Each variable has a finite mean \( \mu = E[X_i] \) and a finite, non-zero variance \( \sigma^2 = \mathrm{Var}(X_i) \). We define the sum of these variables as \( S_N = \sum_{i=1}^{N} X_i \). The raw sum \( S_N \) will have a mean of \( N\mu \) and a variance of \( N\sigma^2 \). As \( N \) increases, the sum grows larger and spreads out more, so it does not converge to a single fixed distribution. To see the limiting shape, we must standardise the sum by subtracting the mean and dividing by the standard deviation of the sum.

The Central Limit Theorem states that as \( N \to \infty \), the random variable \( Z_N \) converges in distribution to a standard Gaussian distribution \( \mathscr{N}(0,1) \). This means that for any real number \( z \), the cumulative distribution function of \( Z_N \) approaches the standard normal cumulative distribution function \( \Phi(z) \):

This result is profound because it implies that the probability density function of \( Z_N \) approaches the standard Gaussian density \( \frac{1}{\sqrt{2\pi}} e^{-z^2/2} \).

Proof via Characteristic Functions

The most elegant proof of the CLT relies on the properties of characteristic functions derived in the previous lecture. Recall that for independent random variables, the characteristic function of their sum is the product of their individual characteristic functions. Let \( \phi_X(t) \) be the characteristic function of a single variable \( X_i \). The characteristic function of the sum \( S_N \) is:

We are interested in the standardised variable \( Z_N \). Using the scaling property of characteristic functions, \( \phi_{aX+b}(t) = e^{ibt}\phi_X(at) \), we can write the characteristic function of \( Z_N \) as:

Because the variance is finite, the characteristic function is twice differentiable at the origin, so it can be expanded for small argument. The cleanest route removes the mean first: write \( Y_i = (X_i - \mu)/\sigma \), so that \( E[Y_i] = 0 \), \( \mathrm{Var}(Y_i) = 1 \) and \( Z_N = \frac{1}{\sqrt{N}}\sum_{i=1}^{N} Y_i \). (This is exactly what the phase factor \( e^{-it\sqrt{N}\mu/\sigma} \) above does: it cancels the mean.) Then \( \phi_Y(0) = 1 \), \( \phi_Y'(0) = iE[Y] = 0 \) and \( \phi_Y''(0) = -E[Y^2] = -1 \), so the expansion is:

Because the \( Y_i \) are independent, the characteristic function of \( Z_N \) is a product of \( N \) identical factors evaluated at \( h = t/\sqrt{N} \):

Using the standard limit \( \lim_{N\to\infty} (1 + \frac{a}{N})^N = e^a \) with \( a = -t^2/2 \):

This is the characteristic function of the standard Gaussian \( \mathscr{N}(0,1) \). By Lévy's continuity theorem, convergence of the characteristic functions implies that \( Z_N \) converges in distribution to \( \mathscr{N}(0,1) \).

Implications for Measurement Error

In experimental physics, a measured value \( Y \) is often the sum of a true signal and a noise term \( \epsilon \). If the noise \( \epsilon \) arises from \( N \) independent sources (e.g., thermal noise in \( N \) resistors, or photon counting statistics), the CLT tells us that \( \epsilon \) will be approximately Gaussian for large \( N \). This justifies the use of the standard deviation \( \sigma \) as a measure of uncertainty. If we take the average of \( N \) independent measurements \( \bar{X} = \frac{1}{N}\sum X_i \), the variance of the mean is \( \mathrm{Var}(\bar{X}) = \frac{\sigma^2}{N} \). Thus, the standard error of the mean decreases as \( \frac{1}{\sqrt{N}} \). This is the fundamental basis for improving precision by repeating measurements.