The Central Limit Theorem

Derivation

The Central Limit Theorem

D-118 Home PU-106 Threads chance Depends on Characteristic Functions and Moments, Variance of Linear Combinations and Covariance

Statement

Let X₁, X₂, …, X_N be independent, identically distributed random variables with finite mean μ = E[X_i] and finite, nonzero variance σ² = Var(X_i). Form the sum S_N = ∑_i=1^N X_i and standardise it: Z_N = (S_N − Nμ)/(σ√N). Then as N → ∞, Z_N converges in distribution to a standard Gaussian: Z_N →^d 𝒩(0,1), meaning P(Z_N ≤ z) → Φ(z) = ∫_−∞^z (1/√(2π)) e^−u²/2 du for every z. The limiting shape is Gaussian regardless of the shape of the parent distribution, provided only that its variance is finite.

Why it matters

This is the reason the bell curve is everywhere. Any measurable quantity that is the accumulation of many small, independent contributions — the total charge from countless electrons, the pressure from myriad molecular impacts, the error left after many independent instrument imperfections — is driven toward a Gaussian by this theorem, and the individual contributions’ own distributions are washed out. It is why a single functional form, with just two parameters (mean and variance), describes the fluctuations of astonishingly disparate systems.

It also underwrites the entire practice of error analysis. The standard-error result fixes the width of a sample mean’s distribution; the central limit theorem fixes its shape, and only that shape lets us convert a standard error into a confidence interval (“68% within 1σ, 95% within 2σ”). Every χ² test, every Gaussian likelihood, every “nσ discovery” in physics inherits its legitimacy from here.

Assumptions

Each X_i has finite variance σ² < ∞ (hence a finite mean).If the second moment diverges (heavy tails: Cauchy, Pareto with index α ≤ 2), the standardising factor σ√N is undefined and the correct normalisation is N^1/α; the limit is then a non-Gaussian stable law, not 𝒩(0,1).

The X_i are independent.Independence is what lets the characteristic function of the sum factorise into a product (Step 4). With dependence the joint characteristic function does not factor, cross-correlations survive, and the limit can be non-Gaussian (e.g. fractional Brownian motion for long-range-correlated increments).

The variables are identically distributed with a common μ and σ².Dropped, the plain theorem no longer applies as stated; convergence to Gaussian still holds for non-identical summands only under an extra smallness condition (Lindeberg/Lyapunov) guaranteeing no single term dominates.

No single summand dominates the sum — the Lindeberg condition.Automatic in the i.i.d. finite-variance case (each term contributes a fraction σ²/(Nσ²) = 1/N → 0 of the variance). If one variable’s variance is comparable to the whole sum’s, its individual (non-Gaussian) shape leaks into the limit and Gaussianity fails.

Finiteness of the second moment guarantees the characteristic function is twice differentiable at the origin, with a valid second-order Taylor expansion (the prior result on characteristic functions and moments).This is exactly what licenses the expansion in Step 5. If E[X²] did not exist, φ would not be C² at 0 and the whole quadratic-order argument collapses.

Derivation

Y_i = (X_i − μ)/σ ⇒ E[Y_i] = 0, Var(Y_i) = E[Y_i²] = 1

Centre and scale each variable. Subtracting μ kills the mean; dividing by σ sets the variance to one. The Y_i are still i.i.d. and independent of each other. A

Z_N = (S_N − Nμ)/(σ√N) = (1/√N) ∑_i=1^N Y_i

Rewrite the standardised sum in terms of the Y_i: S_N − Nμ = ∑(X_i−μ) = σ∑Y_i, and the σ cancels. By the prior variance-of-linear-combinations result, Var(Z_N) = (1/N)∑Var(Y_i) = 1 for every N — the normalisation is chosen precisely so the spread stays fixed as N grows. A

φ_{Z_N}(t) ≡ E[e^itZ_N] = E[ exp( i (t/√N) ∑_i Y_i ) ]

Take the characteristic function of Z_N and substitute Step 2. Working with φ rather than the density is the whole trick: it turns the convolution of N distributions (a sum of variables) into an ordinary product. A

φ_{Z_N}(t) = ∏_i=1^N E[ e^i(t/√N)Y_i ] = [ φ_Y(t/√N) ]^N

Independence. The expectation of a product of functions of independent variables factorises into the product of expectations; identical distribution makes every factor the same, φ_Y(t/√N). This step is the sole place independence is used, and it is where the proof would fail for correlated data. B

φ_Y(s) = 1 + i s E[Y] − (s²/2) E[Y²] + o(s²) = 1 − s²/2 + o(s²)

Taylor-expand φ_Y about s = 0. By the prior result, a finite k-th moment makes φ k-times differentiable at 0 with φ^(k)(0) = i^kE[Y^k]; here E[Y]=0 and E[Y²]=1 from Step 1, so the linear term vanishes and the quadratic coefficient is −1/2. Only two moments are needed — this is why only the variance survives into the limit. B

φ_Y(t/√N) = 1 − t²/(2N) + o(t²/N)

Set s = t/√N. For fixed t, as N→∞ the argument s→0, so the expansion of Step 5 is valid term by term and the remainder is o(1/N). B

ln φ_{Z_N}(t) = N ln( 1 − t²/(2N) + o(1/N) ) = N[ −t²/(2N) + o(1/N) ] = −t²/2 + o(1)

Take logs of the N-th power (Step 4) and use ln(1+u) = u + O(u²) with u = −t²/(2N) + o(1/N) → 0. The factor N exactly cancels the 1/N, and the O(u²) = O(1/N²) corrections, multiplied by N, still vanish. This cancellation — a term of size 1/N summed N times giving an O(1) limit — is the analytic heart of the theorem. C

φ_{Z_N}(t) → e^−t²/2 (N → ∞), for every fixed t

Exponentiate Step 7. The pointwise limit of the characteristic functions is e^−t²/2. A

e^−t²/2 = φ_𝒩(0,1)(t) ⇒ Z_N →^d 𝒩(0,1)

e^−t²/2 is the characteristic function of the standard Gaussian (a known transform pair). By Lévy’s continuity theorem, pointwise convergence of characteristic functions to a limit that is continuous at t=0 implies convergence in distribution. Since the Gaussian’s φ is continuous everywhere, the conclusion follows. C

Result

Z_N = (S_N − Nμ)/(σ√N) ⟶^d 𝒩(0,1), φ_{Z_N}(t) → e^−t²/2

Reading. Add up many independent finite-variance contributions, subtract the mean, and rescale by σ√N; whatever the individual variables looked like, the result is a standard bell curve. All the parent distribution’s idiosyncratic features — skewness, kurtosis, discreteness — are relegated to higher-order corrections that die as powers of 1/√N. Only the first two moments, μ and σ², survive; everything else is forgotten.

Units check. S_N and Nμ both carry the units of X (say joules); their difference does too. The denominator σ√N carries the units of σ, also joules (√N is a dimensionless count). Hence Z_N is dimensionless, as it must be to be a standard normal deviate. The characteristic-function argument t is therefore dimensionless, so e^−t²/2 is well-formed.

Limiting cases

N = 1: Z₁ = (X−μ)/σ is just the standardised parent — no Gaussianity yet; the theorem is an N→∞ statement.
Parent already Gaussian: Z_N is exactly 𝒩(0,1) for every N — the Gaussian is the fixed point of the sum-and-rescale map (a stable law with index 2).
Symmetric parent: odd moments vanish, so the leading correction is the O(1/N) kurtosis term, not the O(1/√N) skewness term — convergence is noticeably faster.
σ → 0 (degenerate): the variables are constants; Z_N is undefined (0/0). The theorem needs a genuinely fluctuating parent.

Breaks when

Infinite variance / heavy tails. For a Cauchy (Lorentzian) parent, σ² does not exist and Step 5 has no quadratic term; instead φ_Y(s) = e^−|s|, giving [φ(t/N)]^N = e^−|t| — the sample mean of Cauchy variables is again Cauchy, never Gaussian. More generally a parent with tail index α < 2 needs normalisation N^1/α and converges to an α-stable law with power-law tails, not a bell curve.
Strong dependence. If the X_i are correlated with slowly decaying (long-range) correlations, Step 4’s factorisation fails, the variance of the sum grows faster than N, and the correct rescaling and limit change — e.g. the increments of fractional Brownian motion converge to a non-standard Gaussian process with the wrong exponent, or to non-Gaussian Hermite processes at criticality.
A dominant term (Lindeberg violated). If one summand carries a finite fraction of the total variance no matter how large N is (e.g. variances σ_i² ∝ 2ⁱ), that term never becomes negligible; its own non-Gaussian shape survives in the limit and the sum does not Gaussianise. Averaging many terms only works when none of them individually matters.
Finite N in the tails. Even with all hypotheses satisfied, the approximation is excellent near the centre but poor many standard deviations out: rare-event (large-deviation) probabilities are governed by the parent’s tail, not the Gaussian, and can be wrong by orders of magnitude for the same N that makes the bulk look perfectly normal.

Failure modes

Thinking the sum itself becomes Gaussian. S_N has mean Nμ and variance Nσ² that both grow with N — it does not converge to anything. Only the standardised variable Z_N has a limit. The theorem is about shape after centring and rescaling.
Applying it to a single measurement or tiny N. The CLT is asymptotic; for N=1 the distribution is just the parent. “Assume it’s Gaussian because CLT” with N=3 and a skewed parent is unjustified.
Ignoring the variance condition. Invoking the CLT for power-law / heavy-tailed data (financial returns, network degrees, city sizes) where σ² is effectively infinite — the sample “mean” then fails to stabilise and confidence intervals are meaningless.
Forgetting independence. Treating N correlated readings as if they were independent inflates the effective sample size and mis-applies the σ√N scaling; the sum can be far from Gaussian.
Trusting the tails. Using the Gaussian to estimate a 6σ or 8σ probability for moderate N — the central limit theorem controls the bulk, not the extreme tail, which converges much more slowly (Cramér / large-deviation regime).
Confusing the CLT with the law of large numbers. The LLN says the sample mean converges to μ (a statement about location); the CLT says the fluctuation about μ, magnified by √N, is Gaussian (a statement about shape). They answer different questions.

Discussion

The proof reveals precisely why the Gaussian is universal: the characteristic function turns summing variables into multiplying functions, and taking a logarithm turns that into adding functions. The theorem is then the statement that when you add N copies of a function that starts as 1 − (something)/2N, only the quadratic term survives the N→∞ limit — the linear term was removed by centring, and every higher term is suppressed by extra powers of 1/√N. The Gaussian is universal because it is the unique fixed point of “add two independent copies and rescale” among finite-variance laws; the sum-and-rescale operation is a flow on the space of distributions, and e^−t²/2 is where everything with finite variance ends up.

This fixed-point language is not a metaphor — it is the seed of the renormalisation group. In statistical physics the CLT is the trivial (Gaussian) fixed point of block-spin summation; a critical system is exactly one where the CLT breaks because correlations are long-ranged and fluctuations do not add in quadrature, giving non-Gaussian scaling exponents. The stable laws (Cauchy, Lévy) are the other fixed points, reached when variance is infinite. So the theorem is the boring, universal centre of a much richer landscape: it tells you what happens when nothing interesting (no criticality, no heavy tails) is going on, which is most of the time, which is why the bell curve is everywhere.

There is a precise hierarchy of corrections beyond the leading Gaussian, made explicit by the Edgeworth expansion. Keeping the cubic term in Step 5 (the skewness γ₁ = E[Y³]) gives lnφ_{Z_N}(t) = −t²/2 + (γ₁/6)(it)³/√N + …, so the leading correction to the density is O(1/√N) and proportional to the parent’s skewness; the next, kurtosis, correction is O(1/N). This is why symmetric parents converge faster (no 1/√N term) and why the Berry–Esseen theorem bounds the distance to Gaussian by Cρ/(σ³√N) with ρ=E|X−μ|³ — a uniform, non-asymptotic control on the rate that the bare theorem does not give.

Common misconceptions. (i) “The CLT says everything is Gaussian.” No — it says sums of many independent finite-variance pieces are, after standardising; a single skewed or heavy-tailed quantity need not be. (ii) “Larger N makes the data Gaussian.” The raw data keep their own distribution forever; it is the standardised sum that Gaussianises. (iii) “CLT and the law of large numbers are the same.” One is about where the mean goes, the other about the shape of its fluctuations. (iv) “It works for any distribution.” Finite variance is essential; Cauchy is the standard counterexample where it fails outright.

Worked examples

Example 1 — Sum of uniforms as a Gaussian generator. A classic quick-and-dirty way to make an approximately standard-normal random number is to add twelve independent U(0,1) variables and subtract 6. Justify this and estimate P(S₁₂ > 7), where S₁₂ = ∑_i=1¹² U_i.

μ = 1/2, σ² = 1/12 (for U(0,1))

Mean and variance of a single uniform on [0,1]. Keep symbols. A

E[S₁₂] = 12μ = 6, Var(S₁₂) = 12σ² = 12×(1/12) = 1

Sum of 12 independent copies; means and variances add. The variance conveniently equals 1, so S₁₂−6 already has unit variance — no further rescaling needed. A

Z = (S₁₂ − 6)/√1 = S₁₂ − 6 ≈ 𝒩(0,1)

By the CLT with N=12, the standardised sum is approximately standard normal (the uniform is bounded and symmetric, so N=12 is already very good). B

P(S₁₂ > 7) = P(Z > 1) = 1 − Φ(1)

Translate the event on S₁₂ into one on the standard normal Z: S₁₂=7 ⇒ Z=1. A

= 1 − 0.8413 = 0.1587

Standard normal table value Φ(1)=0.8413. A

P(S₁₂ > 7) ≈ 0.159 (≈ 16%)

Reading. Twelve flat distributions, each utterly un-bell-shaped, sum to something almost indistinguishable from a Gaussian — the CLT in action at modest N. The −6 centres it and the variance happens to be exactly 1, which is why this recipe was popular before better generators existed. (Its flaw: the sum is bounded to [0,12], so it cannot produce deviates beyond ±6σ — a tail failure exactly as the “Breaks when” box warns.)

Example 2 — Coin flips and the normal approximation to the binomial (de Moivre–Laplace). A fair coin is flipped N = 100 times. Let H be the number of heads. Estimate P(H ≥ 60) using the CLT.

H = ∑_i=1¹⁰⁰ X_i, X_i ∈ {0,1}, p = 1/2

The count of heads is a sum of 100 independent Bernoulli indicators — exactly the CLT setup. A

μ = Np = 50, σ² = Npq = 100×(1/2)×(1/2) = 25, σ = 5

Bernoulli mean p and variance pq; sum over N=100. A

Z = (H − μ)/σ = (H − 50)/5 ≈ 𝒩(0,1)

Standardise. By the CLT the standardised head-count is approximately standard normal. B

P(H ≥ 60) ≈ P(Z ≥ (59.5 − 50)/5) = P(Z ≥ 1.9)

Apply the continuity correction: a discrete H≥60 maps to the continuous H>59.5, improving the approximation. (59.5−50)/5 = 1.9. B

= 1 − Φ(1.9) = 1 − 0.9713 = 0.0287

Standard normal value Φ(1.9)=0.9713. A

P(H ≥ 60) ≈ 0.029 (≈ 2.9%)

Reading. The exact binomial answer is 0.0284, so the CLT (with continuity correction) is right to better than 1% relative — excellent for N=100 and symmetric p=1/2. Getting 60 or more heads is a modest 1.9σ upward fluctuation. Without the continuity correction one would use Z=2.0 and get 0.0228, noticeably worse — the correction matters at these N.

Problems

A machine fills bags with a mean mass μ = 500 g and standard deviation σ = 20 g per bag, bags independent. A pallet holds N = 100 bags. Using the CLT, estimate the probability that the total pallet mass exceeds 50.4 kg.
Solution
Total S₁₀₀ has mean Nμ = 50000 g and variance Nσ² = 100×400 = 40000 g², so SD = √40000 = 200 g. Standardise: Z = (50400 − 50000)/200 = 2.0. So P(S>50400) = P(Z>2) = 1 − Φ(2) = 1 − 0.9772 = 0.0228 (about 2.3%). Note the SD of the total grows only as √N (200 g) while the mean grows as N (50 kg) — the sum is relatively more predictable for larger N.
Explain, using characteristic functions, why the sum of two independent Cauchy variables (each with φ(t)=e^−|t|) rescaled by 1/2 is again Cauchy, and why this makes the CLT fail. Which step of the derivation is invalid for the Cauchy?
Solution
For independent X₁,X₂, φ_X₁+X₂(t) = φ(t)² = e^−2|t|. The sample mean is M = (X₁+X₂)/2, whose characteristic function is φ_X₁+X₂(t/2) = e^−2|t/2| = e^−|t| — identical to a single Cauchy. So averaging does not narrow the distribution at all; the “standardised” mean stays Cauchy for every N and never approaches a Gaussian. The failure is at Step 5: the Taylor expansion φ(s)=1−s²/2+… requires a finite second moment; the Cauchy has E[X²]=∞, and indeed e^−|s| = 1 − |s| + … is not differentiable at s=0 (a |s|, not s², cusp), so there is no quadratic-order term to inherit.
The lifetime of a component is exponentially distributed with mean τ = 5 hours (so μ=5, σ=5 h). A system replaces components one after another. Estimate the probability that the total time to get through N = 50 components exceeds 300 hours.
Solution
For an exponential, μ=τ=5 h and σ=τ=5 h. Total time T = ∑₁⁵⁰ has mean Nμ = 250 h and variance Nσ² = 50×25 = 1250 h², so SD = √1250 ≈ 35.36 h. Standardise: Z = (300 − 250)/35.36 = 50/35.36 ≈ 1.414. So P(T>300) ≈ P(Z>1.41) = 1 − Φ(1.41) ≈ 1 − 0.9207 = 0.079 (about 8%). (The exact answer via the Gamma/Erlang distribution is ≈ 0.083; the exponential is right-skewed, so at N=50 a small O(1/√N) skewness error remains.)
A parent distribution has skewness γ₁ = 2. Roughly how large must N be for the leading Edgeworth skewness correction, of order γ₁/√N, to fall below 0.1? What does this say about the rate of convergence compared with a symmetric parent?
Solution
Require γ₁/√N < 0.1, i.e. √N > γ₁/0.1 = 2/0.1 = 20, so N > 400. You need a few hundred samples for a moderately skewed parent, versus far fewer for a symmetric one where the 1/√N term vanishes identically and the leading correction is the smaller O(1/N) kurtosis term. Skewness is the slowest-dying defect, which is why asymmetric parents (exponential, Poisson at small mean) approach Gaussian noticeably more slowly than symmetric ones (uniform, binomial at p=1/2).
Show that if X is itself Gaussian, then Z_N is exactly 𝒩(0,1) for every N, using characteristic functions. What special property of the Gaussian does this express?
Solution
A single standardised Gaussian summand has φ_Y(s) = e^−s²/2 exactly (no higher-order remainder). Then from Step 4, φ_{Z_N}(t) = [φ_Y(t/√N)]^N = [e^{−(t/√N)²/2}]^N = [e^−t²/(2N)]^N = e^−t²/2 for every N, including N=1. So Z_N is precisely standard normal, not merely asymptotically. This expresses that the Gaussian is a stable law (index 2): a normalised sum of independent Gaussians is Gaussian. It is the attracting fixed point of the sum-and-rescale flow, which is exactly why all finite-variance distributions converge to it rather than to something else.