The Chi-Squared Distribution

Derivation

The Chi-Squared Distribution

D-123 Home PU-106 Threads chance Depends on The Central Limit Theorem

Statement

Let Z₁, …, Z_k be independent standard-normal variables, Z_i ~ N(0, 1), and define Q = Σ_i=1^k Z_i². Then Q follows the chi-squared distribution with k degrees of freedom, with probability density f_Q(x) = x^{k/2 − 1} e^−x/2 ⁄ [2^k/2 Γ(k/2)] for x > 0, mean E[Q] = k, and variance Var(Q) = 2k.

Why it matters

The chi-squared distribution is the exact sampling distribution of a sum of squared Gaussian fluctuations. It governs the estimated variance of any normally distributed measurement, underlies every least-squares goodness-of-fit test, and is the reference distribution against which experimental χ² statistics are compared to accept or reject a model.

It also appears directly in physics: the kinetic energy of a classical particle in a heat bath is a sum of squared Gaussian velocity components, so the chi-squared distribution with k = 3 is the equilibrium energy distribution, and its mean k is exactly the equipartition theorem in disguise.

Assumptions

Each Z_i is standard normal (zero mean, unit variance).A non-zero mean gives the noncentral chi-squared distribution; a variance σ² ≠ 1 rescales Q by σ² and shifts the density.

The Z_i are mutually independent.Correlated components make the moment-generating function fail to factor, so the sum is no longer chi-squared; one must diagonalise the covariance first.

k is a fixed integer, not itself estimated from the data.If the k squared terms are residuals about parameters fitted from the same sample, each fitted parameter removes one degree of freedom, and the effective k drops below the naive count (Cochran's theorem).

The moment-generating function exists in a neighbourhood of t = 0.Without a strip of convergence the uniqueness theorem for MGFs cannot be invoked, and one must instead invert the characteristic function to identify the density.

Derivation

Y = Z², F_Y(y) = P(−√y ≤ Z ≤ √y) = 2Φ(√y) − 1

Start with a single component. The event Z² ≤ y is the symmetric interval, and Φ is the standard-normal CDF. A

f_Y(y) = dF_Y/dy = 2φ(√y) · 1⁄(2√y) = φ(√y) ⁄ √y

Differentiate the CDF; the chain rule supplies the Jacobian factor 1⁄(2√y) with φ = Φ′ the standard-normal density. A

f_Y(y) = (2π)^−1/2 y^−1/2 e^−y/2, y > 0

Insert φ(u) = (2π)^−1/2e^−u²/2 with u = √y. This is already the k = 1 chi-squared density. A

M₁(t) = E[e^tY] = (2π)^−1/2 ∫₀^∞ y^−1/2 e^{−y(1/2 − t)} dy

Form the moment-generating function of one squared normal. The integral converges only for t < 1/2, defining the strip of existence. B

∫₀^∞ y^−1/2 e^−ay dy = Γ(1/2) a^−1/2 = √π a^−1/2, a = 1/2 − t

Evaluate with the Gamma integral ∫₀^∞y^s−1e^−aydy = Γ(s)a^−s at s = 1/2, using Γ(1/2) = √π. B

M₁(t) = (2π)^−1/2 √π (1/2 − t)^−1/2 = (1 − 2t)^−1/2

Collect the constants: (2π)^−1/2√π = 2^−1/2, and (1/2 − t)^−1/2 = 2^1/2(1 − 2t)^−1/2. A

M_Q(t) = ∏_i=1^k M₁(t) = (1 − 2t)^−k/2

The MGF of a sum of independent variables is the product of their MGFs; independence is exactly what licenses this factorisation. C

(1 − 2t)^−k/2 = (1 − θt)^−α with α = k/2, θ = 2 ⇒ Q ~ Gamma(α, θ)

Match against the Gamma MGF (1 − θt)^−α. Because the MGF exists on an open interval it determines the law uniquely, identifying the density. C

f_Q(x) = x^α−1 e^−x/θ ⁄ [Γ(α) θ^α] = x^{k/2 − 1} e^−x/2 ⁄ [2^k/2 Γ(k/2)]

Substitute α = k/2, θ = 2 into the standard Gamma density. This is the chi-squared density with k degrees of freedom. A

K(t) = ln M_Q(t) = −(k/2) ln(1 − 2t)

Take the cumulant-generating function so that derivatives at t = 0 give the mean and variance directly. B

K′(t) = k ⁄ (1 − 2t), E[Q] = K′(0) = k

The first cumulant is the mean. (Consistency check: E[Z²] = 1, summed k times.) A

K″(t) = 2k ⁄ (1 − 2t)², Var(Q) = K″(0) = 2k

The second cumulant is the variance. (Check: Var(Z²) = E[Z⁴] − 1 = 3 − 1 = 2, summed k times.) A

Result

f_Q(x) = x^{k/2 − 1} e^−x/2 ⁄ [2^k/2 Γ(k/2)], x > 0, E[Q] = k, Var(Q) = 2k

Reading. A sum of k independent squared standard-normals is right-skewed, supported on the positive axis, and peaks near x = k − 2. Its centre of mass sits at k and its spread grows as √(2k); each extra degree of freedom adds one unit of expected value and two units of variance. The x^k/2−1 factor controls the behaviour near the origin and the e^−x/2 factor the exponential tail.

Units check. Each Z_i is dimensionless (a value divided by its own standard deviation), so Q and x are dimensionless. A density in x must carry dimension 1⁄x, and indeed f_Q integrates to a pure number: ∫₀^∞ x^k/2−1e^−x/2dx = 2^k/2Γ(k/2), which cancels the normalisation exactly.

Limiting cases

k = 1: f(x) = (2πx)^−1/2e^−x/2, which diverges as x^−1/2 at the origin (integrably) — the density of a single squared Gaussian.
k = 2: f(x) = ½e^−x/2, an exponential with mean 2; equivalently the distribution of twice a Rayleigh-squared radius.
k = 3: f(x) = (x/2π)^1/2e^−x/2 — the Maxwell–Boltzmann energy distribution with mean (3/2), the equipartition value.
k → ∞: by the central limit theorem the standardised variable (Q − k)⁄√(2k) → N(0, 1); the skewness √(8/k) vanishes and the shape becomes symmetric.

Breaks when

Correlated components. If the Z_i are dependent, M_Q no longer factors as a product; the quadratic form ∑Z_i² then follows a weighted sum of independent chi-squares (eigenvalues of the covariance), not a single chi-squared.
Non-zero means. If Z_i ~ N(μ_i, 1) with μ_i ≠ 0, the result is the noncentral chi-squared with noncentrality λ = ∑μ_i²; the mean becomes k + λ and variance 2k + 4λ.
Estimated parameters. When the squared terms are residuals about p parameters fitted from the same data, the true degrees of freedom are k − p, not k; using the naive count biases every goodness-of-fit inference.
Non-Gaussian heavy tails. If the underlying variables are not normal (e.g. Student-t with few degrees of freedom), the sum of squares has heavier tails than chi-squared and the density above is only an approximation that worsens in the tail.

Failure modes

Confusing k with sample size. Treating the degrees of freedom as the number of data points and forgetting the −1 (or −p) reduction when the mean or other parameters are estimated.
Assuming symmetry. Reading off a two-sided confidence interval as if chi-squared were symmetric about its mean; the density is right-skewed and the upper and lower tails differ.
Dropping the Jacobian. In the k = 1 transformation, writing f_Y(y) = φ(√y) and forgetting the 1⁄(2√y) factor, which loses the crucial y^−1/2 singularity.
Using variance = k. Copying the mean as the variance; the correct variance is 2k because Var(Z²) = 2, not 1.
Squaring correlated residuals. Applying the result to fit residuals that share a fitted baseline and are therefore not independent, inflating the apparent number of degrees of freedom.

Discussion

The chi-squared law is the special case Gamma(k/2, 2) of the Gamma family, and this identification is more than bookkeeping: it means that a sum of two independent chi-squares with k₁ and k₂ degrees of freedom is again chi-squared with k₁ + k₂, exactly the additivity we exploited through the MGF product in step 7. Degrees of freedom add like independent squared coordinates — a purely geometric statement about orthogonal directions in a Gaussian cloud.

That geometric picture is the deepest reading. A k-dimensional isotropic Gaussian has spherically symmetric density ∝ e^−r²/2, so the squared radius r² = ∑Z_i² carries all the distributional content; chi-squared is simply the law of that squared radius. The x^k/2−1 prefactor is the surface area of a shell of radius √x, and the exponential is the Gaussian weight on that shell. This is why the mode sits at k − 2 rather than the origin even though the Gaussian peaks at the centre: there is far more volume at large radius.

The result seeds a whole family. Ratios of independent chi-squares (each over its degrees of freedom) define the F distribution used in analysis of variance; a standard normal divided by √(Q/k) is Student's t; and the sample variance obeys (n − 1)s²/σ² ~ χ²_n−1, which is the engine of every confidence interval for a variance.

The clean subtraction of degrees of freedom when parameters are fitted is the content of Cochran's theorem: if a Gaussian vector is decomposed by orthogonal projections onto subspaces of dimensions d₁, d₂, …, the squared norms in each subspace are independent chi-squares with those dimensions as their degrees of freedom. Fitting p parameters projects out a p-dimensional subspace, leaving k − p in the residual — the abstract reason the naive count is wrong, and the same idea that partitions total variance into orthogonal, independently chi-squared pieces in ANOVA.

Common misconceptions. Chi-squared is not symmetric and its mean is not its mode; a small observed χ² is not automatically "good," since a value far below the mean can signal overfitting or overestimated errors just as a large value signals a poor model.

Worked examples

Sample variance of a normal measurement. n = 10 readings from N(μ, σ²) with σ² = 4; find the mean and standard deviation of the sample variance s².

(n − 1)s² ⁄ σ² ~ χ²_n−1, df = n − 1 = 9

One degree of freedom is lost to the estimated mean, so the standardised sum of squared residuals is chi-squared with 9, not 10. B

E[s²] = σ² ⁄ (n−1) · E[χ²₉] = σ²⁄9 · 9 = σ² = 4

Use E[χ²₉] = 9; the estimator is unbiased. A

Var(s²) = (σ²⁄9)² · Var(χ²₉) = (σ⁴⁄81)(2·9) = 2σ⁴⁄9 = 2·16⁄9 = 3.56

Use Var(χ²₉) = 2·9 = 18 and propagate the constant. A

E[s²] = 4, SD(s²) = √3.56 = 1.89

Reading. The variance estimate is centred on the true value 4 but scatters by nearly ±50% with only ten readings — a quantitative warning about trusting variances from small samples.

Units check. s² has units of σ² (the measurement squared); its standard deviation likewise, as required.

Equipartition from three squared velocities. A gas molecule of mass m at temperature T has velocity components v_x, v_y, v_z ~ N(0, k_BT/m). Find the mean and variance of its kinetic energy E = ½m(v_x² + v_y² + v_z²).

Z_i = v_i ⁄ √(k_BT/m) ~ N(0, 1), Q = Z_x² + Z_y² + Z_z² ~ χ²₃

Standardise each independent component; three of them give three degrees of freedom. B

E = ½m · (k_BT/m) Q = ½k_BT Q

Factor the common scale out of the sum; the mass cancels, leaving energy proportional to a χ²₃ variable. A

E[E] = ½k_BT · E[Q] = ½k_BT · 3 = (3/2)k_BT

Insert E[χ²₃] = 3; this is exactly the equipartition theorem, ½k_BT per quadratic degree of freedom. A

Var(E) = (½k_BT)² · Var(Q) = ¼(k_BT)² · (2·3) = (3/2)(k_BT)²

Use Var(χ²₃) = 6 and propagate the constant. A

E[E] = (3/2)k_BT, Var(E) = (3/2)(k_BT)²

Reading. The chi-squared mean reproduces equipartition exactly, and its variance quantifies the thermal energy fluctuation of a single molecule, of order k_BT itself — relative fluctuation √(2/3) ≈ 0.82, large because one molecule is far from the thermodynamic limit.

Units check. Q is dimensionless, k_BT is an energy, so E[E] is an energy and Var(E) an energy squared, as required.

Problems

Show that the χ² distribution with 2 degrees of freedom is an exponential, and compute P(Q > 3).
Solution
Set k = 2 in the density: f(x) = x⁰e^−x/2⁄[2¹Γ(1)] = ½e^−x/2, an exponential with mean 2. Then P(Q > 3) = ∫₃^∞½e^−x/2dx = e^−3/2 = 0.223.
State the mean, variance, and standard deviation of χ²₅.
Solution
Mean = k = 5; variance = 2k = 10; standard deviation = √10 = 3.16.
A sample of n = 25 readings comes from N(μ, σ²) with σ² = 9. Give the degrees of freedom of (n−1)s²/σ² and the standard deviation of s².
Solution
df = n − 1 = 24. Var(s²) = (σ²/24)²·Var(χ²₂₄) = (σ⁴/576)(48) = σ⁴/12 = 81/12 = 6.75, so SD(s²) = √6.75 = 2.60.
Find the mode of the χ² density with k degrees of freedom (k ≥ 2), and evaluate the density at the mode for k = 6.
Solution
Maximise ln f = (k/2 − 1)ln x − x/2; setting the derivative (k/2 − 1)/x − ½ = 0 gives mode x = k − 2. For k = 6 the mode is at x = 4; f(4) = 4²e⁻²⁄[2³Γ(3)] = 16·0.1353⁄(8·2) = 2.165/16 = 0.135.
Using the central-limit approximation, estimate P(χ²₅₀ > 70) and comment on its accuracy.
Solution
For k = 50, mean = 50 and SD = √(2·50) = 10. Approximating Q ~ N(50, 100): P(Q > 70) ≈ P(Z > (70−50)/10) = P(Z > 2) = 0.023. The true value is about 0.021, so the plain normal approximation is close but slightly overestimates the right tail because the distribution is still right-skewed at k = 50; a Wilson–Hilferty cube-root transform sharpens the estimate.