Goodness of Fit and Degrees of Freedom

Derivation

Goodness of Fit and Degrees of Freedom

D-124 Home PU-106 Threads chance Depends on The Chi-Squared Distribution, Linear Least Squares and the Normal Equations

Statement

For N data points with independent Gaussian errors, fitted to a model with M adjustable parameters by minimising χ², the minimised statistic obeys E[χ²_min] = N − M and, for a linear model, is distributed as χ² with ν = N − M degrees of freedom. Each fitted parameter therefore removes exactly one degree of freedom, and the reduced statistic χ²_ν = χ²_min / (N − M) has expectation one, which is the working goodness-of-fit test.

Why it matters

Almost every quantitative measurement in physics ends with a fit: extracting a lifetime, a coupling constant, a slope, a resonance mass. The reduced chi-squared is the single number that tells you whether the model and the error bars are mutually consistent. Without the N − M correction you would systematically expect too large a χ² and reject perfectly good fits.

The result also encodes a deep bookkeeping principle: information "spent" constraining parameters is no longer available to test the model. Overfitting, the bias in the sample variance, the Bessel correction, and the penalty terms in AIC/BIC are all the same accounting seen from different angles.

Assumptions

Errors are Gaussian.If dropped, χ²_min is no longer chi-squared distributed; heavy tails inflate it and the reduced-χ² threshold miscalibrates.

Errors are independent (diagonal covariance).If dropped, the sum-of-squares form is wrong; one must use the full inverse-covariance quadratic form, and the trace argument changes.

The error bars σ_i are known and correct.If dropped, χ²_ν rescales by the true/assumed variance ratio and can no longer separate a bad model from mis-stated uncertainties.

The model is linear in its parameters (or the fit sits in the locally-linear regime).If dropped, the projection matrix is only an approximation; E[χ²_min] = N − M holds only asymptotically (Wilks' theorem), not exactly.

The design has full column rank M (parameters non-degenerate).If dropped, the effective number of independent parameters is less than M and the true degrees of freedom exceed N − M.

Derivation

χ²(a) = Σ_i=1^N ( y_i − f(x_i;a) )² / σ_i²

Definition of the chi-squared statistic for data y_i with independent Gaussian errors σ_i; it is minus twice the log-likelihood up to a constant. A

z_i = y_i/σ_i, n_i = ε_i/σ_i ~ N(0,1), A_ik = φ_k(x_i)/σ_i

Whiten the problem: dividing each row by its own σ_i turns the noise into a standard normal vector and removes all scale from the algebra. For a linear model f = Σ_k a_k φ_k(x). A

χ²(a) = | z − Aa |², z = Aa_true + n, n ~ N(0, I_N)

Rewrite the sum as a squared Euclidean norm of the whitened residual vector; the true data lie on the model subspace plus standard-normal noise. B

∂χ²/∂a = 0 ⇒ Aᵀ(z − Aâ) = 0 ⇒ AᵀA â = Aᵀz

Stationarity of the quadratic form gives the normal equations (prior result: linear-least-squares-normal-equations). B

â = (AᵀA)⁻¹ Aᵀz

Invert the M×M Gram matrix, legal because the design has full column rank M. B

ẑ = Aâ = Hz, H ≡ A(AᵀA)⁻¹Aᵀ

The fitted values are the orthogonal projection of z onto the M-dimensional column space of A; H is the "hat" matrix, symmetric and idempotent (H² = H). B

r̂ = z − ẑ = (I − H)z = (I − H)n

Because HA = A we have (I − H)A = 0, so the deterministic model part Aa_true is annihilated and the residual depends only on the noise. This is the key step that makes the answer independent of the true parameters. C

χ²_min = | r̂ |² = nᵀ(I − H)ᵀ(I − H)n = nᵀ(I − H)n

(I − H) is symmetric and idempotent, so (I − H)ᵀ(I − H) = (I − H); the minimised statistic is a single quadratic form in the noise. B

E[χ²_min] = E[nᵀ(I − H)n] = tr( (I − H) · Cov[n] ) = tr(I − H)

Expectation of a quadratic form: E[nᵀBn] = tr(B Σ) + μᵀBμ, with μ = 0 and Σ = I_N. C

tr(I − H) = tr(I_N) − tr(H) = N − tr( (AᵀA)⁻¹AᵀA ) = N − tr(I_M) = N − M

Cyclic invariance of the trace collapses the hat matrix to the M×M identity; each of the M fitted parameters contributes exactly one unit to tr(H), i.e. removes one degree of freedom. B

(I − H) = Σ_j u_j², eigenvalues {1 (×(N−M)), 0 (×M)} ⇒ χ²_min ~ χ²_N−M

An orthogonal projection of rank N − M has eigenvalues 0 and 1. Rotating n into its eigenbasis (a rotation preserves the standard-normal law) leaves a sum of N − M independent squared standard normals — which is chi-squared with N − M dof (prior result: chi-squared-distribution-from-gaussians). Hence Var[χ²_min] = 2(N − M). C

ν ≡ N − M, χ²_ν ≡ χ²_min / ν ⇒ E[χ²_ν] = 1, Var[χ²_ν] = 2/ν

Divide by the degrees of freedom to define the reduced chi-squared, normalising the expectation to unity so that a "good fit" reads χ²_ν ≈ 1. A

Result

E[χ²_min] = N − M, χ²_min ~ χ²_N−M, χ²_ν = χ²_min/(N − M), E[χ²_ν] = 1

Reading. A well-fitting model over N points with M free parameters should return χ²_min near N − M, i.e. χ²_ν ≈ 1 ± √(2/ν). Substantially larger means the model is wrong or the errors are underestimated; substantially smaller means the errors are overestimated or the model is overfitting. Each parameter you fit costs exactly one degree of freedom because fitting projects the residual out of one more dimension.

Units check. Every residual is divided by its own σ_i, so each term (y_i − f)²/σ_i² is dimensionless; χ², N, M, and ν are all pure numbers, and χ²_ν is a dimensionless ratio of order unity.

Limiting cases

No parameters (M = 0): comparing data to a fully-specified model, χ²_min ~ χ²_N, the classic Pearson goodness-of-fit test.
One parameter (M = 1), constant model: fitting the weighted mean gives ν = N − 1 — this is the origin of the Bessel (N − 1) correction to the sample variance.
Saturated fit (M = N): ν = 0, χ²_min = 0 exactly; the model passes through every point and χ²_ν is undefined — a fit with no predictive test.
Large ν: χ²_ν → 1 with spread √(2/ν) → 0; by the CLT the distribution becomes Gaussian and even small deviations from 1 become statistically significant.

Breaks when

Correlated errors: a non-diagonal covariance means the plain sum of squares is not the correct statistic; the residual "counts" fewer independent numbers and the naive N − M overstates the degrees of freedom. Use χ² = rᵀC⁻¹r with the full covariance C.
Strongly nonlinear model: the hat matrix is only a local linearisation, so χ²_min is chi-squared only asymptotically; near boundaries or with few data E[χ²_min] ≠ N − M and confidence regions are non-elliptical.
Non-Gaussian / outlier-laden errors: heavy tails make χ²_min systematically large and the chi-squared reference distribution invalid; a single 5σ outlier can dominate the sum.
Wrong error scale: if all σ_i are off by a common factor k, then χ²_ν scales by 1/k², so a value far from 1 cannot distinguish a wrong model from mis-estimated uncertainties.
Parameters at a boundary or degenerate: if a parameter hits a physical limit or two parameters are near-degenerate, the effective count of fitted parameters drops below M and the dof accounting fails (Chernoff / boundary corrections needed).

Failure modes

Using N instead of N − M: dividing by the number of points rather than the degrees of freedom, systematically biasing χ²_ν low and passing bad fits.
Forgetting the 1/σ_i² weights: summing raw squared residuals gives a dimensional quantity with no distributional meaning.
Reading χ²_ν > 1 as automatically bad: ignoring the spread √(2/ν); for small ν a value of 1.5 is entirely ordinary.
Celebrating χ²_ν ≈ 0.2: a very small reduced chi-squared is a warning of overestimated error bars or too many free parameters, not a triumph.
Counting fixed or constrained parameters as free: a parameter held at a value, or tied to another, does not cost a full degree of freedom.
Over-parameterising then still quoting ν = N − 1: adding nuisance parameters silently lowers χ²_min but must also lower ν.

Discussion

The whole result is a statement about geometry. The N-dimensional whitened data vector is split by the fit into a component inside the M-dimensional model subspace (the fitted values) and a component in its orthogonal complement (the residual). Only the orthogonal complement, of dimension N − M, is free to fluctuate against the noise, and its squared length is what χ²_min measures. Fitting a parameter is literally the act of projecting out one more direction, which is why the count is exact and integer.

This is the same bookkeeping that appears throughout statistics: the Bessel (N − 1) in the sample variance is the M = 1 case (one parameter, the mean, is estimated from the data); the residual degrees of freedom in ANOVA and ordinary regression are N − M; and the information criteria AIC and BIC add explicit 2M or M ln N penalties precisely because extra parameters buy a lower χ² that is not real predictive power. The reduced chi-squared is the physicist's minimal version of this penalty.

Practically, the test is two-sided. A large χ²_ν flags an inadequate model, systematic error, or underestimated uncertainties; a small one flags overestimated uncertainties or an over-flexible model. Because it cannot by itself separate "wrong model" from "wrong error bars", it should be read alongside a residual plot: structure in the residuals (runs, trends, curvature) diagnoses model failure, while a flat but large scatter diagnoses the error scale.

The exactness χ²_min ~ χ²_N−M holds strictly only for models linear in their parameters with Gaussian, correctly-scaled, uncorrelated errors. For nonlinear models the guarantee is asymptotic and rests on Wilks' theorem: as data accumulate, twice the log-likelihood-ratio between the best fit and the truth converges to a χ² with degrees of freedom equal to the number of fitted parameters, regardless of the parameter values. Boundary cases (a parameter estimated at a physical edge, such as a non-negative rate pinned at zero) break the regularity conditions, and the asymptotic distribution becomes a mixture of chi-squared laws (Chernoff's theorem) rather than a single χ²_N−M.

Common misconceptions. "χ²_ν = 1 proves the model is correct" — no; it proves only consistency with the stated errors, and infinitely many models can be consistent. "Smaller χ²_ν is always better" — false; the target is 1, not 0. "Adding a parameter that lowers χ²_min improves the fit" — only if it lowers χ²_ν, since ν also drops; a genuine improvement must beat the one-unit-per-parameter expectation.

Worked examples

Example 1 — weighted mean of five measurements (M = 1). Five independent readings of a resistance, all with σ = 0.20 Ω: R = {10.1, 9.7, 10.3, 9.9, 10.0} Ω. Fit the single-parameter constant model (the weighted mean).

R̄ = ( Σ R_i/σ_i² ) / ( Σ 1/σ_i² ) = (Σ R_i)/N (equal weights) = 50.0/5 = 10.00 Ω

With equal errors the weighted mean is the ordinary mean; M = 1 parameter. A

residuals R_i − R̄ = { +0.1, −0.3, +0.3, −0.1, 0.0 } Ω

Subtract the fitted constant from each reading. A

χ²_min = Σ (R_i − R̄)²/σ² = (0.01 + 0.09 + 0.09 + 0.01 + 0)/0.04 = 0.20/0.04 = 5.0

Sum the squared standardised residuals. A

ν = N − M = 5 − 1 = 4, χ²_ν = 5.0/4 = 1.25, √(2/ν) = √0.5 = 0.71

One parameter fitted, so four degrees of freedom; compare 1.25 to the expected 1 ± 0.71. B

χ²_min = 5.0, ν = 4, χ²_ν = 1.25

Reading. The reduced chi-squared sits only (1.25 − 1)/0.71 ≈ 0.35 standard deviations above 1 — a completely acceptable fit; the readings are consistent with a single true resistance of 10.00 ± 0.09 Ω (standard error σ/√N).

Example 2 — straight-line fit (M = 2). Five points, all with σ = 0.20: x = {1,2,3,4,5}, y = {2.1, 3.9, 6.2, 7.8, 10.1}. Fit y = a + b x.

b = (N S_xy − S_xS_y)/(N S_xx − S_x²), a = (S_y − b S_x)/N

Equal-weight normal-equation solution for a line (prior result). B

S_x=15, S_y=30.1, S_xx=55, S_xy=110.2, b = (551 − 451.5)/(275 − 225) = 99.5/50 = 1.99

Insert the sums; slope from the determinant form. A

a = (30.1 − 1.99·15)/5 = 0.25/5 = 0.05 ⇒ y = 0.05 + 1.99 x

Back-substitute for the intercept. A

residuals = {+0.06, −0.13, +0.18, −0.21, +0.10}, Σ r² = 0.1070

Subtract fitted values {2.04, 4.03, 6.02, 8.01, 10.00} from the data. A

χ²_min = 0.1070/0.04 = 2.68, ν = N − M = 5 − 2 = 3, χ²_ν = 2.68/3 = 0.89

Two parameters fitted, so three degrees of freedom; √(2/3) = 0.82. B

χ²_min = 2.68, ν = 3, χ²_ν = 0.89

Reading. χ²_ν = 0.89 is within 0.13 standard deviations of 1 — an excellent linear fit. Note that fitting a second parameter dropped ν from 4 to 3: the extra freedom is paid for one degree at a time, exactly as E[χ²_min] = N − M predicts.

Problems

(Easy — dof counting.) A spectrum of 20 channels is fitted with a model having 3 free parameters. State the number of degrees of freedom and the expected value and standard deviation of χ²_min.
Solution
ν = N − M = 20 − 3 = 17. Then E[χ²_min] = 17 and Var = 2ν = 34, so the standard deviation is √34 ≈ 5.8. A good fit would give χ²_min roughly in the range 17 ± 5.8.
(Easy/medium — reduced chi-squared.) The fit in Problem 1 returns χ²_min = 25.3. Compute χ²_ν and assess the fit.
Solution
χ²_ν = 25.3/17 = 1.49. The spread of χ²_ν is √(2/17) = 0.343, so (1.49 − 1)/0.343 = 1.43 standard deviations above 1 (equivalently χ²_min is (25.3−17)/5.8 = 1.43σ high). This is marginal but acceptable — a one-sided p ≈ 0.09; not grounds to reject at the 5% level.
(Medium — weighted mean.) Four voltage readings with unequal errors: V = {5.02 ± 0.02, 4.98 ± 0.02, 5.10 ± 0.04, 4.95 ± 0.04} V. Fit the weighted mean and compute χ²_ν.
Solution
Weights w_i = 1/σ_i²: {2500, 2500, 625, 625}, Σw = 6250. Weighted mean V̄ = Σw_iV_i/Σw_i. Numerator = 2500·5.02 + 2500·4.98 + 625·5.10 + 625·4.95 = 12550 + 12450 + 3187.5 + 3093.75 = 31281.25, so V̄ = 31281.25/6250 = 5.005 V. Residuals V_i−V̄ = {+0.015, −0.025, +0.095, −0.055}. Standardised squares (V_i−V̄)²/σ_i²: 0.000225/0.0004 = 0.5625; 0.000625/0.0004 = 1.5625; 0.009025/0.0016 = 5.641; 0.003025/0.0016 = 1.891. Sum χ²_min = 9.66. With ν = 4 − 1 = 3, χ²_ν = 3.22, i.e. (3.22−1)/√(2/3) = 2.7σ high — the third reading is an outlier and the four values are not mutually consistent.
(Medium — overfitting.) Explain, using the projection picture, why fitting an (N−1)-degree polynomial to N points gives χ²_min = 0, and why this is not a good fit. What is ν?
Solution
An (N−1)-degree polynomial has M = N coefficients, so the model subspace is the whole N-dimensional space. The projection H = I, the orthogonal complement has dimension N − M = 0, and the residual (I−H)n = 0 identically: the curve passes through every point, giving χ²_min = 0. With ν = 0 the reduced chi-squared 0/0 is undefined. There is no residual left to test the model against, so a perfect χ² here signals zero predictive content, not a good description — the fit has simply memorised the noise.
(Hard — significance at large ν.) A cosmology fit has ν = 100 and returns χ²_min = 135. Using the Gaussian approximation to the chi-squared distribution, is the fit acceptable? Contrast with a fit that gives the same χ²_ν at ν = 4.
Solution
χ²_ν = 135/100 = 1.35. For large ν, χ²_min is approximately Gaussian with mean ν = 100 and standard deviation √(2ν) = √200 = 14.1. So z = (135 − 100)/14.1 = 2.48, a one-sided p ≈ 0.007 — the fit is poor and would be rejected at the 1% level. At ν = 4, the same χ²_ν = 1.35 means χ²_min = 5.4, with spread √(2·4) = 2.83, giving z = (5.4 − 4)/2.83 = 0.49 — entirely ordinary. The identical reduced chi-squared is compelling evidence against the model at large ν but meaningless at small ν, because the fractional spread √(2/ν) shrinks as ν grows. Always quote ν alongside χ²_ν.