Estimating the Variance: Bias and Bessel's Correction

The Intuition: Why \(N-1\)?

Imagine you have a set of measurements. To calculate the variance, you need to know how far each point is from the "true" center. In theory, the center is the population mean \(\mu\). However, in practice, you do not know \(\mu\). Instead, you use the sample mean \(\bar{X}\), which you calculated from the very same data points.

Here is the key insight: because \(\bar{X}\) is the average of the data, the data points are, on average, closer to \(\bar{X}\) than they are to the true \(\mu\). Think of it this way: if you move the center of a circle to match the average position of a set of dots, the dots will naturally appear less spread out relative to that new center than they would relative to a fixed, external center. By using \(\bar{X}\), you have "used up" one degree of freedom to fit the center, leaving you with fewer independent pieces of information to estimate the spread. Dividing by \(N-1\) instead of \(N\) corrects for this loss of independence, scaling the result up slightly to give an unbiased estimate of the true population variance.

Defining the Estimators

Let \(X_1, \dots, X_N\) be independent, identically distributed random variables with finite mean \(\mu\) and finite variance \(\sigma^2\). The sample mean is defined as:

The sample variance, denoted \(s^2\), is defined with the divisor \(N-1\):

If we had used \(N\) in the denominator, we would have obtained the biased estimator \(s_N^2 = \frac{1}{N} \sum_{i=1}^{N} (X_i - \bar{X})^2\). The relationship between the two is simply \(s^2 = \frac{N}{N-1} s_N^2\). For large \(N\), the difference is negligible, but for small samples (common in precision experiments), the correction is significant.

Unbiasedness and Expectation

An estimator is unbiased if its expected value equals the true parameter. For the variance, we require \(\mathbb{E}[s^2] = \sigma^2\). The derivation (cited as D-113) shows that this equality holds exactly when the divisor is \(N-1\).

Consider the sum of squared deviations from the sample mean. Algebraically, we can expand \(\sum (X_i - \bar{X})^2\) into \(\sum (X_i - \mu)^2 - N(\bar{X} - \mu)^2\). Taking the expectation, we know that \(\mathbb{E}[\sum (X_i - \mu)^2] = N\sigma^2\). However, the term \(N(\bar{X} - \mu)^2\) is not zero; its expectation is \(N \cdot \text{Var}(\bar{X}) = N \cdot \frac{\sigma^2}{N} = \sigma^2\). Therefore, the expected sum of squared deviations from the sample mean is \(N\sigma^2 - \sigma^2 = (N-1)\sigma^2\). To recover \(\sigma^2\), we must divide by \(N-1\). This confirms that \(s^2\) is an unbiased estimator of \(\sigma^2\).

Degrees of Freedom

The concept of degrees of freedom (dof) provides a physical interpretation of the \(N-1\) divisor. When estimating the variance, we are trying to determine the spread of the data. However, we have already used one constraint to determine the location of the data (the mean). Once the mean is fixed, the data points are no longer completely free to vary; they are constrained to sum to \(N\bar{X}\). This reduces the number of independent pieces of information available to estimate the spread from \(N\) to \(N-1\).

In general, if you estimate \(k\) parameters (such as the mean and the variance) from a sample of size \(N\), the degrees of freedom for the variance estimate will be \(N-k\). For a simple mean and variance estimation, \(k=1\) (the mean), so the dof is \(N-1\). This principle generalizes to linear regression, where the dof for the residual variance is \(N - p\), with \(p\) being the number of fitted parameters.