Fisher Information and the Cramer-Rao Bound

The Intuition of Information

Imagine you are trying to estimate the height of a distant mountain using a theodolite. If the air is clear, a small change in the angle of elevation corresponds to a large, easily detectable change in the calculated height. The data is "informative." If the air is thick with fog, the angle readings are noisy and a small change in height is indistinguishable from random fluctuation. The data is "uninformative."

Fisher information formalizes this idea. It measures the curvature of the likelihood function. A sharply peaked likelihood (high curvature) means the data strongly constrains the parameter, resulting in low variance. A flat likelihood (low curvature) means the data is weak, resulting in high variance. The Fisher information is, in essence, the expected curvature of the log-likelihood surface.

Defining Fisher Information

Let \( \theta \) be a parameter of interest, and let \( \ell(\theta) = \ln L(\theta) \) be the log-likelihood function for a single observation. The score function is the first derivative of the log-likelihood with respect to the parameter: \( s(\theta) = \frac{\partial \ell}{\partial \theta} \).

The Fisher information \( I(\theta) \) is defined as the expected value of the square of the score function:

Under regularity conditions, this is equivalent to the negative expected value of the second derivative of the log-likelihood:

For a sample of \( n \) independent and identically distributed (i.i.d.) observations, the total Fisher information is additive:

This additivity is crucial: the information grows in proportion to \( n \), so the variance bound falls as \( 1/n \) and the smallest achievable standard error as \( 1/\sqrt{n} \).

The Cramér–Rao Bound

The Cramér–Rao bound (CRB) states that for any unbiased estimator \( \hat{\theta} \) of a parameter \( \theta \), the variance of the estimator is bounded below by the inverse of the Fisher information:

Since \( I_n(\theta) = n I(\theta) \), we can write:

This inequality implies that no unbiased estimator can have a variance smaller than \( \frac{1}{n I(\theta)} \). An estimator that achieves this bound is called efficient. The MLE is asymptotically efficient, meaning that as \( n \to \infty \), its variance approaches the CRB.

In the multi-parameter case, the bound generalizes to a matrix inequality. Let \( \boldsymbol{\theta} = (\theta_1, \dots, \theta_p) \) be a vector of parameters. The Fisher information matrix \( \mathbf{I}(\boldsymbol{\theta}) \) has elements \( I_{jk} = -\mathrm{E}\left[ \frac{\partial^2 \ell}{\partial \theta_j \partial \theta_k} \right] \). The covariance matrix of any unbiased estimator \( \hat{\boldsymbol{\theta}} \) satisfies:

Here, \( \succeq \) denotes the Löwner order, meaning that the difference \( \mathrm{Cov}(\hat{\boldsymbol{\theta}}) - \mathbf{I}(\boldsymbol{\theta})^{-1} \) is a positive semi-definite matrix. This implies that the variance of each individual parameter estimate is bounded by the corresponding diagonal element of the inverse Fisher information matrix.

Connection to Maximum Likelihood

Recall from the Method of Maximum Likelihood that the MLE \( \hat{\theta} \) satisfies the score equation \( \sum_{i=1}^n \frac{\partial}{\partial \theta} \ln f(x_i; \theta) = 0 \) at \( \theta = \hat{\theta} \).

Using a Taylor expansion of the score function around the true parameter value \( \theta_0 \), we can approximate the MLE as:

For large \( n \), the denominator converges to \( -n I(\theta_0) \) by the Law of Large Numbers. The numerator is a sum of zero-mean random variables with variance \( n I(\theta_0) \). Thus, the variance of \( \hat{\theta} \) is approximately:

This confirms that the MLE achieves the Cramér–Rao bound asymptotically. In practice, we estimate the Fisher information by evaluating the negative Hessian of the log-likelihood at the MLE, \( \hat{\mathbf{I}} = -\left. \frac{\partial^2 \ell}{\partial \boldsymbol{\theta} \partial \boldsymbol{\theta}^T} \right|_{\hat{\boldsymbol{\theta}}} \). The covariance matrix of the MLE is then estimated as \( \hat{\mathbf{C}} = \hat{\mathbf{I}}^{-1} \).