The Principle of Maximum Likelihood
~6 min read · Tier toggle changes depth (T1 plain · T2 standard · T3 rigour)
You should be comfortable with basic probability density functions, the concept of independent random variables, and elementary calculus (differentiation and logarithms). Familiarity with the Gaussian distribution from the previous lecture is assumed.
Why this matters
Maximum likelihood provides the universal standard for extracting the most probable parameters from noisy experimental data.
In experimental physics, we rarely observe the true state of a system directly; instead, we collect a finite set of measurements that are subject to random fluctuations. We need a rigorous, general method to estimate the underlying physical parameters (such as a decay constant, a mass, or a cross-section) from these observations. The principle of maximum likelihood offers this method by asking a simple question: which parameter values make the observed data most probable? This approach unifies the treatment of different types of noise (Poisson, Gaussian, Binomial) under a single mathematical framework, ensuring that our error bars are statistically consistent and defensible.
The Core Idea
Imagine you are trying to determine the bias of a coin. You flip it ten times and observe seven heads. You might guess the probability of heads is 0.7. Why? Because if the true probability were 0.1, getting seven heads would be extremely unlikely. If it were 0.9, seven heads would be possible but less likely than at 0.7 (about 6% against 27%). The "best" estimate is the one that makes your specific sequence of results the most plausible outcome. Maximum likelihood formalises this intuition. It treats the data as fixed and the parameters as variables. We construct a function, the likelihood, which tells us how probable our data is for any given parameter value. The parameter value that maximises this function is our best estimate. This is not about the probability of the parameter (which is a fixed physical constant), but about the probability of the data given that parameter.
From Joint Probability to Likelihood
Suppose we have a dataset of \( n \) independent measurements \( x_1, x_2, \dots, x_n \). Each measurement is drawn from a probability density function \( f(x; \theta) \), where \( \theta \) is the unknown parameter we wish to estimate. Because the measurements are independent, the joint probability density of observing the entire dataset is the product of the individual densities:
This function \( L(\theta) \) is called the likelihood function. It is crucial to distinguish this from the probability density of the data. In the likelihood, the data \( x_i \) are fixed numbers, and \( \theta \) is the variable. We seek the value \( \hat{\theta} \) that maximises \( L(\theta) \). Since the product of many small numbers can be numerically unstable, and because the natural logarithm is a monotonically increasing function, it is mathematically equivalent and computationally safer to maximise the log-likelihood:
The condition for finding the maximum is that the derivative of the log-likelihood with respect to the parameter vanishes:
This equation is known as the score equation. Solving it yields the maximum-likelihood estimator (MLE).
Example: Estimating a Mean
Consider the case where our measurements are normally distributed with a known variance \( \sigma^2 \) but an unknown mean \( \mu \). The probability density for a single measurement is:
The log-likelihood for \( n \) measurements is:
To find the MLE, we differentiate with respect to \( \mu \) and set the result to zero:
Multiplying by \( \sigma^2 \) and rearranging gives:
Thus, the maximum-likelihood estimator for the mean of a Gaussian distribution is simply the arithmetic mean of the data. This result is intuitive and aligns with our basic statistical expectations.
Uncertainty and the Curvature
Once we have found the best estimate \( \hat{\theta} \), we must quantify its uncertainty. The width of the likelihood function around its maximum tells us how sensitive the data is to changes in the parameter. A sharp peak implies high precision; a broad peak implies low precision.
For large sample sizes, the log-likelihood is approximately quadratic near the maximum. The curvature of this quadratic is related to the Fisher Information. The variance of the MLE is approximately the inverse of the expected curvature. In practice, we often estimate the standard error of the MLE using the second derivative of the log-likelihood evaluated at \( \hat{\theta} \):
This provides a rigorous, model-based way to assign error bars to our estimates, replacing ad-hoc rules of thumb with a principled statistical derivation.
The derivation
The Method of Maximum Likelihood
Given a dataset x = (x1, …, xn) modelled as independent draws from a density (or mass function) f(x; θ) indexed by a parameter θ, the likelihood is the joint density read as a function of θ with the data held fixed, L(θ) = ∏i f(xi; θ). The maximum-likelihood estimator θ̂ maximises L, equivalently the log-likelihood ℓ(θ) = ∑i ln f(xi; θ), and for a regular interior maximum it solves the estimating (score) equation ∑i ∂ ln f(xi; θ)/∂θ = 0.
A common pitfall is assuming that the MLE is always unbiased. While the MLE is asymptotically unbiased (the bias vanishes as \( n \to \infty \)), for small samples, the MLE can be biased. For example, the MLE for the variance of a Gaussian distribution divides by \( n \) rather than \( n-1 \), leading to a slight underestimation of the true variance. Always check for finite-sample bias if your dataset is small. C
Worked example
Problem. A detector records the number of decays in 5 independent time intervals. The counts are \( 12, 15, 11, 14, 13 \). Assume the counts follow a Poisson distribution with mean rate \( \lambda \). Estimate \( \lambda \) and its standard error.
- Write the Poisson probability mass function: \( P(k; \lambda) = \frac{\lambda^k e^{-\lambda}}{k!} \).
- Construct the log-likelihood: \( \ell(\lambda) = \sum_{i=1}^{5} \left( k_i \ln \lambda - \lambda - \ln(k_i!) \right) \).
- Differentiate with respect to \( \lambda \): \( \frac{\partial \ell}{\partial \lambda} = \sum_{i=1}^{5} \frac{k_i}{\lambda} - 5 \).
- Set to zero and solve: \( \frac{\sum k_i}{\lambda} = 5 \implies \hat{\lambda} = \frac{12+15+11+14+13}{5} = \frac{65}{5} = 13 \).
- Calculate the second derivative for uncertainty: \( \frac{\partial^2 \ell}{\partial \lambda^2} = -\frac{\sum k_i}{\lambda^2} \). At \( \hat{\lambda}=13 \), this is \( -\frac{65}{169} \approx -0.3846 \).
- Standard error is \( \sqrt{1/0.3846} \approx \sqrt{2.6} \approx 1.61 \). Alternatively, for Poisson, \( \sigma_{\lambda} = \sqrt{\hat{\lambda}/n} = \sqrt{13/5} \approx 1.61 \).
Units check. The counts are dimensionless, so \( \lambda \) is a dimensionless rate per interval. The standard error has the same units as \( \lambda \).
Three quick tests
- Why do we maximise the log-likelihood instead of the likelihood directly?
Answer
Logarithms convert products into sums, simplifying differentiation and improving numerical stability. - If the likelihood function is symmetric around its maximum, is the MLE necessarily unbiased?
Answer
No, symmetry of the likelihood does not guarantee unbiasedness of the estimator, especially for small samples. - What is the relationship between the curvature of the log-likelihood and the uncertainty of the estimate?
Answer
The uncertainty is inversely proportional to the square root of the curvature; sharper curvature means smaller uncertainty.
Problem set for PU-106 →
Each derivation in this unit ends with five graded problems with full solutions.
Lecture notes drafted with a local model and checked line by line by the editor; every number in the worked example was recomputed.
Derivations cited: D-119 · Threads: chance