Variance of Linear Combinations and Covariance

Derivation

Variance of Linear Combinations and Covariance

D-109 Home PU-106 Threads chance Depends on Expectation, Variance and Their Algebra

Statement

For random variables X₁, …, X_n with finite variances and fixed real constants a₁, …, a_n, the variance of the linear combination S = ∑_i a_i X_i is the double sum Var(S) = ∑_i ∑_j a_i a_j Cov(X_i, X_j), which collapses to the weighted sum of variances ∑_i a_i² Var(X_i) when the variables are pairwise uncorrelated.

Why it matters

Almost every measured quantity in physics is a combination of several noisy inputs: a resistance inferred from voltage and current, a rate averaged over many counts, a fitted slope built from many data points. This identity is the master formula of error propagation for linear functions — it tells you exactly how the uncertainties of the parts assemble into the uncertainty of the whole, and crucially, whether the pieces reinforce or cancel.

It also underpins the single most important result in all of statistical inference: that averaging n independent measurements shrinks the variance by a factor of n, so the standard error falls as 1/√n. The full covariance version warns you when that scaling silently breaks — correlated noise does not average away.

Assumptions

Each X_i has a finite second moment, E[X_i²] < ∞.If any variance is infinite (e.g. a Cauchy-distributed input), Var(S) is undefined and the whole identity is vacuous; the sum of a heavy-tailed input does not have a meaningful spread. The coefficients a_i are deterministic constants, not random.If the weights are themselves random or correlated with the X_i, then Var(∑ a_i X_i) picks up extra cross-terms from Var(a_i) and Cov(a_i, X_i), and this formula understates the true spread. Expectation is linear over the (possibly dependent) collection.Linearity of E holds for any integrable variables regardless of dependence, so no independence is needed to pull constants out or split sums; dropping integrability of the products X_iX_j (guaranteed here by Cauchy–Schwarz from finite variances) would make the covariances themselves ill-defined.

Derivation

S = ∑_i=1ⁿ a_i X_i

Define the linear combination whose variance we seek. A

μ_S = E[S] = E[∑_i a_i X_i] = ∑_i a_i E[X_i] = ∑_i a_i μ_i

Linearity of expectation moves the sum and the constants outside E; valid for dependent variables since no product is involved. A

S − μ_S = ∑_i a_i X_i − ∑_i a_i μ_i = ∑_i a_i (X_i − μ_i)

Subtract the mean and factor each term; the deviation of a linear combination is the linear combination of the deviations. A

(S − μ_S)² = ( ∑_i a_i (X_i − μ_i) )( ∑_j a_j (X_j − μ_j) )

Write the square as a product of two copies, using an independent summation index j for the second factor. B

= ∑_i ∑_j a_i a_j (X_i − μ_i)(X_j − μ_j)

Distribute the product of finite sums; every term of the first factor multiplies every term of the second, giving the full n×n grid of pairs. B

Var(S) = E[(S − μ_S)²] = E[ ∑_i ∑_j a_i a_j (X_i − μ_i)(X_j − μ_j) ]

Apply the definition of variance from the prior result moments–expectation–variance: Var(S) = E[(S − μ_S)²]. B

= ∑_i ∑_j a_i a_j E[(X_i − μ_i)(X_j − μ_j)]

Linearity of expectation over the finite double sum pulls the constants a_ia_j out and pushes E onto each term; the products are integrable by Cauchy–Schwarz. C

Cov(X_i, X_j) ≡ E[(X_i − μ_i)(X_j − μ_j)]

Recognise each inner expectation as the definition of covariance; the diagonal case i = j is Cov(X_i, X_i) = Var(X_i). A

Var(S) = ∑_i ∑_j a_i a_j Cov(X_i, X_j)

Substitute the covariance name back in — the general identity. A

= ∑_i a_i² Var(X_i) + 2 ∑_i<j a_i a_j Cov(X_i, X_j)

Split the grid into diagonal (i = j) and off-diagonal terms; since Cov(X_i, X_j) = Cov(X_j, X_i) the two triangles are equal, giving the factor of 2. B

X_i ⊥ X_j for i ≠ j ⇒ Cov(X_i, X_j) = 0 ⇒ Var(S) = ∑_i a_i² Var(X_i)

Pairwise independence (indeed mere uncorrelatedness) kills every off-diagonal term, leaving additivity of variances. Only uncorrelatedness is required, not full independence. C

Result

Var( ∑_i a_i X_i ) = ∑_i ∑_j a_i a_j Cov(X_i, X_j) ⟶ ^uncorr. ∑_i a_i² Var(X_i)

Reading. The spread of a weighted sum is a weighted sum of all pairwise spreads — each variable with itself (variance) and each pair with the other (covariance). Weights enter squared on the diagonal, so scaling a variable by a scales its variance contribution by a². Off-diagonal covariance terms can be positive (uncertainties reinforce) or negative (they partially cancel). When inputs are uncorrelated, only the diagonal survives and variances simply add.

Units check. If X_i carries physical units [x_i] then a_i carries [S]/[x_i], so each a_i is dimensionless times a ratio and every product a_i a_j Cov(X_i, X_j) has units [S]²/[x_i][x_j] × [x_i][x_j] = [S]². A variance must have the squared units of S — consistent.

Limiting cases

Single variable (n = 1): Var(aX) = a² Var(X) — the familiar scaling rule.
Difference of two uncorrelated variables: Var(X − Y) = Var(X) + Var(Y); variances add even for a difference, because the weight −1 squares to +1.
Perfectly correlated pair (Cov = σ_Xσ_Y): Var(X + Y) = (σ_X + σ_Y)², i.e. standard deviations add directly.
Sample mean of iid (a_i = 1/n, Var(X_i) = σ²): Var(X̄) = σ²/n, the origin of the 1/√n standard error.
Zero net weight with anticorrelation: a suitably weighted difference of positively correlated variables can have vanishing variance, the principle behind differential measurement.

Breaks when

Infinite or undefined variance. If any input has heavy tails with E[X_i²] = ∞ (Cauchy, some power-law noise), the covariances and hence Var(S) do not exist; the formula returns a finite number only by illegally assuming convergence.
Random or data-dependent weights. If the a_i are estimated from the same data (as in a fitted regression, or portfolio weights chosen after seeing returns), they correlate with the X_i and the treated-as-constant derivation omits genuine variance, systematically underestimating the true spread.
Nonlinear combination. For g(X₁, …, X_n) that is not linear, this is only the first-order (delta-method) approximation with a_i = ∂g/∂X_i evaluated at the mean; it breaks down when curvature or fluctuations are large.
Non-stationary / drifting means. If the μ_i shift during data collection, the empirical spread mixes true variance with mean drift and no fixed covariance matrix describes it.

Failure modes

Adding standard deviations instead of variances. Writing σ_S = |a₁|σ₁ + |a₂|σ₂ for uncorrelated inputs. Correct is σ_S² = a₁²σ₁² + a₂²σ₂²; linear addition of σ only holds for perfect correlation.
Forgetting the factor of 2 on cross terms — counting the (i,j) pair but not (j,i), halving the covariance contribution.
Sign error on Var(X − Y). Assuming subtraction subtracts variances; the weight is squared, so uncorrelated variances still add.
Dropping covariances by reflex. Assuming inputs are independent when they share a common systematic (same instrument, same calibration), which can dominate the total.
Confusing uncorrelated with independent. Using Cov = 0 to claim independence — the derivation needs only the former; the converse does not hold.

Discussion

The double sum is best read as a quadratic form. Collect the deviations into a vector and the coefficients into a vector a; then Var(S) = a^T Σ a, where Σ_ij = Cov(X_i, X_j) is the covariance matrix. Every derived identity — scaling, additivity, error propagation — is one geometric statement about how this quadratic form responds to the direction a. Because variance is non-negative for every choice of weights, Σ is positive semi-definite; that is not an extra assumption but a consequence of the formula.

Physically, the diagonal encodes each source's own noise while the off-diagonal encodes shared fluctuation. Two thermometers on the same bench drift together with room temperature: their readings are positively correlated, and a naive sum of their variances understates the uncertainty of their total while a difference (which cancels the common drift) can be far more precise than either alone. This is exactly why lock-in and differential techniques win — they choose weights a that place the measurement in a low-variance direction of the quadratic form.

The result requires strictly less than independence. Nothing in steps 6–9 assumes the joint distribution factorises; only the numerical value Cov(X_i, X_j) = 0 is needed to reach additivity, and uncorrelatedness is weaker than independence (independent ⇒ uncorrelated, but not conversely — e.g. X and X² for symmetric X are uncorrelated yet dependent). Thus additivity of variances is a second-moment statement, blind to any higher-order structure in the dependence. The Central Limit Theorem inherits this: it is the σ²/n scaling, not full independence of every moment, that fixes the width of the limiting Gaussian.

Common misconceptions. "Variances always add" — only for uncorrelated inputs; correlation adds cross terms that can dominate. "Independent is the requirement" — only Cov = 0 is used. "A difference reduces uncertainty" — for uncorrelated inputs a difference has the same variance as the sum; reduction requires positive correlation to cancel.

Worked examples

Two-asset portfolio. Return R = w₁R₁ + w₂R₂. Given σ₁ = 0.20, σ₂ = 0.10, correlation ρ = −0.5, weights w₁ = 0.4, w₂ = 0.6 (dimensionless).

Set up the two-term instance of the identity. A

Var(R) = w₁²σ₁² + w₂²σ₂² + 2 w₁ w₂ ρ σ₁ σ₂

Use Cov = ρσ₁σ₂ in the n = 2 formula. A

= (0.4)²(0.20)² + (0.6)²(0.10)² + 2(0.4)(0.6)(−0.5)(0.20)(0.10)

Insert numbers only now that symbols are arranged. A

= 0.0064 + 0.0036 − 0.0048 = 0.0052

Arithmetic; the negative correlation subtracts, reducing risk. A

Var(R) = 0.0052, σ_R = √0.0052 ≈ 0.072 (7.2%)

Reading. The blended portfolio (7.2%) is less volatile than either asset alone (20% and 10%) because the negative correlation lets one asset's swings partly offset the other's.

Units check. Returns are dimensionless fractions; variance is a pure number, σ_R a fraction — consistent.

Standard error of a mean. Estimate a length by averaging n = 25 independent caliper readings, each with σ = 0.30 mm. Estimator L̄ = (1/n) ∑_i X_i, so a_i = 1/n.

Identify weights and the common variance. A

Var(L̄) = ∑_i (1/n)² Var(X_i) = n · (1/n²) σ² = σ²/n

Independent readings zero the covariances; the n identical diagonal terms sum. B

Var(L̄) = (0.30 mm)² / 25 = 0.09 mm² / 25 = 0.0036 mm²

Insert numbers. A

σ_L̄ = √0.0036 mm² = 0.060 mm = σ / √n

Reading. Averaging 25 readings cuts the uncertainty fivefold (√25 = 5), from 0.30 mm to 0.060 mm — the classic 1/√n improvement.

Units check. Var in mm², standard error in mm — correct dimensions for a length uncertainty.

Problems

Two uncorrelated timers each have σ = 0.5 s. Find the standard deviation of their difference T₁ − T₂.
Solution
Var(T₁ − T₂) = (1)²(0.5)² + (−1)²(0.5)² = 0.25 + 0.25 = 0.50 s². So σ = √0.50 = 0.71 s. The variances add despite the subtraction, because both weights square to 1.
A resistance is computed as R = V/I near V₀ = 10 V, I₀ = 2 A, with independent σ_V = 0.1 V, σ_I = 0.02 A. Using the linear (delta-method) approximation, find σ_R.
Solution
Partials: ∂R/∂V = 1/I₀ = 0.5 Ω/V, ∂R/∂I = −V₀/I₀² = −10/4 = −2.5 Ω/A. Var(R) = (0.5)²(0.1)² + (2.5)²(0.02)² = 0.0025 + 0.0025 = 0.0050 Ω². So σ_R = 0.071 Ω about R₀ = 5 Ω (a 1.4% relative error, matching √(0.01² + 0.01²)).
Show that for constants a, b and any two variables, Var(aX + bY) = a²Var(X) + b²Var(Y) + 2ab Cov(X,Y), then find the value of b/a that minimises the variance for fixed a.
Solution
Expanding the n = 2 double sum gives the stated form directly. Treat as a function of b: d/db [b²Var(Y) + 2ab Cov] = 2b Var(Y) + 2a Cov = 0 ⇒ b/a = −Cov(X,Y)/Var(Y). This is the optimal hedge ratio; substituting back gives the minimum variance a²Var(X)(1 − ρ²).
Ten counts of a radioactive source are summed, each Poisson with mean and variance λ = 100, mutually independent. Find the variance and the fractional uncertainty of the total N = ∑_i=1¹⁰ X_i.
Solution
All weights a_i = 1, independent so covariances vanish: Var(N) = ∑ 1² · 100 = 1000. Mean total = 1000, so σ_N = √1000 ≈ 31.6, fractional uncertainty 31.6/1000 = 3.2% — consistent with Poisson 1/√(∑λ) = 1/√1000.
Two sensors reading the same quantity have σ₁ = σ₂ = 2 units but share a systematic drift giving Cov = +3 units². Compare the variance of the average (X₁ + X₂)/2 against the naive "independent" prediction.
Solution
With a_i = 1/2: Var = (1/4)(4) + (1/4)(4) + 2(1/2)(1/2)(3) = 1 + 1 + 1.5 = 3.5 units². The naive independent estimate would give only 1 + 1 = 2 units². The positive covariance means averaging does not beat the common systematic down — the true spread (σ = 1.87) is larger than the σ = 1.41 one would wrongly predict.