Expectation, Variance and Their Algebra

Derivation

Expectation, Variance and Their Algebra

D-108 Home PU-106 Threads chance Depends on —

Statement

For random variables X, Y on a common probability space and constants a, b ∈ ℝ, expectation is linear, E[aX + bY] = aE[X] + bE[Y], the variance obeys Var(X) = E[X²] − (E[X])², and under an affine map the moments scale as E[aX + b] = aE[X] + b and Var(aX + b) = a²Var(X) — each following directly from the definition of expectation as a probability-weighted sum or integral.

Why it matters

These four identities are the entire arithmetic of first and second moments. Linearity holds without any independence assumption, which is why it is the single most-used tool in probability: it lets us decompose the mean of a complicated sum (a total across many trials, a sample mean, a sum of indicators) into a sum of simple pieces even when those pieces are strongly dependent.

The computational form Var(X) = E[X²] − (E[X])² converts a definition that requires knowing the mean first into a single pass over the second moment, and the affine scaling rules tell us exactly how the mean and spread respond to a change of units or origin — the foundation of standardisation, the z-score, and error propagation.

Assumptions

The expectation exists (is finite): E[|X|] < ∞, and for the variance identity E[X²] < ∞.If the defining sum or integral diverges (e.g. a Cauchy distribution, or a Pareto with tail index ≤ 2), E[X] or Var(X) is undefined and every identity below is vacuous — sample means will not stabilise no matter how much data you collect. The variables share one probability space, so X + Y is itself a random variable with a joint law.If X and Y are defined by unrelated experiments with no joint distribution, the object E[X+Y] has no meaning until a coupling is specified. The series/integral defining E[|aX + bY|] converges absolutely, licensing term-by-term splitting.Without absolute convergence a conditionally convergent series may be rearranged to any value (Riemann), so pulling the sum apart into aE[X] + bE[Y] is not justified and can give a different — or meaningless — answer.

Derivation

E[X] = ∑_x x p(x) (discrete), E[X] = ∫_−∞^∞ x f(x) dx (continuous)

Definition of expectation as the probability-weighted average of the values A

E[g(X,Y)] = ∑_x,y g(x,y) p(x,y)

Law of the unconscious statistician: the expectation of a function is taken against the joint law, no need to first find the law of g B

E[aX + bY] = ∑_x,y (a x + b y) p(x,y)

Apply step 2 with g(x,y) = ax + by A

= a ∑_x,y x p(x,y) + b ∑_x,y y p(x,y)

Absolute convergence lets the double sum split and the constants factor out B t3

∑_x,y x p(x,y) = ∑_x x ∑_y p(x,y) = ∑_x x p_X(x) = E[X]

Marginalise: summing the joint over y gives the marginal p_X; likewise the other sum is E[Y] B

∴ E[aX + bY] = a E[X] + b E[Y]

Substitute step 5 into step 4 — linearity, proved with no independence used A

Var(X) ≡ E[(X − μ)²], μ ≡ E[X]

Definition of variance as the mean squared deviation about the mean A

(X − μ)² = X² − 2μX + μ²

Expand the square algebraically; μ is a constant, not random A

Var(X) = E[X²] − 2μ E[X] + μ²

Apply linearity (step 6) term by term, pulling the constants −2μ and μ² out; note E[μ²] = μ² A

= E[X²] − 2μ² + μ² = E[X²] − μ²

Substitute E[X] = μ and collect the two μ² terms A

∴ Var(X) = E[X²] − (E[X])²

The computational identity; note it forces E[X²] ≥ (E[X])² A

E[aX + b] = a E[X] + b E[1] = a E[X] + b

Affine case of linearity with the degenerate variable Y = 1, for which E[1] = 1 A

Var(aX + b) = E[ ((aX+b) − (aμ+b))² ] = E[a²(X − μ)²]

Use step 12 for the mean; the constant b cancels inside the deviation, and a factors out squared B

∴ Var(aX + b) = a² Var(X)

Pull the constant a² through the expectation — shift-invariance and quadratic scaling A

Result

E[aX + bY] = aE[X] + bE[Y] · Var(X) = E[X²] − (E[X])² · Var(aX + b) = a²Var(X)

Reading. The mean of a linear combination is the same linear combination of the means, always — dependence is irrelevant. The variance is the second moment minus the square of the first. Adding a constant slides the whole distribution and leaves the spread untouched; multiplying by a stretches deviations by |a|, so variance (a squared quantity) scales by a².

Units check. If X carries units [u], then E[X] is in [u], E[X²] and (E[X])² are both in [u²], so Var(X) is in [u²] and the standard deviation σ = √Var returns to [u]. In Var(aX+b)=a²Var(X) the constant a carries [u′/u] and b carries [u′], matching both sides at [u′²].

Limiting cases

Degenerate variable (X = c almost surely): E[X²] = c² = (E[X])², so Var(X) = 0 — no spread, as expected.
Pure shift (a = 1): Var(X + b) = Var(X) — translating data never changes its spread.
Sign flip (a = −1, b = 0): Var(−X) = Var(X) and E[−X] = −E[X].
Standardisation (a = 1/σ, b = −μ/σ): Z = (X−μ)/σ has E[Z]=0, Var(Z)=1.
Independent sum: linearity gives the mean freely; the variance of a sum needs Var(X+Y)=Var(X)+Var(Y)+2Cov(X,Y), reducing to the sum only when Cov = 0.

Breaks when

Heavy tails with no finite mean. For a Cauchy variable the integral ∫ x f(x) dx diverges, so E[X] does not exist; "linearity of expectation" is a statement about a quantity that isn't there, and sample means wander forever instead of converging.
Finite mean but infinite second moment. A Pareto with tail index 1 < α ≤ 2 has a well-defined E[X] but E[X²] = ∞, so Var(X) = E[X²] − (E[X])² is ∞ — the identity is formally correct but useless for error bars, and the CLT in its usual form fails.
Conditionally convergent expectation. If E[|X|] = ∞ while the signed sum appears to converge, the term-by-term split in step 4 is a rearrangement of a conditionally convergent series and can yield any value; linearity genuinely fails.
No joint law. If X and Y come from separate, uncoupled experiments, E[X+Y] is undefined until a joint distribution is imposed, so the linearity equation has no left-hand side.

Failure modes

"Var(X+Y) = Var(X)+Var(Y)" reflex. True only when Cov(X,Y)=0. Students borrow the independence-free status of mean linearity and wrongly grant it to variance.
Dropping the square on the constant. Writing Var(aX+b) = a Var(X) or, worse, Var(aX+b)=a²Var(X)+b. The constant contributes nothing and the multiplier is squared.
Sign error in the computational form. Writing Var(X) = (E[X])² − E[X²], which is ≤ 0 — a guaranteed red flag since variance is non-negative.
Confusing E[X²] with (E[X])². Squaring the mean of a dataset instead of averaging the squares; by Jensen these differ by exactly Var(X) ≥ 0.
Assuming E[XY]=E[X]E[Y] to get linearity. Linearity of the mean needs no such factorisation; only products (and hence covariance) do.
Using the identities when the mean/variance is undefined for a heavy-tailed sample, then trusting a spuriously finite sample statistic.

Discussion

The deepest fact here is operational: expectation is a linear functional on the space of integrable random variables, E: L¹ → ℝ. Steps 3–6 are nothing more than the statement that integration against a fixed measure is linear, and that is why no independence is needed — independence is a property of the joint measure, whereas linearity is a property of the integral. The two live at different levels, which is precisely why students who conflate them mis-apply the variance rule.

The identity Var(X) = E[X²] − (E[X])² is the probabilistic face of the parallel-axis structure: the second moment about any point c is E[(X−c)²] = Var(X) + (E[X]−c)², minimised at c = E[X]. The mean is thus the least-squares centre of the distribution, and the variance is the irreducible residual — the same decomposition that underlies moment of inertia in mechanics and the bias–variance split in statistics.

Linearity scales without limit: E[∑_i X_i] = ∑_i E[X_i] for any finite collection, dependent or not. This is the engine behind the mean of the binomial (a sum of n Bernoulli indicators gives np in one line), the expected number of fixed points of a random permutation (exactly 1, via indicators despite ferocious dependence), and the analysis of randomised algorithms.

Rigorously, expectation is the Lebesgue integral E[X] = ∫_Ω X dP, and steps 4–5 are Fubini–Tonelli plus dominated convergence: absolute integrability (Tonelli on |X|) is exactly the hypothesis that licenses interchanging the order of summation/integration. The variance is then the squared L²-norm of the centred variable, Var(X) = ‖X − μ‖²₂, and the identity Var(X)=E[X²]−μ² is the Pythagorean relation in L²(P) between X, its projection onto the constants (μ), and the residual — the constant subspace and the mean-zero subspace being orthogonal complements.

Common misconceptions. "Linearity of expectation requires independence" — false, it never does. "Variance adds like the mean" — false, it adds only up to 2Cov. "E[X²] = (E[X])²" — false in general; equality holds iff X is constant.

Worked examples

Fair six-sided die: p(x) = 1/6, x ∈ {1,…,6}

Compute E[X], E[X²], then Var(X) and the effect of the affine map Y = 3X + 2 A

E[X] = (1/6)(1+2+3+4+5+6) = 21/6 = 3.5

Definition, step 1 A

E[X²] = (1/6)(1+4+9+16+25+36) = 91/6 ≈ 15.1667

LOTUS with g(x)=x² A

Var(X) = 91/6 − (7/2)² = 91/6 − 49/4 = 182/12 − 147/12 = 35/12 ≈ 2.9167

Computational identity, step 11 A

E[Y] = 3(3.5)+2 = 12.5, Var(Y) = 3²(35/12) = 9·35/12 = 105/4 = 26.25

Affine rules, steps 12 and 14; the +2 shifts the mean but not the variance A

E[X] = 3.5, Var(X) = 35/12 ≈ 2.917 (σ ≈ 1.708); E[3X+2] = 12.5, Var(3X+2) = 26.25

Reading. The die averages 3.5 with a spread of about 1.7 pips. Tripling and shifting the payout triples the average pip count and adds 2, while the variance grows nine-fold — the +2 is invisible to the spread.

Units check. With X in pips, Var is in pips² and σ back in pips; Var(Y) in (payout-unit)².

Sum of two independent fair dice: S = X₁ + X₂

Get E[S] from linearity (no independence needed) and Var(S) from the covariance rule (here independence is used) B

E[S] = E[X₁] + E[X₂] = 3.5 + 3.5 = 7

Linearity, step 6 A

Var(S) = Var(X₁) + Var(X₂) + 2Cov(X₁,X₂), Cov = 0 (independent)

General variance-of-sum; covariance vanishes under independence B

Var(S) = 35/12 + 35/12 = 70/12 = 35/6 ≈ 5.833

Add the equal variances A

Cross-check via E[S²]: E[S²] = Var(S) + (E[S])² = 35/6 + 49 = 329/6 ≈ 54.83

Rearranged identity, confirming consistency A

E[S] = 7, Var(S) = 35/6 ≈ 5.833 (σ ≈ 2.415)

Reading. Two dice centre on 7 — the familiar peak of the sum — with a spread of about 2.4, larger than one die but only by √2 in standard deviation, not by 2.

Units check. Pips throughout; Var in pips², σ in pips.

Problems

A Bernoulli variable takes value 1 with probability p and 0 otherwise. Find E[X] and Var(X).
Solution
E[X] = 1·p + 0·(1−p) = p. Since X² = X (values are 0 or 1), E[X²] = p, so Var(X) = p − p² = p(1−p). Maximal at p = 1/2 giving 1/4.
Using linearity and problem 1, find the mean and variance of a Binomial(n,p) variable written as S = ∑_i=1ⁿ X_i of independent Bernoullis.
Solution
E[S] = ∑ E[X_i] = np by linearity (no independence needed). For the variance, independence gives zero covariances, so Var(S) = ∑ Var(X_i) = n p(1−p). E.g. n=20, p=0.3: E[S]=6, Var(S)=20(0.3)(0.7)=4.2, σ≈2.05.
Exam marks X have E[X] = 62, Var(X) = 144. The marks are rescaled by Y = 1.1X + 5. Find E[Y], Var(Y), and the standard deviation σ_Y.
Solution
E[Y] = 1.1(62)+5 = 68.2+5 = 73.2. Var(Y) = (1.1)²(144) = 1.21×144 = 174.24. σ_Y = 1.1×√144 = 1.1×12 = 13.2. The additive 5 leaves the spread untouched; only the 1.1 factor scales it.
A continuous variable is uniform on [0, L] with density f(x) = 1/L. Derive E[X] and Var(X) from the integral definition.
Solution
E[X] = ∫₀^L x(1/L)dx = (1/L)(L²/2) = L/2. E[X²] = ∫₀^L x²(1/L)dx = (1/L)(L³/3) = L²/3. Then Var(X) = L²/3 − (L/2)² = L²/3 − L²/4 = L²/12. Standard deviation L/(2√3).
Two variables have Var(X)=9, Var(Y)=16, and correlation ρ = 0.5. Find Var(X+Y) and Var(X−Y). Comment on why they differ.
Solution
Cov(X,Y) = ρ σ_Xσ_Y = 0.5×3×4 = 6. Then Var(X+Y) = 9+16+2(6) = 37 and Var(X−Y) = 9+16−2(6) = 13. Positive correlation means the two tend to move together, so their sum swings more and their difference less. This shows variance is not additive in general — only the mean is, by pure linearity.