physics2u
PhysicsMathsChemistryBiology
Tier
⌕ Search ⌘K
Derivation

Singular Value Decomposition

Statement

Let \(A:\mathcal{V}\to\mathcal{W}\) be a linear map between finite-dimensional inner-product spaces over \(\mathbb{C}\) (or \(\mathbb{R}\)), with \(\dim\mathcal{V}=n\) and \(\dim\mathcal{W}=m\). Then there exist an orthonormal basis \(\{v_i\}\) of \(\mathcal{V}\), an orthonormal basis \(\{u_j\}\) of \(\mathcal{W}\), and real numbers \(\sigma_1\ge\sigma_2\ge\cdots\ge\sigma_p\ge 0\) with \(p=\min(m,n)\), such that in these bases \(A=U\Sigma V^{\dagger}\), where \(U\) and \(V\) are unitary and \(\Sigma\in\mathbb{R}^{m\times n}\) is diagonal with non-negative entries \(\Sigma_{ii}=\sigma_i\).

Why it matters

The singular value decomposition (SVD) is the universal normal form of a linear map: every matrix, square or rectangular, real or complex, normal or not, factors into a rotation/reflection, an axis-aligned stretch, and another rotation/reflection. Where the spectral theorem applies only to self-adjoint (or normal) operators, the SVD applies to all of them by trading a single eigenbasis for two coupled orthonormal bases in domain and codomain.

It is the computational and conceptual backbone of rank, pseudo-inverses, low-rank approximation (Eckart–Young), operator norms, principal component analysis, and the polar decomposition. In physics it underlies the Schmidt decomposition of bipartite quantum states and the entanglement spectrum, and it gives the tightest numerically stable measure of how far a map is from singular.

Assumptions
Finite dimension.In infinite dimensions the argument via the spectral theorem needs compactness; without it the singular "values" form a continuous spectrum and no discrete factorization \(A=U\Sigma V^{\dagger}\) exists. An inner product on each space.Adjoints, orthonormality and unitarity are all defined through \(\langle\cdot,\cdot\rangle\); drop it (a bare Banach or general vector space) and \(A^{\dagger}\), "orthonormal" and "unitary" have no meaning, so the theorem cannot even be stated. Field admitting non-negative square roots (\(\mathbb{R}\) or \(\mathbb{C}\)).The singular values are \(\sigma_i=\sqrt{\lambda_i}\) of eigenvalues \(\lambda_i\ge 0\); over a field without an order or square roots (e.g. a finite field) this real, non-negative \(\Sigma\) does not exist and the decomposition fails. Positive semi-definiteness of \(A^{\dagger}A\) from the spectral theorem.We rely on \(A^{\dagger}A\) being self-adjoint with real, non-negative spectrum; if that prior result is unavailable the reality and non-negativity of the \(\sigma_i\) are not guaranteed.
Derivation
1
\[ M \equiv A^{\dagger}A,\qquad M^{\dagger}=(A^{\dagger}A)^{\dagger}=A^{\dagger}A=M \]
Form the Gram operator on the domain. The adjoint of a product reverses order, \((XY)^{\dagger}=Y^{\dagger}X^{\dagger}\), and \((A^{\dagger})^{\dagger}=A\), so \(M\) is self-adjoint. A
2
\[ \langle x, Mx\rangle=\langle x, A^{\dagger}Ax\rangle=\langle Ax, Ax\rangle=\lVert Ax\rVert^{2}\ge 0 \]
Moving \(A^{\dagger}\) across the inner product by the defining property of the adjoint shows \(M\) is positive semi-definite: every eigenvalue is real and non-negative. B
3
\[ M=V\Lambda V^{\dagger},\qquad \Lambda=\operatorname{diag}(\lambda_1,\dots,\lambda_n),\quad \lambda_1\ge\cdots\ge\lambda_r>0=\lambda_{r+1}=\cdots=\lambda_n \]
The spectral theorem for self-adjoint operators (prior result) gives an orthonormal eigenbasis \(\{v_i\}\) with real eigenvalues; step 2 forces \(\lambda_i\ge 0\). Order them and set \(r=\operatorname{rank}M=\operatorname{rank}A\). B
4
\[ \sigma_i\equiv\sqrt{\lambda_i}\ \ge 0,\qquad i=1,\dots,n \]
Define the singular values as the non-negative square roots. This is well defined precisely because \(\lambda_i\ge 0\). A
5
\[ u_i\equiv\frac{1}{\sigma_i}\,A v_i,\qquad i=1,\dots,r \]
For the strictly positive singular values, define candidate output vectors. Division by \(\sigma_i\) is legal since \(\sigma_i>0\) for \(i\le r\). A
6
\[ \langle u_i,u_j\rangle=\frac{\langle Av_i,Av_j\rangle}{\sigma_i\sigma_j}=\frac{\langle v_i,A^{\dagger}Av_j\rangle}{\sigma_i\sigma_j}=\frac{\lambda_j\langle v_i,v_j\rangle}{\sigma_i\sigma_j}=\frac{\lambda_j}{\sigma_i\sigma_j}\,\delta_{ij}=\delta_{ij} \]
Using \(A^{\dagger}Av_j=\lambda_j v_j\) and orthonormality of \(\{v_i\}\); on the diagonal \(\lambda_i/\sigma_i^{2}=1\). So \(\{u_1,\dots,u_r\}\) is orthonormal. C
7
\[ \{u_1,\dots,u_r\}\ \longrightarrow\ \{u_1,\dots,u_r,u_{r+1},\dots,u_m\}\quad\text{orthonormal basis of }\mathcal{W} \]
Extend the orthonormal set to a full orthonormal basis of the \(m\)-dimensional codomain (Gram–Schmidt on any completing set). The added \(u_j\) span \((\operatorname{im}A)^{\perp}\). B
8
\[ i>r:\quad \lVert Av_i\rVert^{2}=\langle v_i,A^{\dagger}Av_i\rangle=\lambda_i=0\ \Rightarrow\ Av_i=0=\sigma_i u_i \]
On the kernel eigenvectors the map annihilates, and there \(\sigma_i=0\) too, so the relation \(Av_i=\sigma_i u_i\) holds for every \(i=1,\dots,n\). C
9
\[ AV=U\Sigma \quad\Longrightarrow\quad A=U\Sigma V^{\dagger},\qquad \Sigma_{ji}=\sigma_i\,\delta_{ji} \]
Stacking \(Av_i=\sigma_i u_i\) as columns gives \(AV=U\Sigma\); right-multiplying by \(V^{\dagger}=V^{-1}\) (unitary, prior result: unitaries preserve the inner product hence \(V^{\dagger}V=I\)) yields the factorization. B
Result
\[ A=U\Sigma V^{\dagger},\qquad U^{\dagger}U=I_m,\quad V^{\dagger}V=I_n,\quad \Sigma=\operatorname{diag}(\sigma_1,\dots,\sigma_p)\ge 0 \]

Reading. Any linear map is a rotation/reflection of the domain (\(V^{\dagger}\)), a non-negative rescaling along orthogonal axes (\(\Sigma\)), and a rotation/reflection into the codomain (\(U\)). The right singular vectors \(v_i\) are the principal input axes, the left singular vectors \(u_i\) their images, and \(\sigma_i\) the stretch factors; the number of non-zero \(\sigma_i\) is the rank.

Units check. \(U\) and \(V\) are unitary, hence dimensionless (their columns are unit vectors). If the entries of \(A\) carry a physical dimension \([A]\), dimensional consistency of \(A=U\Sigma V^{\dagger}\) forces every singular value to carry that same dimension, \([\sigma_i]=[A]\). The eigenvalues of \(A^{\dagger}A\) then carry \([A]^{2}\), consistent with \(\sigma_i=\sqrt{\lambda_i}\).

Limiting cases
  • Self-adjoint \(A=A^{\dagger}\): \(\sigma_i=|\lambda_i|\), and \(u_i=\operatorname{sgn}(\lambda_i)\,v_i\); the SVD reduces to the spectral decomposition up to signs.
  • Unitary \(A\): \(A^{\dagger}A=I\) so all \(\sigma_i=1\) and \(\Sigma=I\); a pure rotation/reflection stretches nothing.
  • Rank-one \(A=\sigma\,u v^{\dagger}\): a single non-zero singular value; the map projects onto \(v\) and outputs along \(u\).
  • Square, \(m=n\): \(|\det A|=\prod_i\sigma_i\), so \(A\) is invertible iff every \(\sigma_i>0\); the condition number is \(\kappa=\sigma_1/\sigma_n\).
Breaks when
  • Non-compact infinite-dimensional operators. The multiplication operator \((Af)(x)=x\,f(x)\) on \(L^2[0,1]\) is bounded, self-adjoint, yet has purely continuous spectrum and no eigenvectors; there is no discrete \(\Sigma\) and no factorization \(U\Sigma V^{\dagger}\).
  • No inner product (general Banach or bare vector spaces). Without \(\langle\cdot,\cdot\rangle\) there is no adjoint \(A^{\dagger}\), no notion of orthonormal or unitary, so the statement is undefined — one only gets the weaker rank normal form \(A=P\left(\begin{smallmatrix}I&0\\0&0\end{smallmatrix}\right)Q\) with invertible (not unitary) \(P,Q\).
  • Fields without ordered square roots. Over \(\mathbb{Q}\) or a finite field, \(\sqrt{\lambda_i}\) need not exist in the field and "non-negative" is meaningless, so a real diagonal \(\Sigma\ge 0\) cannot be formed.
Failure modes
  • Singular values equal eigenvalues of \(A\). For non-normal \(A\) they differ; e.g. \(\left(\begin{smallmatrix}0&1\\0&0\end{smallmatrix}\right)\) has eigenvalues \(0,0\) but singular values \(1,0\).
  • Using \(\sigma_i=\lambda_i\) instead of \(\sqrt{\lambda_i}\). Forgetting the square root when passing from \(A^{\dagger}A\) to \(\Sigma\).
  • Reusing the \(v_i\) as the \(u_i\). The two bases live in different spaces and coincide only for special (self-adjoint, up to sign) maps.
  • Building \(V\) from eigenvectors of \(AA^{\dagger}\) (or \(U\) from \(A^{\dagger}A\)). Swapping the domain and codomain Gram operators; \(V\) comes from \(A^{\dagger}A\), \(U\) from \(AA^{\dagger}\).
  • Claiming \(U,V\) are unique. Repeated \(\sigma_i\) leave rotational freedom in the corresponding subspace, and each column has a phase/sign ambiguity — only the multiset \(\{\sigma_i\}\) is unique.
  • Forgetting to normalise \(u_i=Av_i/\sigma_i\). Omitting the \(1/\sigma_i\) breaks orthonormality of \(U\).
Discussion

The SVD is best read geometrically: the image of the unit sphere under any linear map is a (possibly degenerate) hyper-ellipsoid, and the SVD hands you exactly its principal semi-axes. The directions \(u_i\) point along those axes, the lengths \(\sigma_i\) are the semi-axis magnitudes, and the \(v_i\) are the pre-images on the domain sphere. This is why the largest singular value is the operator norm, \(\lVert A\rVert_2=\sigma_1=\max_{\lVert x\rVert=1}\lVert Ax\rVert\), and the smallest non-zero one measures the distance to the nearest lower-rank map.

The construction exposes the four fundamental subspaces at once: \(\{v_1,\dots,v_r\}\) spans the row space, \(\{v_{r+1},\dots,v_n\}\) the kernel, \(\{u_1,\dots,u_r\}\) the image, and \(\{u_{r+1},\dots,u_m\}\) the cokernel. The relation \(Av_i=\sigma_i u_i\) couples them pairwise, which is precisely the data that makes the Moore–Penrose pseudo-inverse \(A^{+}=V\Sigma^{+}U^{\dagger}\) (invert the non-zero \(\sigma_i\)) the correct least-squares solver.

The SVD is the "unpolarised" cousin of the polar decomposition \(A=QP\) with \(Q=UV^{\dagger}\) unitary and \(P=V\Sigma V^{\dagger}\) positive semi-definite: the SVD diagonalises the stretch \(P\) in its own eigenbasis. In continuum mechanics this is exactly the split of the deformation gradient into rotation and principal stretches, so the singular values are the physical stretch ratios of a deformed body.

In quantum information the SVD of the coefficient matrix of a bipartite pure state \(|\psi\rangle=\sum_{ij}C_{ij}\,|i\rangle_A|j\rangle_B\) is the Schmidt decomposition: writing \(C=U\Sigma V^{\dagger}\) gives \(|\psi\rangle=\sum_k \sigma_k\,|\tilde u_k\rangle_A|\tilde v_k\rangle_B\) with \(\sum_k\sigma_k^{2}=1\). The singular values are the Schmidt coefficients; the reduced density matrix \(\rho_A=CC^{\dagger}\) has eigenvalues \(\sigma_k^{2}\), the number of non-zero ones is the Schmidt rank, and the entanglement entropy is \(-\sum_k\sigma_k^{2}\log\sigma_k^{2}\). Thus "how entangled" is literally "how far the singular value spectrum spreads."

Common misconceptions. The SVD is not "just diagonalisation of \(A\)" — it diagonalises the map between two adapted bases, so it exists even when \(A\) is not diagonalisable at all. And a small determinant does not mean a matrix is near-singular in the SVD sense; near-singularity is controlled by the smallest singular value, not by \(\det A=\prod_i\sigma_i\), which can be small merely because the dimension is large.

Worked examples

Example 1 — a full-rank \(2\times2\) map. Find the SVD of \(A=\begin{pmatrix}3&0\\4&5\end{pmatrix}\).

1
\[ A^{\dagger}A=\begin{pmatrix}3&4\\0&5\end{pmatrix}\begin{pmatrix}3&0\\4&5\end{pmatrix}=\begin{pmatrix}25&20\\20&25\end{pmatrix} \]
Form the Gram matrix (real, so \(A^{\dagger}=A^{T}\)). A
2
\[ \lambda=25\pm 20\ \Rightarrow\ \lambda_1=45,\ \lambda_2=5;\qquad v_1=\tfrac{1}{\sqrt2}\begin{pmatrix}1\\1\end{pmatrix},\ v_2=\tfrac{1}{\sqrt2}\begin{pmatrix}1\\-1\end{pmatrix} \]
The matrix \(\begin{pmatrix}25&20\\20&25\end{pmatrix}\) has eigenvectors \((1,\pm1)\); normalise. A
3
\[ \sigma_1=\sqrt{45}=3\sqrt5\approx 6.708,\qquad \sigma_2=\sqrt5\approx 2.236 \]
Take non-negative square roots. Check: \(\sigma_1\sigma_2=\sqrt{225}=15=|\det A|\). A
4
\[ u_1=\frac{Av_1}{\sigma_1}=\frac{1}{3\sqrt5}\cdot\frac{1}{\sqrt2}\begin{pmatrix}3\\9\end{pmatrix}=\frac{1}{\sqrt{10}}\begin{pmatrix}1\\3\end{pmatrix},\quad u_2=\frac{Av_2}{\sigma_2}=\frac{1}{\sqrt5}\cdot\frac{1}{\sqrt2}\begin{pmatrix}3\\-1\end{pmatrix}=\frac{1}{\sqrt{10}}\begin{pmatrix}3\\-1\end{pmatrix} \]
Apply \(u_i=Av_i/\sigma_i\). Both are unit vectors and \(u_1\cdot u_2=0\), as required. B
\[ A=\underbrace{\frac{1}{\sqrt{10}}\begin{pmatrix}1&3\\3&-1\end{pmatrix}}_{U}\underbrace{\begin{pmatrix}3\sqrt5&0\\0&\sqrt5\end{pmatrix}}_{\Sigma}\underbrace{\frac{1}{\sqrt2}\begin{pmatrix}1&1\\1&-1\end{pmatrix}}_{V^{\dagger}} \]

Reading. The map stretches the input direction \((1,1)/\sqrt2\) by \(3\sqrt5\) into \((1,3)/\sqrt{10}\), and \((1,-1)/\sqrt2\) by \(\sqrt5\) into \((3,-1)/\sqrt{10}\). Condition number \(\kappa=\sigma_1/\sigma_2=3\).

Units check. If \(A\)'s entries were in metres, both non-zero entries of \(\Sigma\) (namely \(3\sqrt5,\sqrt5\)) are in metres while \(U,V\) are pure numbers — consistent.

Example 2 — a rank-deficient rectangular map. Find the SVD of the \(2\times3\) matrix \(A=\begin{pmatrix}1&0&1\\0&1&1\end{pmatrix}\).

1
\[ AA^{\dagger}=\begin{pmatrix}1&0&1\\0&1&1\end{pmatrix}\begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix}=\begin{pmatrix}2&1\\1&2\end{pmatrix} \]
Use the smaller Gram matrix on the codomain (\(2\times2\)) to get \(U\) and the singular values first. B
2
\[ \lambda=2\pm1\ \Rightarrow\ \lambda_1=3,\ \lambda_2=1;\qquad \sigma_1=\sqrt3,\ \sigma_2=1;\qquad u_1=\tfrac{1}{\sqrt2}\begin{pmatrix}1\\1\end{pmatrix},\ u_2=\tfrac{1}{\sqrt2}\begin{pmatrix}1\\-1\end{pmatrix} \]
Eigenvalues of \(\begin{pmatrix}2&1\\1&2\end{pmatrix}\) with eigenvectors \((1,\pm1)\); take non-negative roots. A
3
\[ v_1=\frac{A^{\dagger}u_1}{\sigma_1}=\frac{1}{\sqrt3}\cdot\frac{1}{\sqrt2}\begin{pmatrix}1\\1\\2\end{pmatrix}=\frac{1}{\sqrt6}\begin{pmatrix}1\\1\\2\end{pmatrix},\qquad v_2=\frac{A^{\dagger}u_2}{\sigma_2}=\frac{1}{\sqrt2}\begin{pmatrix}1\\-1\\0\end{pmatrix} \]
Recover the right singular vectors by \(v_i=A^{\dagger}u_i/\sigma_i\) (the mirror of step 5). B
4
\[ Av=0:\ x_1+x_3=0,\ x_2+x_3=0\ \Rightarrow\ v_3=\frac{1}{\sqrt3}\begin{pmatrix}1\\1\\-1\end{pmatrix},\qquad \sigma_3=0 \]
Complete \(\{v_1,v_2\}\) with a unit kernel vector so that \(V\) is a full orthonormal \(3\times3\) basis. C
\[ A=\underbrace{\frac{1}{\sqrt2}\begin{pmatrix}1&1\\1&-1\end{pmatrix}}_{U}\underbrace{\begin{pmatrix}\sqrt3&0&0\\0&1&0\end{pmatrix}}_{\Sigma}\underbrace{\begin{pmatrix}1/\sqrt6&1/\sqrt6&2/\sqrt6\\1/\sqrt2&-1/\sqrt2&0\\1/\sqrt3&1/\sqrt3&-1/\sqrt3\end{pmatrix}}_{V^{\dagger}} \]

Reading. The map has rank \(2\) (two non-zero singular values), so it is onto \(\mathbb{R}^2\) with a one-dimensional kernel spanned by \((1,1,-1)/\sqrt3\). The third row of \(V^{\dagger}\) carries \(\sigma_3=0\) and contributes nothing to \(A\).

Units check. \(\Sigma\) is \(2\times3\) matching \(A\); the two non-zero entries \(\sqrt3,1\) carry the dimension of \(A\)'s entries, and the unitary factors are dimensionless.

Problems
  1. Show directly from \(A=U\Sigma V^{\dagger}\) that \(A^{\dagger}A=V\,\Sigma^{\dagger}\Sigma\,V^{\dagger}\) and hence that the columns of \(V\) are eigenvectors of \(A^{\dagger}A\) with eigenvalues \(\sigma_i^{2}\).
    SolutionSince \(U^{\dagger}U=I\), compute \(A^{\dagger}A=(U\Sigma V^{\dagger})^{\dagger}(U\Sigma V^{\dagger})=V\Sigma^{\dagger}U^{\dagger}U\Sigma V^{\dagger}=V(\Sigma^{\dagger}\Sigma)V^{\dagger}\). Here \(\Sigma^{\dagger}\Sigma=\operatorname{diag}(\sigma_1^{2},\dots,\sigma_n^{2})\) (an \(n\times n\) diagonal matrix). Right-multiplying by the \(i\)-th standard vector, \(A^{\dagger}A\,v_i=V(\Sigma^{\dagger}\Sigma)e_i=\sigma_i^{2}Ve_i=\sigma_i^{2}v_i\). So each \(v_i\) is an eigenvector with eigenvalue \(\sigma_i^{2}\), confirming the construction.
  2. Compute the singular values of \(A=\begin{pmatrix}0&2\\0&0\end{pmatrix}\) and compare them with its eigenvalues.
    Solution\(A^{\dagger}A=\begin{pmatrix}0&0\\2&0\end{pmatrix}\begin{pmatrix}0&2\\0&0\end{pmatrix}=\begin{pmatrix}0&0\\0&4\end{pmatrix}\), eigenvalues \(0,4\), so \(\sigma_1=2,\ \sigma_2=0\). The eigenvalues of \(A\) itself are both \(0\) (it is nilpotent, \(A^2=0\)). This shows singular values and eigenvalues genuinely differ for non-normal maps: \(A\) has operator norm \(\sigma_1=2\) despite having zero spectrum.
  3. Let \(A=\begin{pmatrix}2&0\\0&3\end{pmatrix}\). Write its SVD and its polar decomposition \(A=QP\).
    Solution\(A^{\dagger}A=\operatorname{diag}(4,9)\), so \(\sigma_1=3,\ \sigma_2=2\) with \(v_1=(0,1),\ v_2=(1,0)\) (ordering by decreasing \(\sigma\)). Then \(u_i=Av_i/\sigma_i\) gives \(u_1=(0,1),\ u_2=(1,0)\). Thus \(U=V=\begin{pmatrix}0&1\\1&0\end{pmatrix}\) and \(\Sigma=\operatorname{diag}(3,2)\); indeed \(U\Sigma V^{\dagger}=\operatorname{diag}(2,3)=A\). Polar: \(Q=UV^{\dagger}=I\) and \(P=V\Sigma V^{\dagger}=\operatorname{diag}(2,3)=A\) — as expected for a positive-definite matrix, the rotation is trivial and \(A\) is its own positive part.
  4. For \(A=\begin{pmatrix}3&0\\4&5\end{pmatrix}\) of Example 1, use the SVD to write the best rank-one approximation \(A_1\) and give the approximation error in the operator norm.
    SolutionBy Eckart–Young the best rank-one approximation keeps the largest singular triple: \(A_1=\sigma_1 u_1 v_1^{\dagger}=3\sqrt5\cdot\frac{1}{\sqrt{10}}\begin{pmatrix}1\\3\end{pmatrix}\cdot\frac{1}{\sqrt2}\begin{pmatrix}1&1\end{pmatrix}\). The scalar prefactor is \(3\sqrt5/\sqrt{20}=3\sqrt5/(2\sqrt5)=3/2\), so \(A_1=\frac{3}{2}\begin{pmatrix}1&1\\3&3\end{pmatrix}=\begin{pmatrix}1.5&1.5\\4.5&4.5\end{pmatrix}\). The error is the next singular value: \(\lVert A-A_1\rVert_2=\sigma_2=\sqrt5\approx 2.236\).
  5. A bipartite state has coefficient matrix \(C=\frac{1}{\sqrt2}\begin{pmatrix}1&0\\0&1\end{pmatrix}\). Find its Schmidt coefficients and the entanglement entropy \(S=-\sum_k\sigma_k^{2}\log_2\sigma_k^{2}\).
    Solution\(C\) is already \(\tfrac{1}{\sqrt2}I\), so \(C^{\dagger}C=\tfrac12 I\) with both eigenvalues \(\tfrac12\); hence \(\sigma_1=\sigma_2=\tfrac{1}{\sqrt2}\) and the Schmidt coefficients are \(1/\sqrt2,1/\sqrt2\) (normalisation \(\sum\sigma_k^2=1\) holds). The Schmidt rank is \(2\). Entropy: \(S=-2\cdot\tfrac12\log_2\tfrac12=\log_2 2=1\) bit — a maximally entangled two-qubit (Bell) state, as expected.