Singular Value Decomposition

Definition

The Singular Value Decomposition (SVD) factorizes any $m \times n$ matrix $A$ into two orthogonal matrices and a diagonal matrix of singular values: $A = U \Sigma V^T$ where $U$ is $m \times m$ (orthogonal), $\Sigma$ is $m \times n$ (diagonal), and $V$ is $n \times n$ (orthogonal).

• How to read: “A equals U times Sigma times V-transpose.”
• Meaning: Decompose any matrix into rotation ($V^T$), scaling ($\Sigma$), and rotation ($U$)—works even for non-square and non-symmetric matrices.

Why It Matters

SVD is the ‘Swiss Army Knife’ of linear algebra; it allows for massive data compression and noise reduction by identifying the most important ‘directions’ in any matrix, making it indispensable for modern machine learning and image processing.

Core Concepts

• Singular Values ($\sigma_i$): The square roots of the eigenvalues of $A^T A$ (and $AA^T$). They are non-negative and usually ordered $\sigma_1 \geq \sigma_2 \geq \dots \geq \sigma_r > 0$.

  • How to read: “The singular values are ordered such that sigma one is greater than or equal to sigma two, continuing in decreasing order down to a positive value.”
  • Meaning: Diagonal entries of $\Sigma$—measure how much $A$ stretches along each principal direction.

• Bases for Fundamental Subspaces:

  • Columns of $U$: Orthonormal basis for the Column Space $C(A)$ and Left Nullspace $N(A^T)$.
  • Columns of $V$: Orthonormal basis for the Row Space $C(A^T)$ and Nullspace $N(A)$.
  • How to read: “Columns of U; columns of V.”
  • Meaning: Columns of $U$ span the column space $C(A)$ and left nullspace $N(A^T)$; columns of $V$ span the row space $C(A^T)$ and nullspace $N(A)$.

• Stability: Unlike eigenvalues, singular values are numerically stable even for non-symmetric or rectangular matrices.

• Rank-k Approximation: $A_k = \sum_{i=1}^k \sigma_i u_i v_i^T$ is the best rank-$k$ approximation of $A$ (Eckart-Young Theorem).

  • How to read: “The rank k approximation of matrix A equals the sum from index i equals one to k of the singular value sigma i times the vector u i times the transpose of vector v i.”
  • Meaning / when to use: Keep only the top $k$ singular values/vectors for compression—minimizes Frobenius-norm error.

Connected Concepts

• Matrix Algebra: Inverses
• Principal Component Analysis
• Pseudoinverse
• Dimensionality Reduction