3. Matrix Decompositions and Spectral Methods

Introduction and context

Matrices are among the central mathematical objects in machine learning. A data table, a linear transformation, a covariance structure, a system of equations, and the parameters of a neural network layer can all be represented using matrices. Yet a matrix in its original form often conceals the structure that is most useful for analysis. Matrix decompositions reveal that structure by expressing a matrix as a product of simpler matrices with interpretable properties.

Spectral methods are especially important because they describe linear transformations through eigenvalues and eigenvectors. They help identify directions of high variance, stable and unstable modes, low-dimensional structure, and the geometry of optimization problems. Singular value decomposition extends these ideas to arbitrary rectangular matrices and is consequently one of the most broadly useful tools in scientific computing and machine learning.

This chapter develops eigenanalysis, positive-definite matrices, quadratic forms, eigendecomposition, singular value decomposition, QR factorization, and the Moore–Penrose pseudoinverse. The emphasis is not only on definitions, but also on when a decomposition is appropriate, how numerical stability affects algorithmic choices, and how these methods support compression, regression, dimensionality reduction, inverse problems, and optimization.

Learning objectives

After studying this chapter, the reader should be able to:

1. Interpret eigenvalues and eigenvectors geometrically and computationally.

2. Distinguish positive-definite, positive-semidefinite, indefinite, and singular matrices.

3. Use quadratic forms to analyse curvature, energy, and optimization objectives.

4. Explain the conditions under which eigendecomposition exists and is useful.

5. Derive and interpret the singular value decomposition of a matrix.

6. Compare SVD, eigendecomposition, and QR factorization in machine-learning workflows.

7. Use pseudoinverses to solve underdetermined, overdetermined, and rank-deficient systems.

8. Evaluate numerical stability, conditioning, computational cost, and regularization choices.

9. Apply matrix decompositions to compression, principal component analysis, regression, and spectral clustering.

1. Eigenvalues as directions of intrinsic action

Let $A$ be an $n\times n$ matrix. A nonzero vector $v$ is an eigenvector of $A$ if

$$Av=\lambda v,$$

where $\lambda$ is the corresponding eigenvalue. The equation states that applying $A$ to $v$ changes its magnitude and possibly reverses its direction, but does not rotate it into a different direction. The scalar $\lambda$ is therefore a measure of the action of the transformation along a special direction.

To find eigenvalues, rearrange the equation as $(A-\lambda I)v=0$. A nonzero solution exists only when $A-\lambda I$ is singular:

$$\det(A-\lambda I)=0.$$

This equation is the characteristic equation. Its roots are the eigenvalues, and solving $(A-\lambda I)v=0$ for each root gives the associated eigenspaces.

For a two-dimensional example,

$$A=\begin{bmatrix}3&1\\0&2\end{bmatrix}.$$

The characteristic polynomial is $(3-\lambda)(2-\lambda)$, so the eigenvalues are $3$ and $2$. The eigenvector associated with $3$ is proportional to $(1,0)^T$, while an eigenvector associated with $2$ is proportional to $(-1,1)^T$. These directions are not necessarily perpendicular because $A$ is not symmetric.

Eigenvalues provide global information about a matrix. Their sum equals the trace, and their product equals the determinant, counting algebraic multiplicity. In dynamical systems, eigenvalues can indicate growth, decay, or oscillation. In iterative optimization, they describe curvature and influence convergence rates. In graph methods, eigenvectors of a graph Laplacian encode connectivity and useful partitions.

The spectral radius is

$$\rho(A)=\max_i|\lambda_i|.$$

For repeated application of a matrix, as in $A^kx$, eigenvalues with the largest magnitude tend to dominate, provided the matrix has an appropriate eigenstructure. This explains why power iteration can estimate a dominant eigenvector and why certain recurrent computations may become unstable when eigenvalues have magnitude greater than one.

2. Symmetry, orthogonality, and the spectral theorem

A real matrix is symmetric if $A=A^T$. Symmetric matrices have particularly strong properties:

• all eigenvalues are real;
• eigenvectors associated with distinct eigenvalues are orthogonal;
• there exists an orthogonal matrix $Q$ such that

$$A=Q\Lambda Q^T,$$

where $\Lambda$ is diagonal and contains the eigenvalues.

This result is the real spectral theorem. Since $Q^TQ=I$, the transformation can be understood as a rotation or reflection into an eigenvector coordinate system, followed by independent scaling by the eigenvalues, followed by a return to the original coordinates.

For symmetric matrices, eigendecomposition is not merely an algebraic convenience. It gives a coordinate system in which many operations become simple. For example,

$$A^k=Q\Lambda^kQ^T,$$

and, when the relevant function is defined on the eigenvalues,

$$f(A)=Qf(\Lambda)Q^T.$$

Thus a matrix square root, exponential, or inverse can be computed spectrally. In machine learning, covariance matrices are symmetric and positive semidefinite, making their eigenvectors natural principal directions.

A matrix is normal when $A^TA=AA^T$. Complex normal matrices can also be unitarily diagonalized, although their eigenvalues need not be real. Symmetry is the most important real-valued special case in standard machine-learning applications.

Not every matrix is diagonalizable. A matrix may lack enough linearly independent eigenvectors, as occurs with a Jordan block such as

$$\begin{bmatrix}1&1\\0&1\end{bmatrix}.$$

Although its only eigenvalue is $1$, it has only one independent eigenvector. In practical numerical work, SVD and Schur decomposition are often preferred when diagonalizability is uncertain or when stability is important.

3. Quadratic forms and definiteness

Given a symmetric matrix $A$ and a vector $x$, the expression

$$q(x)=x^TAx$$

is called a quadratic form. It maps a vector to a scalar and appears throughout machine learning. Least-squares loss, Mahalanobis distance, Gaussian log-likelihoods, energy functions, and second-order optimization models all contain quadratic forms.

A symmetric matrix is:

• positive definite (PD) if $x^TAx>0$ for every nonzero $x$;
• positive semidefinite (PSD) if $x^TAx\geq0$ for every $x$;
• negative definite if $x^TAx<0$ for every nonzero $x$;
• indefinite if the quadratic form takes both positive and negative values.

For a symmetric matrix, definiteness is determined by its eigenvalues. The matrix is PD exactly when every eigenvalue is positive; it is PSD exactly when every eigenvalue is nonnegative. A PSD matrix can be singular because one or more eigenvalues may be zero.

The geometry is informative. In two dimensions, $x^TAx=c$ may describe an ellipse when $A$ is PD, a degenerate ellipse when it is PSD and singular, or a hyperbola when it is indefinite. The eigenvectors give the principal axes, and the eigenvalues determine curvature or inverse squared axis lengths, depending on the representation.

Several tests are useful. For a symmetric matrix, Cholesky factorization $A=LL^T$ exists without pivoting precisely when $A$ is PD. Sylvester’s criterion states that a symmetric matrix is PD if and only if all leading principal minors are positive. In computation, however, eigenvalue checks or Cholesky factorization with suitable safeguards are often more informative than symbolic determinant calculations.

The Hessian matrix of a twice-differentiable function determines local curvature. If the Hessian is PD at a stationary point, the point is a strict local minimum; if it is negative definite, it is a strict local maximum; if it is indefinite, it is a saddle point. A poorly conditioned Hessian has eigenvalues that differ greatly in magnitude, producing narrow valleys that slow gradient-based optimization.

4. Eigendecomposition and its uses

If an $n\times n$ matrix has $n$ linearly independent eigenvectors, they can be collected as columns of $V$, giving

$$A=V\Lambda V^{-1}.$$

When $A$ is symmetric, $V$ can be chosen orthogonal, so the expression becomes $A=Q\Lambda Q^T$. The symmetric case is numerically and conceptually preferable because orthogonal transformations preserve Euclidean lengths and do not amplify rounding errors as arbitrary eigenvector matrices can.

Eigendecomposition is useful for covariance analysis. If the centered data matrix is $X$, the sample covariance matrix is often written as

$$C=\frac{1}{m-1}X^TX.$$

The eigenvectors of $C$ are directions in feature space, while the corresponding eigenvalues are variances along those directions. Ordering eigenvalues from largest to smallest identifies directions that explain the greatest variance. Selecting the first $k$ eigenvectors produces a lower-dimensional representation.

The fraction of variance explained by the first $k$ components is

$$\frac{\sum_{i=1}^{k}\lambda_i}{\sum_{i=1}^{r}\lambda_i},$$

where $r$ is the rank of the covariance matrix. This provides a principled, though not task-specific, criterion for choosing a reduced dimension.

Eigendecomposition also supports graph spectral methods. For a graph adjacency matrix $W$, define the degree matrix $D$ by $D_{ii}=\sum_jW_{ij}$. The unnormalized graph Laplacian is $L=D-W$. It is PSD, and the multiplicity of its zero eigenvalue equals the number of connected components. The eigenvectors associated with the smallest eigenvalues can be used to embed nodes before clustering, forming the basis of spectral clustering.

A caution is essential: eigenvectors are not always uniquely determined. If an eigenvalue is repeated, any orthonormal basis of its eigenspace is valid. Even with distinct eigenvalues, eigenvectors are defined only up to sign. Consequently, interpretations should focus on subspaces and reconstructed results rather than arbitrary signs or individual coordinates.

5. Singular value decomposition: a universal factorization

Every real $m\times n$ matrix $X$ has a singular value decomposition

$$X=U\Sigma V^T,$$

where $U$ and $V$ are orthogonal (or have orthonormal columns in the reduced form), and $\Sigma$ is diagonal or rectangular diagonal with nonnegative entries

$$\sigma_1\geq\sigma_2\geq\cdots\geq\sigma_r>0.$$

The $\sigma_i$ are singular values. They are the square roots of the nonzero eigenvalues of $X^TX$ and $XX^T$. The columns of $V$ are right singular vectors, and the columns of $U$ are left singular vectors. The relation

$$Xv_i=\sigma_i u_i,\qquad X^Tu_i=\sigma_i v_i$$

shows how the matrix maps one orthonormal direction into another while scaling it by $\sigma_i$.

Geometrically, the unit sphere is transformed into an ellipsoid. The right singular vectors identify input directions, the singular values give the lengths of the principal semiaxes, and the left singular vectors identify output directions. This interpretation remains valid for rectangular, rank-deficient, and nonsymmetric matrices.

The rank of $X$ is the number of nonzero singular values. The operator or spectral norm is $\|X\|_2=\sigma_1$, while the Frobenius norm satisfies

$$\|X\|_F^2=\sum_i\sigma_i^2.$$

The best rank-$k$ approximation in the Frobenius norm and spectral norm is obtained by truncating the SVD:

$$X_k=\sum_{i=1}^{k}\sigma_i u_iv_i^T.$$

This is the Eckart–Young theorem. It provides the mathematical foundation for low-rank compression, denoising, latent semantic analysis, and approximate recommendation systems. A sharp drop in singular values suggests that much of the matrix can be represented using fewer dimensions, although the choice of $k$ should also consider validation performance, storage, and computational cost.

SVD is closely related to PCA. Applying SVD directly to a centered data matrix $X$ gives principal directions in the columns of $V$. The covariance eigenvalues are $\sigma_i^2/(m-1)$. Computing PCA through SVD is often more stable than first forming $X^TX$, because explicitly forming the covariance matrix squares the condition number and can lose numerical accuracy.

6. QR factorization and stable least squares

For a matrix $A\in\mathbb{R}^{m\times n}$ with linearly independent columns, QR factorization writes

$$A=QR,$$

where $Q$ has orthonormal columns and $R$ is upper triangular. In the square case, $Q$ is orthogonal. The factorization separates the geometric orientation of the columns from their triangular coordinate representation.

One way to obtain QR is Gram–Schmidt orthogonalization, but classical Gram–Schmidt can be numerically unstable when columns are nearly dependent. Modified Gram–Schmidt is better, while Householder reflections are generally preferred in high-quality numerical libraries. Givens rotations are useful when sparsity or incremental updates matter.

QR is central to least squares. For an overdetermined system $Ax\approx b$, minimizing $\|Ax-b\|_2^2$ and substituting $A=QR$ gives

$$\min_x\|QRx-b\|_2^2.$$

Since $Q$ preserves lengths, the solution satisfies

$$Rx=Q^Tb,$$

which is solved by back substitution. This avoids forming the normal equations

$$A^TAx=A^Tb,$$

whose condition number is approximately the square of that of $A$. Consequently, QR is usually more stable than normal-equation methods. SVD is even more informative when rank deficiency or severe ill-conditioning is present.

7. Pseudoinverses and inverse problems

An ordinary inverse exists only for a square, nonsingular matrix. The Moore–Penrose pseudoinverse, denoted $A^+$, generalizes inversion to rectangular and rank-deficient matrices. If

$$A=U\Sigma V^T,$$

then

$$A^+=V\Sigma^+U^T,$$

where $\Sigma^+$ is obtained by replacing each nonzero singular value $\sigma_i$ with $1/\sigma_i$ and transposing the rectangular arrangement.

For a consistent overdetermined system, $x=A^+b$ gives the minimum-norm least-squares solution. For an underdetermined system with infinitely many exact solutions, it gives the solution with smallest Euclidean norm. If $A$ has full column rank, $A^+=(A^TA)^{-1}A^T$; if it has full row rank, $A^+=A^T(AA^T)^{-1}$. These formulas are mathematically useful but should not automatically be implemented by explicitly computing inverses.

Pseudoinverses can be unstable when very small singular values are inverted. A common remedy is truncated SVD: singular values below a threshold are treated as zero. Another is ridge regularization, which replaces the inverse factor $1/\sigma_i$ with

$$\frac{\sigma_i}{\sigma_i^2+\lambda}.$$

This shrinks unstable directions rather than discarding them entirely. Regularization expresses a bias–variance trade-off: it increases bias but can substantially reduce variance and sensitivity to noise.

8. Conditioning, stability, and computational judgment

The condition number of a full-rank matrix in the 2-norm is

$$\kappa_2(A)=\frac{\sigma_{\max}(A)}{\sigma_{\min}(A)}.$$

A large condition number means that small perturbations in data or rounding may cause large changes in a solution. Ill-conditioning is not the same as algorithmic instability: the problem itself may be sensitive, while a stable algorithm avoids adding unnecessary error. Good numerical practice therefore requires both condition assessment and an appropriate factorization.

Typical choices include the following. Symmetric eigenproblems should use symmetric eigensolvers. Least-squares problems generally favour QR; SVD is appropriate when rank or conditioning must be diagnosed. Cholesky is efficient for known PD matrices. Sparse iterative methods are useful when matrices are large and only a few extreme eigenpairs or singular vectors are required. Power iteration is simple but slow when the leading eigenvalues are close; Lanczos and randomized SVD can be more effective for large-scale applications.

Never confuse a mathematically valid formula with a numerically sound implementation. Explicitly forming $A^{-1}$, $A^TA$, or a dense covariance matrix may be unnecessary or harmful. Solving triangular systems, using orthogonal transformations, scaling features, and monitoring singular values are generally better practices.

9. Applications in machine learning and data systems

Matrix decompositions support several important workflows:

Dimensionality reduction. PCA uses dominant eigenvectors or right singular vectors to replace a high-dimensional representation with a compact one. This can reduce storage, accelerate downstream models, and remove low-variance noise, but variance preservation does not guarantee preservation of predictive information.

Regression. QR and SVD provide stable solutions for linear and multivariate regression. SVD exposes near-collinearity among predictors, while ridge regression controls amplification caused by small singular values.

Compression. A large weight matrix or image can be approximated by a low-rank product. Instead of storing all $mn$ entries, a rank-$k$ representation stores approximately $k(m+n+1)$ values. The approximation must be evaluated against both reconstruction error and task accuracy.

Recommendation and language analysis. User–item matrices and term–document matrices often contain latent structure. Truncated SVD identifies latent factors, although sparse data, missing-not-at-random observations, and changing user behaviour require methods beyond naïve factorization.

Graph learning. Laplacian eigenvectors provide embeddings that reflect connectivity. Modern graph neural networks may use related spectral ideas, although practical architectures often implement localized or spatial approximations rather than a full eigendecomposition.

Optimization. Hessian eigenvalues describe local curvature. Preconditioning attempts to transform the problem so that curvature is more balanced, improving convergence of iterative solvers.

10. An illustrative case study: stabilizing a sensor-prediction model

Background

A manufacturing organization collects hundreds of correlated sensor measurements from industrial equipment. The goal is to predict a continuous quality measure and identify abnormal operating conditions. Measurements are on different scales, several sensors are redundant, and some records contain noise.

Situation and problem

A linear regression model fitted using normal equations produces very large coefficients. Small changes in the training sample lead to markedly different estimates, and predictions on new operating conditions are unstable. A preliminary analysis shows that several sensors are strongly correlated.

Analysis

After centering and scaling the features, the data matrix $X$ is examined through SVD. The singular values decline rapidly after the first twelve components, while the remaining values are close to zero. The condition number is consequently large. This indicates that the feature matrix contains directions that are nearly redundant. In those directions, ordinary least squares attempts to explain small noise differences using large opposing coefficients.

Approach

The team compares three approaches. First, QR factorization is used to compute an ordinary least-squares solution without forming $X^TX$. Second, truncated SVD retains the dominant twelve singular directions. Third, ridge regression is fitted over a validation-selected range of regularization parameters. The reduced representation is also used to visualize operating regimes, while residual monitoring is retained to detect failures not represented in the training data.

Outcome

The factorized approaches produce stable estimates. The truncated SVD model is compact and interpretable at the level of latent sensor patterns. Ridge regression retains all original features but shrinks unstable coefficients. The final selection is based not only on reconstruction error but also on held-out prediction accuracy, calibration, maintenance costs, and robustness under sensor drift.

Learning points

The case illustrates that decomposition is a diagnostic tool as well as a computational technique. Singular values reveal identifiability problems; QR improves least-squares stability; regularization controls sensitivity; and PCA-like compression must be validated against the actual prediction objective. A mathematically optimal low-rank reconstruction is not automatically the best operational model.

11. Limitations, risks, and professional considerations

Decompositions are not interpretation-free. Principal components maximize variance, not causal relevance. Eigenvectors can change substantially when eigenvalues are close, so claims about a uniquely meaningful direction may be unjustified. Sign ambiguity and rotations within repeated eigenspaces should be acknowledged in reporting.

Data preprocessing affects results. Failure to center variables changes PCA, while failure to scale features causes high-unit variables to dominate. Missing values, outliers, categorical encodings, and sampling bias can all distort covariance structure. Low-rank approximation may remove rare but safety-critical events.

From a governance perspective, compression should not erase information needed for auditing, fairness analysis, or safety monitoring. If a model uses a pseudoinverse or regularization, the choice of threshold or penalty should be documented. Reproducible pipelines should record scaling parameters, retained rank, solver tolerances, random seeds for randomized methods, and the version of numerical libraries used.

Privacy risks may also arise. A compressed representation can still preserve sensitive information, and reconstruction attacks may be possible when sufficient factors or side information are available. Consequently, matrix factorization is not a substitute for access control, anonymization assessment, or formal privacy techniques.

12. Emerging computational directions and best practice

Large-scale machine learning increasingly relies on approximate decompositions. Randomized SVD uses carefully designed random projections to estimate dominant subspaces efficiently. Streaming and incremental PCA update components as data arrive, avoiding storage of the entire data matrix. Sparse eigensolvers compute only selected spectral components. GPU and distributed libraries exploit blocked QR, SVD, and eigensolvers for large arrays.

A sound workflow is to begin with the mathematical structure of the problem. Determine whether the matrix is symmetric, sparse, rectangular, PSD, or rank-deficient. Scale and center data when the application requires it. Choose a factorization rather than an explicit inverse. Inspect singular values or eigenvalues, assess conditioning, and validate the retained subspace on the downstream task. Finally, test sensitivity to perturbations, preprocessing decisions, and solver tolerances.

Chapter Summary

Eigenvalues and eigenvectors identify invariant directions and scaling factors of a linear transformation. Symmetric matrices admit orthogonal eigendecomposition, and their eigenvalues determine definiteness and curvature in quadratic forms. Positive-definite matrices are central to optimization, covariance modelling, and Cholesky factorization.

SVD provides a universal orthogonal factorization for rectangular and rank-deficient matrices. Its singular values quantify rank, norm, compression quality, and conditioning. Truncated SVD gives the best rank-constrained approximation and underlies PCA and many latent-factor methods. QR factorization solves least-squares problems stably without forming normal equations, while the pseudoinverse generalizes inversion and must be handled carefully near small singular values. Across applications, numerical stability, regularization, validation, and responsible interpretation are as important as algebraic correctness.

Key Terms

Eigenvalue: A scalar $\lambda$ for which $Av=\lambda v$ for a nonzero vector $v$.

Eigenvector: A nonzero vector whose direction is unchanged by a matrix transformation.

Eigenspace: The set of eigenvectors associated with an eigenvalue, together with the zero vector.

Spectral radius: The largest absolute value of a matrix’s eigenvalues.

Symmetric matrix: A real matrix satisfying $A=A^T$.

Orthogonal matrix: A matrix $Q$ satisfying $Q^TQ=I$.

Quadratic form: A scalar expression of the form $x^TAx$.

Positive-definite matrix: A symmetric matrix for which $x^TAx>0$ for every nonzero $x$.

Positive-semidefinite matrix: A symmetric matrix for which $x^TAx\geq0$ for every vector $x$.

Eigendecomposition: Representation of a diagonalizable matrix as $V\Lambda V^{-1}$.

Singular value: A nonnegative square root of a nonzero eigenvalue of $A^TA$.

Singular value decomposition: Factorization $A=U\Sigma V^T$ using orthogonal factors.

Rank: The dimension of a matrix’s column space, equal to its number of nonzero singular values.

QR factorization: Representation $A=QR$ with orthonormal $Q$ and upper-triangular $R$.

Pseudoinverse: The Moore–Penrose generalized inverse of a possibly rectangular or singular matrix.

Condition number: A measure of sensitivity, commonly $\sigma_{\max}/\sigma_{\min}$ in the 2-norm.

Truncated SVD: An approximation formed by retaining only the largest singular values and vectors.

Regularization: A method that controls instability by imposing a penalty or shrinkage on solutions.

Principal component analysis: A variance-based dimensionality-reduction method using covariance eigenvectors or data SVD.

Review Questions

Short Answer Questions

1. What does an eigenvector reveal about the action of a matrix?

2. State the spectral theorem for real symmetric matrices.

3. How are positive definiteness and eigenvalues related?

4. What is the geometric meaning of a singular value?

5. Why is QR factorization preferred to normal equations in many least-squares problems?

Long Answer Questions

1. Explain eigendecomposition for symmetric matrices and discuss its role in PCA.

2. Derive the relationship between SVD and the eigenvalues of $A^TA$.

3. Compare QR factorization, SVD, and normal equations for solving least-squares problems.

4. Explain quadratic forms, definiteness, and their connection with Hessian-based optimization.

5. Describe the Moore–Penrose pseudoinverse, including its use in overdetermined and underdetermined systems.

Analytical / Application Questions

1. A data matrix has singular values $20, 5, 1, 0.01$. Analyse its conditioning and explain whether truncated SVD or ridge regression may be appropriate.

2. A covariance matrix has one zero eigenvalue. Determine what this implies about the data and discuss the consequences for inversion and PCA.

3. A company compresses a weight matrix using rank-$k$ SVD and observes lower reconstruction error but worse classification accuracy. Analyse possible reasons and recommend an evaluation strategy.

22. Multiple Choice Questions with Answers

Question 1

For a real symmetric matrix, which statement is guaranteed to be true?

A. All eigenvalues are real and an orthonormal eigenbasis exists

B. All eigenvalues are positive

C. The matrix must be invertible

D. Every eigenvalue has exactly one eigenvector

Correct Answer: A. All eigenvalues are real and an orthonormal eigenbasis exists

Question 2

A symmetric matrix is positive definite if and only if:

A. At least one eigenvalue is positive

B. All eigenvalues are nonnegative

C. All eigenvalues are strictly positive

D. Its determinant is zero

Correct Answer: C. All eigenvalues are strictly positive

Question 3

In the SVD X = UΣVᵀ, the columns of V primarily represent:

A. Output directions after transformation

B. Input directions associated with singular values

C. Rows of the covariance matrix

D. Residual error directions only

Correct Answer: B. Input directions associated with singular values

Question 4

Why can forming AᵀA be numerically undesirable in least-squares computation?

A. It always changes the exact least-squares solution

B. It squares the condition number and may amplify numerical error

C. It makes A rectangular

D. It prevents QR factorization

Correct Answer: B. It squares the condition number and may amplify numerical error

Question 5

The best rank-k approximation of a matrix in the spectral and Frobenius norms is obtained by:

A. Retaining the k smallest singular values

B. Setting all singular values equal

C. Retaining the k largest singular values and corresponding vectors

D. Using only the diagonal entries of the matrix

Correct Answer: C. Retaining the k largest singular values and corresponding vectors

Question 6

If a matrix has a very large 2-norm condition number, its solutions are likely to be:

A. Insensitive to perturbations

B. Highly sensitive to small data or rounding perturbations

C. Guaranteed to have small norm

D. Independent of the smallest singular value

Correct Answer: B. Highly sensitive to small data or rounding perturbations

Question 7

For an underdetermined consistent system, the pseudoinverse solution provides the:

A. Maximum-residual solution

B. Solution with the largest Euclidean norm

C. Minimum-norm exact solution

D. Solution requiring the matrix to be square

Correct Answer: C. Minimum-norm exact solution

Question 8

In PCA based on a centered data matrix, the eigenvectors of the covariance matrix correspond to:

A. Principal directions in feature space

B. Individual observations only

C. Regression residuals

D. The inverse singular values

Correct Answer: A. Principal directions in feature space

Question 9

Which factorization is especially appropriate for a known symmetric positive-definite matrix?

A. Cholesky factorization

B. Jordan factorization only

C. Pseudoinverse factorization only

D. A factorization requiring the matrix to be indefinite

Correct Answer: A. Cholesky factorization

Question 10

A small singular value becomes problematic in a pseudoinverse because it is:

A. Multiplied by zero automatically

B. Replaced by a large reciprocal that can amplify noise

C. Always associated with a positive-definite direction

D. Unable to affect the solution

Correct Answer: B. Replaced by a large reciprocal that can amplify noise