Linear Algebra and Matrix Methods

Introduction and context

Linear algebra provides the language for describing multivariable systems. In electronics and communication engineering, voltages and currents are related by circuit equations, sampled signals are represented as vectors, transformations are implemented by matrices, and measured data are estimated through least squares. The same ideas support network analysis, digital filtering, image compression, antenna arrays, control, coding, and machine learning.

A matrix method is valuable not merely because it shortens arithmetic. It exposes structure: existence and uniqueness of solutions, dependence among measurements, dominant signal modes, numerical sensitivity, and the geometry of approximation.

Learning objectives

After completing this chapter, a student should be able to:

1. Formulate linear systems in matrix form and solve them by elimination.
2. Interpret rank, null space, column space, basis, and dimension.
3. Use inner products, orthogonality, projections, and Gram–Schmidt orthogonalization.
4. Compute and interpret eigenvalues and eigenvectors.
5. Diagonalize suitable matrices and identify when diagonalization fails.
6. Analyze quadratic forms and classify conics or energy expressions.
7. Construct least-squares estimates and understand residual orthogonality.
8. Explain the singular value decomposition (SVD), calculate a small example, and relate it to signal and data applications.

1. Vectors, matrices, and linear systems

A vector is an ordered list of numbers. A column vector with n components belongs to ℝⁿ (or ℂⁿ for complex signals). A matrix A with m rows and n columns is written A ∈ ℝᵐˣⁿ. Its entry in row i and column j is aᵢⱼ.

Matrix addition is defined only for matrices of the same size. Matrix multiplication AB is defined when the number of columns of A equals the number of rows of B. If A ∈ ℝᵐˣⁿ and x ∈ ℝⁿ, then Ax ∈ ℝᵐ and

The product is a linear combination of the columns of A. A linear system of m equations in n unknowns is therefore

where x is the unknown vector and b is the known excitation or measurement vector.

For a square matrix, the determinant indicates invertibility: A⁻¹ exists exactly when det(A) ≠ 0. Although x = A⁻¹b is theoretically correct in this case, explicitly computing A⁻¹ is usually less efficient and less numerically stable than elimination or factorization.

Elementary row operations and Gaussian elimination

The following operations preserve the solution set of a system:

• interchange two rows;
• multiply a row by a nonzero scalar;
• add a multiple of one row to another row.

Gaussian elimination converts the augmented matrix [A | b] to upper triangular or row-echelon form. Back substitution then determines the unknowns. Partial pivoting exchanges rows so that a suitably large pivot is used; this reduces round-off error when calculations are performed numerically.

Worked example 1: solving a circuit-like system

Suppose node equations produce

Thus

Replace the second row by 2 times the second row minus the first row:

Hence i₂ = 7/5 = 1.4 and i₁ = (5 − 1.4)/2 = 1.8. Substitution verifies both original equations. If the unknowns represent currents, their units must be amperes and the coefficients must be dimensionally consistent with the chosen circuit formulation.

2. Rank, consistency, and vector spaces

The column space Col(A) is the set of all linear combinations of the columns of A. The null space N(A) is the set of vectors x satisfying Ax = 0. The rank of A is the dimension of its column space and also the number of pivot columns in its row-reduced form.

For an m × n system Ax = b, the system is consistent exactly when

If this common rank is n, the solution is unique. If it is less than n, infinitely many solutions exist. If the augmented rank is larger than rank(A), no solution exists.

The rank–nullity theorem states

A basis is a linearly independent spanning set. The number of vectors in any basis of a finite-dimensional vector space is its dimension. A set is linearly dependent if a nontrivial linear combination of its vectors is zero.

Homogeneous and nonhomogeneous systems

The homogeneous system Ax = 0 always has the zero solution. Nonzero solutions exist when the columns of A are linearly dependent, equivalently when rank(A) < n. For a consistent nonhomogeneous system, the complete solution can be written as one particular solution plus every vector in N(A).

Engineering interpretation

Rank measures independent information. In a sensor system, dependent rows indicate redundant measurements; dependent columns indicate parameters that cannot be distinguished from the available data. Rank deficiency is therefore a modeling and identifiability issue, not merely an arithmetic inconvenience.

3. Inner products, orthogonality, and projection

For real vectors u and v, the Euclidean inner product is

The norm is ||u|| = √(uᵀu). Vectors are orthogonal if uᵀv = 0. For complex vectors, the appropriate inner product is uᴴv, where ᴴ denotes conjugate transpose.

The orthogonal projection of b onto a nonzero vector a is

For a matrix Q whose columns are orthonormal, the projection onto Col(Q) is QQᵀb. More generally, if A has independent columns, the projection onto Col(A) is

provided AᵀA is invertible. P_A is symmetric and idempotent: P_Aᵀ = P_A and P_A² = P_A.

Gram–Schmidt process

Given independent vectors v₁, …, vₖ, construct orthogonal vectors by

Normalize each nonzero uⱼ to obtain qⱼ = uⱼ/||uⱼ||. Then Q = [q₁ … qₖ] has orthonormal columns. In numerical software, modified Gram–Schmidt or Householder QR is generally preferred because it is more stable.

4. Eigenvalues, eigenvectors, and diagonalization

A nonzero vector v is an eigenvector of a square matrix A if

where λ is the corresponding eigenvalue. Rearranging gives

A nonzero solution exists precisely when

This determinant equation is the characteristic equation. Eigenvalues describe directions whose orientation is unchanged by A; λ gives the scaling, including reversal when λ is negative.

Worked example 2: eigenvalues and eigenvectors

Let

The characteristic polynomial is

Thus λ₁ = 4 and λ₂ = 2. For λ₁ = 4,

so v₁ = [1, 1]ᵀ is an eigenvector. For λ₂ = 2, v₂ = [1, −1]ᵀ is an eigenvector. Normalizing gives q₁ = [1,1]ᵀ/√2 and q₂ = [1,−1]ᵀ/√2. Therefore

where Q = [q₁ q₂] and Λ = diag(4,2). In this symmetric case, the eigenvectors are orthonormal and diagonalization is orthogonal.

Diagonalization and powers

A matrix is diagonalizable if it has a basis of eigenvectors. If V is the matrix of independent eigenvectors and Λ contains their eigenvalues, then

Consequently, Aᵏ = VΛᵏV⁻¹ for nonnegative integer k. A matrix with n distinct eigenvalues is diagonalizable, but repeated eigenvalues require further inspection. A defective matrix, such as [[1,1],[0,1]], has too few independent eigenvectors and cannot be diagonalized.

For real symmetric matrices, the spectral theorem guarantees an orthogonal diagonalization A = QΛQᵀ. This is especially important in energy analysis and quadratic forms.

5. Quadratic forms

A quadratic form is

where A is usually taken symmetric because the skew-symmetric part contributes zero: xᵀKx = 0 when Kᵀ = −K. In two variables,

corresponds to A = [[a,b],[b,c]].

For a symmetric matrix, q is positive definite if xᵀAx > 0 for every nonzero x, positive semidefinite if q(x) ≥ 0, negative definite if q(x) < 0, and indefinite if it takes both positive and negative values. Positive definiteness is equivalent to all eigenvalues being positive; for a 2 × 2 symmetric matrix it is also equivalent to a > 0 and ac − b² > 0.

Worked example 3: classifying and evaluating a quadratic form

Consider

so A = [[2,1],[1,2]]. Its eigenvalues are 3 and 1, both positive; hence q is positive definite. At x = [1,−2]ᵀ,

The level set q(x,y)=1 is an ellipse. The eigenvectors [1,1]ᵀ and [1,−1]ᵀ give its principal directions, and the eigenvalues determine curvature along those directions. In engineering, positive-definite quadratic forms model stored energy, weighted error, and power-like measures; the matrix must match the physical units of the variables.

6. Least squares and overdetermined systems

Measurements often produce more equations than unknown parameters. If Ax = b has no exact solution, choose x̂ to minimize the squared residual

Differentiation gives the normal equations

If A has independent columns, AᵀA is invertible and

The residual r = b − A x̂ is orthogonal to every column of A: Aᵀr = 0. In numerical practice, QR or SVD is preferred to explicitly forming AᵀA, because forming it can worsen the condition number and amplify round-off effects.

Applied case: calibration of a two-parameter sensor array

Background. Three sensors measure an approximately linear response bᵢ = α + βtᵢ, where t is a known normalized position or time coordinate. The measured data are

Problem. Estimate the offset α and slope β from all measurements.

Analysis. Here

The determinant of AᵀA is 15−9 = 6, so the parameters are identifiable. Its inverse is (1/6)[[5,−3],[−3,3]]. Thus

The fitted predictions are

The residual is approximately [−0.005, 0.010, −0.005]ᵀ, whose entries sum to zero and whose weighted first moment is zero, confirming Aᵀr = 0.

Outcome and learning points. The estimate uses all observations rather than selecting two and discarding the third. The intercept and slope have the units of the sensor output and output per unit t, respectively. Calibration should also report uncertainty, inspect residuals for nonlinearity, and avoid extrapolation beyond the measured range. If measurement noise has unequal variance, weighted least squares with a positive-definite weight matrix W minimizes (Ax−b)ᵀW(Ax−b) and gives AᵀWAx̂ = AᵀWb.

7. Singular value decomposition

Every real m × n matrix A has an SVD

where U and V are orthogonal (with appropriate rectangular dimensions) and Σ contains nonnegative singular values σ₁ ≥ σ₂ ≥ … ≥ 0. For complex matrices, transpose is replaced by conjugate transpose. The right singular vectors are eigenvectors of AᵀA, and the singular values satisfy σᵢ = √λᵢ(AᵀA).

The SVD separates a transformation into three actions: Vᵀ rotates or changes coordinates, Σ scales orthogonal directions, and U rotates the result. The rank equals the number of nonzero singular values. The condition number for a full-rank matrix is κ₂(A) = σ_max/σ_min.

Worked example 4: SVD and low-rank approximation

Let

This matrix is already diagonal, so one valid SVD is U = I, V = I, and Σ = diag(3,1). The singular values are 3 and 1. The direction [1,0]ᵀ is stretched by 3, while [0,1]ᵀ is stretched by 1. The 2-norm condition number is κ₂(A)=3.

The best rank-one approximation in the spectral and Frobenius norms is obtained by retaining the largest singular value:

The discarded singular value is 1, so the spectral-norm error ||A−A₁||₂ is 1, while the Frobenius-norm error is also 1 in this example. This illustrates why SVD is used for compression: it preserves the strongest orthogonal mode first.

For a genuinely rectangular matrix, consider A = [[1,0],[0,1],[1,0]]. Its columns are orthogonal but not unit length. The squared column lengths are 2 and 1, so an SVD can be formed with singular values √2 and 1, right singular vectors equal to the coordinate vectors, and left singular vectors u₁ = [1,0,1]ᵀ/√2 and u₂ = [0,1,0]ᵀ. The remaining left-singular direction is associated with singular value zero. This decomposition reveals that the data occupy only a two-dimensional column space inside ℝ³.

Applications and limitations

SVD supports noise filtering, pseudoinverse solutions, principal-component analysis, beamforming, image compression, and ill-conditioned inverse problems. The Moore–Penrose pseudoinverse is

where each nonzero σᵢ is replaced by 1/σᵢ and zeros remain zero. The minimum-norm least-squares solution is x̂ = A⁺b. Very small singular values may represent weakly observable parameters or noise; truncating them can stabilize an inverse problem, but the threshold must be justified by noise level and application requirements.

8. Computational practice, limitations, and professional responsibility

For small exact systems, elimination is transparent. For large numerical systems, LU, QR, or SVD should be selected according to structure and conditioning. A residual check ||Ax̂−b|| is necessary but not sufficient: a small residual can coexist with large parameter error when A is ill-conditioned. Scaling variables, using pivoting, and reporting significant figures consistent with measurement precision are essential.

In communications and sensing, matrix models can encode personal or safety-critical information. Engineers should document assumptions, preserve data integrity, test sensitivity to noise and model mismatch, and avoid presenting a numerical estimate as certain when the system is rank-deficient or poorly conditioned. Reproducible code should record units, preprocessing, matrix dimensions, tolerances, and the treatment of missing data. AI-based numerical tools can assist computation, but their output requires independent dimensional, residual, and boundary-condition checks.

Chapter Summary

Matrices represent linear relations among circuit variables, signal samples, parameters, and measurements. Gaussian elimination solves systems while rank determines consistency, independence, and identifiability. Vector spaces are organized through span, basis, dimension, column space, and null space; rank–nullity connects the latter two. Inner products define length and orthogonality, and projections provide the geometric basis of approximation. Eigenvalues identify invariant directions and scaling factors; symmetric matrices admit orthogonal diagonalization. Quadratic forms describe energy, error, and geometry and can be classified through eigenvalues. Least squares projects inconsistent measurements onto a model subspace, with residual orthogonal to that subspace. SVD provides an orthogonal scaling description, reveals rank and conditioning, and supports pseudoinverse computation and low-rank approximation. Reliable engineering practice combines these methods with unit checks, conditioning analysis, residual verification, and honest reporting of uncertainty.

Key Terms

Matrix: Rectangular array of numbers representing a linear mapping or data arrangement.

Vector: Ordered list of components representing a point, signal, or set of unknowns.

Linear system: Collection of linear equations written as Ax = b.

Gaussian elimination: Row-operation method for reducing a linear system.

Pivot: Leading nonzero entry used to eliminate other entries in a column.

Rank: Dimension of a matrix’s column space.

Null space: Set of vectors x satisfying Ax = 0.

Basis: Linearly independent set that spans a vector space.

Dimension: Number of vectors in a basis.

Inner product: Operation defining angle and length, such as uᵀv.

Orthogonality: Perpendicularity defined by zero inner product.

Projection: Closest component of a vector in a specified subspace.

Eigenvalue: Scalar λ satisfying Av = λv for a nonzero v.

Eigenvector: Nonzero vector whose direction is preserved by a matrix.

Diagonalization: Representation A = VΛV⁻¹ using an eigenvector basis.

Quadratic form: Expression xᵀAx describing a scalar function of x.

Least squares: Minimization of ||Ax−b||².

Singular value: Nonnegative scale factor in an SVD.

Condition number: Ratio of largest to smallest relevant singular value.

Pseudoinverse: Generalized inverse used for minimum-norm least-squares solutions.

Review Questions

Short Answer Questions

1. What condition determines whether Ax = b is consistent in terms of ranks?
2. State the rank–nullity theorem for A ∈ ℝᵐˣⁿ.
3. Define orthogonality using the Euclidean inner product.
4. What equation determines the eigenvalues of a square matrix?
5. Why is QR or SVD often preferred to explicitly forming (AᵀA)⁻¹ in least squares?

Long Answer Questions

1. Explain Gaussian elimination, including the three elementary row operations, pivoting, and back substitution.
2. Discuss column space, null space, rank, basis, and their significance in parameter identifiability.
3. Derive the normal equations for an unweighted least-squares problem and interpret the residual geometrically.
4. Explain diagonalization and contrast a symmetric matrix with a defective matrix.
5. Describe the SVD and explain how its singular values reveal rank, conditioning, and compression potential.

Analytical / Application Questions

1. For A = [[1,1],[2,2]] and b = [3,6]ᵀ, determine the rank, describe the solution set, and state whether the solution is unique.
2. A measurement matrix has singular values 10, 1, and 0.02. Calculate its 2-norm condition number using the nonzero extreme singular values and explain the implication for inversion.
3. Fit bᵢ = α + βtᵢ to t = [0,1,2]ᵀ and b = [2,3,5]ᵀ using least squares. Form AᵀA and Aᵀb, solve for α and β, and calculate the residual vector.

Multiple Choice Questions with Answers

Question 1

For a consistent system Ax=b with n unknowns, when is the solution unique?

A. rank(A)=rank([A|b])=n

B. rank(A)=0

C. rank([A|b])>rank(A)

D. rank(A)<n and rank([A|b])=n

Correct Answer: A. rank(A)=rank([A|b])=n

Question 2

For A = [[3,1],[1,3]], what are its eigenvalues?

A. 1 and 3

B. 2 and 4

C. −2 and 4

D. 3 and 6

Correct Answer: B. 2 and 4

Question 3

What is the defining condition for two real vectors u and v to be orthogonal?

A. uᵀv=1

B. ||u||=||v||

C. uᵀv=0

D. u=v

Correct Answer: C. uᵀv=0

Question 4

For the quadratic form q(x,y)=2x²+2xy+2y², what is q(1,−2)?

A. 2

B. 4

C. 6

D. 10

Correct Answer: C. 6

Question 5

In ordinary least squares, the residual r=b−A x̂ satisfies which condition?

A. A r=0

B. Aᵀr=0

C. rᵀr=0 for every dataset

D. Aᵀb=0

Correct Answer: B. Aᵀr=0

Question 6

In the sensor calibration example, what is Aᵀb?

A. [4.53, 5.02]ᵀ

B. [4.53, 5.54]ᵀ

C. [5.54, 4.53]ᵀ

D. [3.00, 5.00]ᵀ

Correct Answer: B. [4.53, 5.54]ᵀ

Question 7

For A=diag(3,1), what is its 2-norm condition number?

A. 1/3

B. 2

C. 3

D. 9

Correct Answer: C. 3

Question 8

Which decomposition is the singular value decomposition of a real matrix A?

A. A=UΣVᵀ

B. A=U+Σ+V

C. A=ΛVΛ⁻¹ for every matrix

D. A=QRQᵀ for every matrix

Correct Answer: A. A=UΣVᵀ

Question 9

If a square matrix has n distinct eigenvalues, which statement is guaranteed?

A. It is singular

B. It is diagonalizable

C. All eigenvalues are zero

D. It is symmetric

Correct Answer: B. It is diagonalizable

Question 10

What does a zero singular value indicate about a matrix?

A. The matrix has no entries

B. The matrix is necessarily symmetric

C. There is a nontrivial null-space direction

D. Every vector is an eigenvector

Correct Answer: C. There is a nontrivial null-space direction