Linear Algebra and Vector Spaces

Introduction and context

Linear algebra provides the language for representing structured data and the computational machinery for transforming, comparing, compressing, and modelling it. A customer record can be represented as a vector, a collection of records as a matrix, a recommendation system as a system of linear equations, and a dimensionality-reduction method as a projection onto a lower-dimensional subspace.

In data science, linear algebra is not merely a collection of calculation rules. It supplies concepts—span, independence, basis, rank, orthogonality, eigenvectors, and linear transformations—that explain what information a dataset contains, which variables are redundant, and how algorithms manipulate representations. Numerical linear algebra also determines whether a computation is stable, efficient, and reproducible.

Learning objectives

By the end of this chapter, you should be able to:

• represent observations, features, and transformations using vectors and matrices;
• solve and interpret systems of linear equations;
• explain span, linear independence, basis, dimension, rank, and null space;
• use inner products, norms, orthogonality, and projections;
• distinguish linear maps from affine transformations;
• interpret matrix factorisation and eigenvector-based methods in data analysis;
• formulate a least-squares problem and evaluate its assumptions and limitations;
• recognise numerical, ethical, and governance issues arising from linear representations.

1. Vectors as representations

A vector is an ordered list of numbers. A column vector with n components belongs to the Euclidean vector space ℝⁿ:

The components may represent measurements, features, coefficients, or coordinates. For example, a customer vector might be x = (age, income, visits)ᵀ. The interpretation of each component depends on the application; the algebra alone does not make variables comparable.

Two vectors of the same dimension may be added componentwise and multiplied by a scalar:

Here α is a real scalar. These operations satisfy closure, associativity, distributivity, and the existence of a zero vector and additive inverses. These axioms define a real vector space. Complex vector spaces are also important in signal processing, but this chapter focuses primarily on ℝⁿ.

Inner products, norms, and distance

The standard inner product, or dot product, is

It measures alignment between vectors. The induced Euclidean norm is

and Euclidean distance is d(x,y) = ‖x − y‖₂. The norm is non-negative, equals zero only for the zero vector, is absolutely homogeneous, and obeys the triangle inequality. Two nonzero vectors are orthogonal when xᵀy = 0.

The angle θ between nonzero vectors satisfies

This formula assumes the Euclidean inner product and nonzero vectors. In text mining, cosine similarity often compares document vectors while ignoring their magnitudes.

Other norms are useful. The Manhattan norm is ‖x‖₁ = ∑ᵢ₌₁ⁿ |xᵢ|, and the maximum norm is ‖x‖∞ = maxᵢ |xᵢ|. Choice of norm affects geometry, optimisation, and the meaning of “closeness.”

2. Matrices and linear systems

An m × n matrix A contains m rows and n columns:

A matrix may represent a dataset, a linear transformation, a graph, or coefficients in a system. If rows are observations and columns are features, the matrix X is often called a design or data matrix.

Matrix multiplication is defined when the inner dimensions agree. If A is m × n and B is n × p, then AB is m × p, with entries

Multiplication is associative and distributive, but generally not commutative: AB ≠ BA. The transpose Aᵀ exchanges rows and columns. A square matrix is symmetric if Aᵀ = A.

A linear system can be written as

where A is an m × n coefficient matrix, x is an n-dimensional unknown vector, and b is an m-dimensional response vector. Elementary row operations—swapping rows, multiplying a row by a nonzero scalar, and adding a multiple of one row to another—preserve the solution set when applied consistently to the augmented matrix [A | b]. Gaussian elimination converts the system to row-echelon or reduced row-echelon form.

Worked example: solving a system

Consider

Subtract three times the first equation from the second:

Thus y = 2, and substitution into the first equation gives x = 1. The unique solution is (x,y) = (1,2). The coefficient matrix has two pivots, so its rank is 2 and its columns are linearly independent.

A system may have one solution, no solution, or infinitely many solutions. It is consistent precisely when the rank of A equals the rank of the augmented matrix [A | b]. If this common rank equals the number of unknowns, the solution is unique. If it is smaller, infinitely many solutions exist.

For a square matrix A, an inverse A⁻¹ exists exactly when A is nonsingular, equivalently when its determinant is nonzero, its columns are independent, and its rank equals its size. When it exists, the unique solution is x = A⁻¹b. In numerical practice, directly computing A⁻¹ is usually less stable and less efficient than solving Ax = b by factorisation.

3. Subspaces, span, basis, and dimension

Given vectors v₁, …, vₖ, their span is the set of all linear combinations:

A subspace is a subset closed under vector addition and scalar multiplication. The column space Col(A) is the span of the columns of A; the null space is

The row space is the span of the rows, and the left null space is Null(Aᵀ).

Vectors are linearly independent if

implies α₁ = … = αₖ = 0. Dependence means that at least one vector can be expressed as a combination of the others. A basis is a linearly independent spanning set. Every vector in a finite-dimensional vector space has a unique coordinate representation relative to a chosen basis.

The dimension of a finite-dimensional space is the number of vectors in any basis. For an m × n matrix A, rank–nullity states

where rank(A) is the dimension of the column space and nullity(A) is the dimension of the null space. This theorem is fundamental for understanding information loss: null-space directions are changes to x that produce no change in Ax.

Rank and data redundancy

The rank of a data matrix is the number of independent directions represented by its rows or columns. If a feature is an exact linear combination of other features, the matrix is rank deficient. Near dependence produces ill-conditioning: small changes in data can produce large changes in estimated coefficients. Rank is therefore related to redundancy, identifiability, compression, and model stability.

4. Orthogonality and projection

A set of nonzero vectors is orthonormal when every vector has unit norm and distinct vectors are mutually orthogonal. If Q has orthonormal columns, then

where I is the identity matrix. Orthonormal bases simplify coordinates because the coefficient of qᵢ is qᵢᵀx.

The orthogonal projection of a vector b onto a nonzero vector a is

The denominator is nonzero because a ≠ 0. The residual r = b − projₐ(b) is orthogonal to a. For a subspace with orthonormal basis columns in Q, projection onto the subspace is

The matrix P = QQᵀ is an orthogonal projector: Pᵀ = P and P² = P.

Least squares

When Ax = b has no exact solution, least squares chooses x̂ that minimises the squared residual:

At an optimum, the residual is orthogonal to every column of A. Therefore

and the normal equations are

If A has full column rank, the solution is unique and can be written

The inverse expression is a theoretical formula, not always the preferred computational method. QR factorisation is generally more numerically stable, while singular value decomposition is especially useful for rank-deficient or ill-conditioned problems.

Worked example: fitting a line

Suppose observations are (1, 2), (2, 3), and (3, 5). Fit y ≈ β₀ + β₁x. The design matrix and response are

The normal-equation components are

Solving (AᵀA)β = Aᵀb gives β₁ = 1.5 and β₀ = 1/3. Thus the fitted line is ŷ = 1/3 + 1.5x. Predicted values are 11/6, 10/3, and 29/6. The residuals sum to zero because the model includes an intercept, and the residual is orthogonal to both columns of A.

5. Linear transformations

A function T: ℝⁿ → ℝᵐ is linear if

for all vectors x, y and scalars α, β. Every such transformation can be represented by a matrix A, with T(x) = Ax. The matrix’s columns are the images of the standard basis vectors.

Examples include scaling, rotation, reflection, shearing, and dimensionality reduction. A rotation preserves lengths and angles; a projection removes components orthogonal to a target subspace. Translation, such as T(x) = Ax + c, is affine rather than linear unless c = 0. This distinction matters in modelling and software implementation.

The image or range of T is Col(A), and the kernel is Null(A). The rank measures the dimension of the image. A transformation is injective if its kernel contains only the zero vector. For a square transformation, injectivity, surjectivity, and invertibility are equivalent.

6. Eigenvalues, eigenvectors, and factorisation

A nonzero vector v is an eigenvector of a square matrix A if

where λ is its eigenvalue. Eigenvectors identify directions that a transformation preserves, while λ gives the scaling along those directions. Eigenvalues satisfy the characteristic equation

For a real symmetric matrix, all eigenvalues are real and there exists an orthonormal eigenvector basis. This spectral theorem underlies principal component analysis (PCA). For a covariance matrix, eigenvectors give principal directions and eigenvalues give variance along those directions, subject to the covariance matrix being correctly constructed from appropriately centred data.

The singular value decomposition (SVD) exists for every real m × n matrix:

where U and V have orthonormal columns or form orthogonal square matrices, and Σ contains nonnegative singular values in decreasing order. The rank equals the number of positive singular values. Truncating the SVD gives a best low-rank approximation in the Frobenius norm and supports compression, denoising, latent semantic analysis, and recommender systems.

For centred data matrix X, PCA commonly computes eigenvectors of the sample covariance matrix

where N is the number of observations. If features have very different units or scales, standardisation may be appropriate; however, standardisation changes the covariance structure and should be justified rather than applied automatically.

7. Applied case: reducing sensor data for predictive maintenance

This is an illustrative case. A manufacturer records vibration, temperature, acoustic energy, and motor current from machines. Each observation is a four-dimensional feature vector. The objective is to identify operating regimes and build an early-warning model without storing every correlated measurement.

Background and problem

Several sensors respond to the same underlying mechanical effects. Raw features have different units and variances, and missing or abnormal readings occur. Direct clustering in the raw space may give excessive weight to high-variance temperature measurements.

Analysis

Engineers first define data quality rules, remove impossible readings, document missingness, and separate training, validation, and operational data by time to prevent leakage. Features are centred using training-set means. If scale comparability is required, each feature is divided by its training-set standard deviation. The resulting matrix X is analysed using SVD:

The first two right singular vectors define a two-dimensional feature subspace. Each observation is projected using its coordinates in that subspace. The retained variance ratio is

where σᵢ are singular values. This ratio is descriptive, not a guarantee that the discarded directions are irrelevant for failure prediction.

Approach and outcome

The team compares a predictive model using all features with one using the two-dimensional representation. Performance is evaluated on temporally later machines using calibration, false-alarm rate, missed-failure rate, and operational cost—not variance explained alone. If the reduced model performs similarly and is easier to monitor, it can support deployment. Loadings are inspected with domain experts so that a mathematically dominant component is not incorrectly interpreted as a physical cause.

Learning points

The case demonstrates that linear algebra supports representation and compression, but modelling validity depends on preprocessing, sampling, temporal structure, measurement quality, and domain interpretation. PCA is unsupervised: it maximises variance, not predictive relevance. A low-dimensional representation must therefore be validated against the actual decision objective.

8. Numerical computation and professional practice

Exact algebra and floating-point computation differ. Computers represent most real numbers approximately, and subtraction of nearly equal quantities can cause loss of precision. The condition number of a nonsingular matrix, informally, measures sensitivity of a problem to perturbations. A large condition number indicates that accurate input data may still produce an unstable solution.

Best practices include:

• use QR or SVD rather than explicitly forming an inverse when solving least-squares problems;
• scale variables when units and magnitudes make the problem ill-conditioned;
• inspect singular values and rank tolerances instead of assuming full rank;
• use reproducible preprocessing, especially when centring and scaling data;
• validate transformations on held-out or future data;
• record units, feature definitions, missing-value treatment, and numerical tolerances;
• distinguish mathematical correlation or projection from causal explanation.

Ethical and governance considerations

A vector representation can conceal socially important attributes, measurement error, or unequal data quality. Dimensionality reduction may remove information needed to detect performance differences across demographic groups. Conversely, retaining sensitive features may create privacy or discrimination risks. Data scientists should document feature provenance, assess subgroup performance, test for proxy variables, and apply relevant privacy, data-protection, and sector-specific governance requirements. Linear algebra does not neutralise bias: it can efficiently propagate patterns already present in data.

Challenges and limitations

Linear models are interpretable and computationally efficient, but they impose structural assumptions. A linear transformation cannot represent arbitrary nonlinear relationships. Least squares is sensitive to outliers because squared residuals receive disproportionate weight. Rank can depend on numerical tolerance, and estimated principal directions can be unstable when eigenvalues are close. Sparse or missing data require specialised methods. Finally, a mathematically elegant representation may be operationally unsuitable if stakeholders cannot interpret it or if it obscures accountability.

Chapter Summary

Vectors represent observations, features, and coefficients; matrices organise vectors and encode transformations and systems of equations. Inner products define geometric alignment, norms define magnitude, and orthogonality supports decompositions and projections. Span, independence, basis, and dimension describe the structure of vector spaces. Rank measures independent information, while rank–nullity connects observable output directions with unobservable input directions.

Systems Ax = b are solved through elimination or stable factorisations. When an exact solution is unavailable, least squares projects the response onto the column space of the design matrix. Linear transformations are represented by matrices, whereas translations produce affine maps. Eigenvectors identify invariant directions, and SVD provides a general factorisation for compression, denoising, and latent representations. In professional data science, these tools must be combined with scaling discipline, numerical stability checks, validation, domain expertise, and responsible governance.

Key Terms

Vector: An ordered collection of components representing a mathematical or data object.

Scalar: A single number used to scale vectors or matrices.

Inner product: A product such as xᵀy that measures geometric alignment.

Norm: A function measuring vector magnitude.

Orthogonality: A relation in which the inner product of two vectors is zero.

Matrix: A rectangular array of numbers or symbols.

Transpose: The operation that exchanges rows and columns.

Linear system: A set of linear equations, commonly represented as Ax = b.

Span: The set of all linear combinations of specified vectors.

Subspace: A subset closed under vector addition and scalar multiplication.

Linear independence: The property that only the trivial combination equals zero.

Basis: A linearly independent spanning set.

Dimension: The number of vectors in a basis.

Rank: The dimension of a matrix’s column space.

Null space: The set of vectors x satisfying Ax = 0.

Projection: The component of a vector lying in a specified subspace.

Least squares: Optimisation that minimises squared residual length.

Eigenvector: A nonzero vector whose direction is preserved by a transformation.

Singular value decomposition: A factorisation A = UΣVᵀ using orthogonal factors.

Conditioning: The sensitivity of a computational problem to perturbations.

Review Questions

Short Answer Questions

1. What is the difference between a vector space and a subspace?
2. State the rank–nullity theorem and explain the meaning of nullity.
3. Why is matrix multiplication generally non-commutative?
4. What condition makes two vectors orthogonal under the standard inner product?
5. Distinguish a linear transformation from an affine transformation.

Long Answer Questions

1. Explain how Gaussian elimination determines whether a system has zero, one, or infinitely many solutions.
2. Derive the normal equations for the least-squares problem minₓ ‖Ax − b‖₂².
3. Discuss the roles of span, basis, dimension, and rank in representing a dataset.
4. Compare eigenvalue decomposition and SVD, including their uses in data science.
5. Explain why explicitly computing a matrix inverse is often inferior to using QR or SVD in numerical work.

Analytical / Application Questions

1. A dataset contains three features where the third is twice the first plus the second. Analyse the effect on rank, coefficient identifiability, and possible remedies.
2. Design a preprocessing and validation procedure for applying PCA to sensor data with unequal units, missing values, and temporal drift.
3. A least-squares model has excellent overall accuracy but substantially higher error for one subgroup. Analyse how the linear-algebraic representation and governance process should be investigated before deployment.

Multiple Choice Questions with Answers

Question 1

Which condition is equivalent to a square matrix A being invertible?

A. Its columns are linearly dependent

B. Its determinant is zero

C. Its rank equals its number of columns

D. Its null space contains a nonzero vector

Correct Answer: C. Its rank equals its number of columns

Question 2

For vectors x and y under the standard inner product, when are they orthogonal?

A. xᵀy = 1

B. xᵀy = 0

C. ‖x‖₂ = ‖y‖₂

D. x = y

Correct Answer: B. xᵀy = 0

Question 3

If A is an m × n matrix, what does rank–nullity state?

A. rank(A) + nullity(A) = m

B. rank(A) − nullity(A) = n

C. rank(A) × nullity(A) = n

D. rank(A) + nullity(A) = n

Correct Answer: D. rank(A) + nullity(A) = n

Question 4

What is the orthogonal projection of b onto a nonzero vector a?

A. (aᵀa)/(aᵀb)

B. ((aᵀb)/(aᵀa))a

C. (aᵀa)b

D. (aᵀb)a

Correct Answer: B. ((aᵀb)/(aᵀa))a

Question 5

In least squares, the residual at the optimum is orthogonal to which space?

A. The null space of A

B. The row labels of A

C. The column space of A

D. Every vector in ℝⁿ

Correct Answer: C. The column space of A

Question 6

Which factorisation exists for every real matrix A?

A. A = A⁻¹A

B. A = UΣVᵀ

C. A = λI only

D. A = QR with Q and R both square for every rectangular A

Correct Answer: B. A = UΣVᵀ

Question 7

If a feature is an exact linear combination of other features, what is the most direct consequence?

A. The data matrix becomes rank deficient

B. All observations become identical

C. The matrix must become symmetric

D. The null space becomes empty

Correct Answer: A. The data matrix becomes rank deficient

Question 8

For a matrix Q with orthonormal columns, which identity holds?

A. QᵀQ = 0

B. QᵀQ = I

C. QQᵀ = Q

D. Q⁻¹ = 0

Correct Answer: B. QᵀQ = I

Question 9

What does an eigenvalue λ represent in Av = λv?

A. The dimension of v

B. The number of rows of A

C. The scaling factor along the eigenvector direction

D. The determinant of every matrix

Correct Answer: C. The scaling factor along the eigenvector direction

Question 10

Why can explicitly forming (AᵀA)⁻¹ be undesirable in numerical least squares?

A. It always changes the number of observations

B. It can worsen numerical conditioning and is less stable than suitable factorisations

C. It prevents any matrix multiplication

D. It guarantees a nonlinear model

Correct Answer: B. It can worsen numerical conditioning and is less stable than suitable factorisations