2. Vectors, Matrices and Linear Systems

Introduction and Context

Machine-learning systems represent observations, parameters, transformations, and predictions using mathematical objects. Among these, vectors and matrices are the basic language of data representation. A customer described by age, income, and transaction frequency can be represented as a vector; a dataset containing thousands of such customers can be represented as a matrix; and a predictive model can often be expressed through matrix–vector operations. Linear systems provide the mathematical framework for determining unknown parameters, enforcing constraints, and understanding whether a model is identifiable.

The importance of this topic extends beyond notation. Concepts such as subspaces, rank, orthogonality, norms, and matrix inverses explain why some datasets contain redundant information, why least-squares regression works, why numerical instability occurs, and why dimensionality reduction is possible. Later topics—including linear regression, principal component analysis, neural-network layers, optimization, and numerical methods—depend directly on the ideas developed here.

This chapter develops the algebra and geometry of vectors, matrices, and linear systems together. The emphasis is not only on calculation, but also on interpretation: what a matrix operation means, what rank reveals about information, and how these structures support reliable machine-learning practice.

Learning Objectives

By the end of this chapter, students should be able to:

1. Define vector spaces, subspaces, spans, bases, and dimension.

2. Compute and interpret vector norms, inner products, distances, and angles.

3. Explain orthogonality and orthogonal projection in geometric and statistical terms.

4. Perform matrix addition, multiplication, transposition, and block operations.

5. Interpret matrix rank, null spaces, column spaces, and determinants.

6. Determine when a matrix is invertible and solve linear systems using appropriate methods.

7. Distinguish exact, underdetermined, overdetermined, and inconsistent systems.

8. Apply these ideas to data representation, regression, feature engineering, and machine-learning models.

9. Recognize numerical, modelling, ethical, and practical limitations of linear representations.

Vectors as Representations of Data and Parameters

A vector is an ordered list of numbers. In an n-dimensional real vector space, a vector is written as

x = [x₁, x₂, …, xₙ]ᵀ,

where the superscript T denotes transposition from a row to a column. The components may represent physical measurements, features, model parameters, or coordinates relative to a chosen basis. For example, a vehicle might be represented by

x = [1, 1500, 4, 7]ᵀ,

where the entries indicate an intercept term, weight, number of cylinders, and age. The meaning of a vector is therefore determined by its context and by the scale and ordering of its coordinates.

Two operations define the basic algebra of vectors. Vector addition is componentwise, and scalar multiplication multiplies every component by the same scalar:

x + y = [x₁ + y₁, …, xₙ + yₙ]ᵀ,

αx = [αx₁, …, αxₙ]ᵀ.

A set closed under these operations and satisfying the usual algebraic rules is a vector space. The set Rⁿ of all real n-tuples is the most common space in machine learning, although complex vector spaces and function spaces also occur in advanced applications.

A linear combination of vectors v₁, …, vₖ is an expression α₁v₁ + … + αₖvₖ. The span of these vectors is the set of all such combinations. A subspace is a subset of a vector space that contains the zero vector and is closed under addition and scalar multiplication. The span of any collection of vectors is a subspace.

A collection is linearly independent if the equation

α₁v₁ + … + αₖvₖ = 0

has only the solution α₁ = … = αₖ = 0. If a non-zero combination produces the zero vector, the vectors are linearly dependent. Dependence is important in data analysis because it indicates redundant features or parameters. For example, if one feature is exactly twice another, the two columns do not provide independent directions of information.

A basis is a linearly independent set that spans a space. Every vector in the space has a unique coordinate representation relative to a basis. The number of vectors in a basis is the dimension of the space. In Rⁿ, the standard basis consists of vectors with one coordinate equal to one and all others equal to zero.

Norms, Inner Products and Geometry

A norm measures the size or length of a vector. The Euclidean, or ℓ₂, norm is

‖x‖₂ = √(x₁² + … + xₙ²).

The ℓ₁ norm is

‖x‖₁ = |x₁| + … + |xₙ|,

and the infinity norm is ‖x‖∞ = maxᵢ |xᵢ|. Different norms encode different notions of magnitude. The ℓ₂ norm is associated with ordinary geometric distance, whereas the ℓ₁ norm often supports sparse solutions because it penalizes the total absolute magnitude of coefficients.

An inner product is a function that maps two vectors to a scalar and satisfies linearity, symmetry, and positive definiteness. In Rⁿ, the standard inner product is the dot product:

⟨x, y⟩ = xᵀy = Σᵢ xᵢyᵢ.

The Euclidean distance between x and y is ‖x − y‖₂. The dot product also determines the angle θ between non-zero vectors:

cos θ = (xᵀy)/(‖x‖₂‖y‖₂).

If xᵀy = 0, the vectors are orthogonal. Orthogonality means that the vectors have no component in each other’s direction. In machine learning, orthogonal features can simplify interpretation, reduce certain forms of redundancy, and support stable numerical computation.

The Cauchy–Schwarz inequality states that |xᵀy| ≤ ‖x‖₂‖y‖₂. The triangle inequality states that ‖x + y‖₂ ≤ ‖x‖₂ + ‖y‖₂. These properties make norms mathematically consistent measures of length and distance.

For a non-zero vector u, the projection of x onto u is

projᵤ(x) = (xᵀu)/(uᵀu) u.

The residual x − projᵤ(x) is orthogonal to u. More generally, projection onto a subspace identifies the point in that subspace closest to x under Euclidean distance. This idea is central to least-squares estimation.

Matrices as Operators and Data Tables

A matrix is a rectangular array of numbers. An m × n matrix has m rows and n columns and is written A = [aᵢⱼ]. In a data matrix, rows commonly represent observations and columns represent features. In a linear transformation, the same matrix maps vectors from Rⁿ to Rᵐ:

A: Rⁿ → Rᵐ, x ↦ Ax.

The product Ax is defined when the number of columns of A equals the number of entries in x. If A is m × n and x is n × 1, then Ax is m × 1. The ith component of Ax is the inner product of the ith row of A with x.

Matrix addition is defined only for matrices of identical dimensions and is componentwise. Matrix multiplication is not generally commutative: AB may exist while BA does not, or both may exist but produce different results. Associativity does hold, so A(BC) = (AB)C whenever the products are defined. Distributivity also holds: A(B + C) = AB + AC.

The transpose Aᵀ exchanges rows and columns. A matrix is symmetric when Aᵀ = A. Symmetric matrices arise in covariance matrices, Gram matrices, and quadratic forms. The product AᵀA is always symmetric and positive semidefinite because

xᵀAᵀAx = ‖Ax‖₂² ≥ 0.

A diagonal matrix has non-zero entries only on its main diagonal. It scales coordinates independently. The identity matrix I leaves vectors unchanged, and the zero matrix maps every vector to zero. A block matrix partitions a large matrix into smaller submatrices, which is useful in algorithms, covariance modelling, and systems with several groups of variables.

The outer product xyᵀ produces a matrix, unlike the inner product xᵀy, which produces a scalar. Outer products appear in covariance estimation, low-rank approximation, and neural-network gradient calculations.

Subspaces, Rank and Information Content

Every matrix defines important subspaces. The column space Col(A) consists of all linear combinations of the columns of A. It is the set of vectors that can be produced as Ax. The null space Null(A) consists of all vectors x satisfying Ax = 0. The row space is spanned by the rows of A.

The rank of A is the dimension of its column space, equivalently the dimension of its row space. Rank measures the number of independent directions represented by a matrix. If A has n columns and rank n, its columns are independent and the transformation has no non-zero vector in its null space. If rank is smaller than n, some input directions are lost.

A matrix with more columns than its rank contains linearly dependent features. In a data matrix, this may result from duplicated variables, exact linear relationships, or excessive one-hot encoding. Rank deficiency can make parameter estimates non-unique and prevent the ordinary inverse of AᵀA from existing.

The rank–nullity theorem states that for an m × n matrix,

rank(A) + nullity(A) = n,

where nullity is the dimension of the null space. This theorem connects information retained by a transformation with information discarded by it.

Gaussian elimination transforms a matrix into row-echelon form using elementary row operations: swapping rows, multiplying a row by a non-zero scalar, and adding a multiple of one row to another. These operations preserve the solution set of a corresponding linear system and reveal pivots, rank, and free variables.

Determinants and Inverses

The determinant is defined for square matrices. For a 2 × 2 matrix,

A = [[a, b], [c, d]],

det(A) = ad − bc.

Geometrically, the absolute determinant gives the volume-scaling factor of the associated linear transformation; a negative determinant also indicates orientation reversal. A determinant of zero means that the transformation collapses at least one dimension. Therefore, a square matrix is invertible if and only if its determinant is non-zero.

For an invertible matrix A, the inverse A⁻¹ satisfies A⁻¹A = AA⁻¹ = I. The equation Ax = b then has the unique solution x = A⁻¹b. Although this formula is conceptually useful, explicitly computing an inverse is usually less efficient and less numerically stable than solving the system through factorization.

The following statements are equivalent for an n × n matrix: A is invertible; det(A) ≠ 0; rank(A) = n; Null(A) contains only the zero vector; the columns are linearly independent; and Ax = b has a unique solution for every b in Rⁿ. This equivalence provides several ways to diagnose a model or system.

A matrix may be invertible in exact arithmetic but numerically ill-conditioned. Its condition number compares sensitivity in the output to perturbations in the input. A large condition number indicates that small errors in measurements or computation may produce large errors in the solution. Feature scaling and appropriate numerical algorithms are therefore essential in practical machine learning.

Linear Systems and Their Solutions

A system of m linear equations in n unknowns can be written as

Ax = b.

The augmented matrix [A | b] combines the coefficient matrix and right-hand side. A system is consistent if it has at least one solution. It is inconsistent if no solution exists. When a consistent system has one solution, it has a unique solution; otherwise it has infinitely many solutions.

For a square full-rank matrix, Gaussian elimination produces a unique solution. If a row becomes [0 0 … 0 | c] with c ≠ 0, the system is inconsistent. If a row becomes entirely zero and at least one variable is free, the system has infinitely many solutions.

A homogeneous system Ax = 0 is always consistent. It has only the trivial solution when the columns of A are independent. If the null space is non-trivial, infinitely many solutions exist. Homogeneous systems describe constraints, dependencies, and directions eliminated by transformations.

Systems may be classified by their dimensions. A determined system has as many equations as unknowns, but equal dimensions do not guarantee a unique solution. An overdetermined system has more equations than unknowns and is often inconsistent because real observations contain noise. An underdetermined system has fewer equations than unknowns and generally has infinitely many solutions when consistent.

For an overdetermined system, the least-squares problem chooses x to minimize

‖Ax − b‖₂².

The residual r = b − Ax̂ is orthogonal to the column space of A. This gives the normal equations

AᵀA x̂ = Aᵀb.

If the columns of A are linearly independent, the solution is x̂ = (AᵀA)⁻¹Aᵀb. In practice, QR decomposition or singular value decomposition is generally preferred to directly forming and inverting AᵀA, particularly when features are highly correlated.

The least-squares solution can be interpreted geometrically: Ax̂ is the orthogonal projection of b onto Col(A). The residual is the part of b that cannot be represented by the model’s feature directions. This interpretation links linear algebra to regression error and model fit.

A Worked Example: Solving and Interpreting a Small System

Consider

2x + y = 5,

4x + 2y = 10.

The second equation is twice the first. The coefficient matrix has rank one, not two. There are infinitely many solutions: choosing y = t gives x = (5 − t)/2. The system is consistent but the parameters are not uniquely identifiable.

Now change the second equation to 4x + 2y = 11. The coefficient rows remain dependent, but the right-hand side is not in the same proportion. Elimination produces 0 = 1, so the system is inconsistent. In a data context, this may represent contradictory exact constraints or noisy observations that cannot all be satisfied simultaneously.

Finally consider a matrix with independent columns and more observations than features. Exact equality may not be possible, but least squares finds the coefficient vector whose predictions are closest to the observations in Euclidean distance. The distinction between exact solution and approximate projection is fundamental in statistical modelling.

Applications in Machine Learning and Industry

A feature vector is the basic input to many predictive models. Linear regression computes a weighted sum, often written ŷ = wᵀx + b. By appending a constant feature, the intercept can be incorporated into matrix notation. For many observations, predictions become ŷ = Xw.

Classification models also rely on vector operations. A linear classifier separates observations using a hyperplane wᵀx + b = 0. The vector w is normal to the hyperplane, and its direction determines the orientation of the decision boundary. Distances from the boundary depend on the norm of w, which explains why normalization appears in margin-based methods.

Covariance matrices describe pairwise feature variation. A covariance matrix is symmetric and positive semidefinite. Its eigenvectors, introduced more fully in later chapters, identify directions of changing variance. Gram matrices contain inner products between observations and support kernel methods.

In computer vision, an image can be flattened into a high-dimensional vector or retained as a matrix or tensor. In natural-language processing, documents and tokens are represented by vectors whose geometric relationships support similarity search. Recommendation systems often use matrix factorization to approximate a large user–item matrix with lower-dimensional latent factors.

In operations and engineering, linear systems model resource allocation, electrical networks, structural relationships, and state transitions. In finance, matrices represent portfolios, covariance structures, and risk constraints. In industrial analytics, rank and conditioning help diagnose sensor redundancy and unstable calibration models.

Illustrative Case Study: Diagnosing Redundant Features in a Demand Model

Background

A retailer develops a linear model to forecast weekly demand. The dataset includes advertising expenditure, website visits, number of promotional emails, total promotional contacts, price, and seasonal indicators. The analyst forms a design matrix X and estimates coefficients using least squares.

Situation and Problem

The fitted model produces unstable coefficients. Small changes in the training period cause the advertising coefficient to change sign, even though predictive accuracy remains similar. The matrix XᵀX is nearly singular, and the analyst initially considers deleting observations at random.

Analysis

Inspection shows that total promotional contacts equals website visits plus promotional emails for every observation. Thus one column is an exact linear combination of two others. The columns of X are linearly dependent, so X has deficient rank. The coefficient vector is not uniquely determined: different coefficient combinations produce identical fitted values.

In addition, website visits and advertising expenditure are strongly correlated, although not exactly dependent. This creates near-collinearity and a high condition number. Predictions may remain stable because the fitted vector Xw is similar, while individual coefficients are highly sensitive.

Approach and Solution

The modelling team removes the derived total-contact feature and retains the two underlying variables after confirming that both have business meaning. They standardize continuous variables to comparable scales and examine the singular values of X. Because correlation remains substantial, they compare ordinary least squares with ridge regression and evaluate performance using time-based validation rather than random splitting.

Ridge regression adds a penalty proportional to ‖w‖₂², which improves numerical stability and controls coefficient magnitude. The team reports both predictive performance and coefficient sensitivity. They also document the feature-generation rule so that future data pipelines do not reintroduce the redundant variable.

Outcome

Forecast accuracy changes only slightly, but coefficient estimates become more stable across validation periods. The organization gains a more interpretable model and avoids treating unstable coefficients as causal evidence about advertising effectiveness.

Learning Points

The case illustrates that rank is an information property, not merely a computational detail. Redundant features can prevent unique parameter estimation. Near-dependence may be as practically important as exact dependence because it amplifies noise. Removing variables should be guided by domain meaning, validation, and governance rather than by an arbitrary numerical rule.

Common Mistakes, Limitations and Risks

A frequent error is confusing the dimensions of a matrix product. The inner dimensions must agree, and the resulting matrix has the outer dimensions. Another mistake is assuming that a square matrix is automatically invertible. Squareness is necessary but not sufficient; full rank is required.

Students often interpret correlation as orthogonality without centering and scaling data. The dot product of raw feature columns depends on units and means. Orthogonality is a geometric condition, whereas statistical independence is a stronger probabilistic concept and should not be treated as equivalent.

Directly computing matrix inverses or normal equations can create numerical problems. Stable software libraries use QR, singular value decomposition, or carefully implemented factorizations. Poor feature scaling, extreme outliers, and ill-conditioned matrices can produce unreliable results even when formulas are mathematically correct.

Linear representations also impose modelling assumptions. A linear model may miss nonlinear interactions, thresholds, and changing relationships. A high-dimensional vector can encode sensitive attributes or proxies for protected characteristics. The mathematical ability to solve a system does not establish that the variables are appropriate, that the model is causal, or that its use is fair.

Data practitioners should preserve feature definitions, units, ordering, missing-value rules, and transformations. A matrix without metadata is not a meaningful model representation. Reproducibility requires versioned data pipelines, documented preprocessing, and tests verifying matrix dimensions and column semantics.

Contemporary Developments and Best Practices

Modern machine-learning systems increasingly operate with very large, sparse, structured, or distributed matrices. Sparse matrix formats avoid storing zeros and are essential for text features and recommender systems. Automatic differentiation represents gradients through vector–Jacobian and Jacobian–vector products without explicitly materializing enormous matrices.

Randomized linear algebra enables approximate low-rank decompositions and fast least-squares solutions for large datasets. Singular value decomposition remains a central tool for diagnosing rank, reducing dimensionality, and constructing pseudoinverse solutions. In large-scale learning, iterative methods such as conjugate gradients solve systems using matrix–vector products rather than explicit inverses.

Good practice includes checking dimensions before computation, standardizing features when appropriate, identifying exact and near dependencies, using stable decompositions, monitoring condition numbers or singular values, and validating results on held-out data. It is also important to distinguish algebraic fit from predictive usefulness and predictive usefulness from ethical acceptability.

Chapter Summary

Vectors represent observations, parameters, and directions in a coordinate space. Vector spaces, subspaces, spans, bases, and dimension describe how these representations are constructed. Norms measure magnitude, inner products measure alignment, and orthogonal projections identify closest points in subspaces.

Matrices represent data tables and linear transformations. Their column spaces, row spaces, and null spaces reveal what information a transformation can express or loses. Rank measures independent information, while determinants and inverses characterize square systems with unique solutions. Linear systems may be unique, inconsistent, or underdetermined, depending on rank and dimensions.

Least squares extends exact linear-system solving to noisy and overdetermined data by projecting observations onto a matrix’s column space. These ideas support regression, classification, dimensionality reduction, recommendation, computer vision, and industrial analytics. Practical use requires attention to scaling, conditioning, redundancy, numerical stability, data meaning, and responsible model governance.

Key Terms

Vector: An ordered collection of numbers representing a point, direction, observation, or parameter set.

Vector space: A set closed under vector addition and scalar multiplication.

Subspace: A subset of a vector space that is itself a vector space under the same operations.

Linear combination: A weighted sum of one or more vectors.

Span: The set of all linear combinations of a specified collection of vectors.

Linear independence: A property in which only the zero coefficients produce the zero vector.

Basis: A linearly independent set that spans a vector space.

Norm: A function measuring the magnitude or length of a vector.

Inner product: A scalar-valued operation measuring algebraic alignment between vectors.

Orthogonality: A relationship in which the inner product of two vectors is zero.

Projection: The closest component of a vector lying in a specified direction or subspace.

Matrix: A rectangular array representing data or a linear transformation.

Rank: The dimension of a matrix’s column space, equivalently its row space.

Null space: The set of vectors mapped to zero by a matrix.

Determinant: A scalar describing volume scaling and invertibility of a square matrix.

Inverse: A matrix that reverses the action of an invertible square matrix.

Condition number: A measure of the sensitivity of a numerical problem to perturbations.

Linear system: A set of linear equations represented as Ax = b.

Least squares: A method minimizing the squared residual between observed and modelled values.

Residual: The difference between an observed vector and its model prediction.

Review Questions

Short Answer Questions

1. What is the difference between a vector space and a subspace?

2. Define the inner product of two vectors in Rⁿ.

3. What does the rank of a matrix measure?

4. State the condition under which a square matrix is invertible.

5. Why is an overdetermined system often solved using least squares?

Long Answer Questions

1. Explain linear independence, span, basis, and dimension, using a data-feature example.

2. Describe the geometric meaning of norms, inner products, angles, and orthogonal projections.

3. Explain the row space, column space, and null space of a matrix and relate them through rank–nullity.

4. Compare exact solution methods for linear systems with least-squares methods for noisy data.

5. Discuss why directly computing matrix inverses may be inappropriate in practical machine-learning systems.

Analytical / Application Questions

1. A design matrix contains a feature equal to the sum of two other features. Analyze the effect on rank, parameter identifiability, and ordinary least squares.

2. Given a system with more equations than unknowns, explain how you would determine whether it has an exact solution and, if not, how you would obtain an approximate solution.

3. A model’s predictions are stable, but its coefficients vary greatly between samples. Use rank, conditioning, and regularization concepts to explain this behaviour and propose a diagnostic approach.

22. Multiple Choice Questions with Answers

Question 1

Which condition guarantees that a square matrix A has a unique solution to Ax = b for every b?

A. A has at least one zero row

B. A has full rank

C. A has more columns than rows

D. The rows of A are all identical

Correct Answer: B. A has full rank

Question 2

If xᵀy = 0 for two non-zero vectors, what does this imply?

A. They have equal lengths

B. They are linearly dependent

C. They are orthogonal

D. Their sum is zero

Correct Answer: C. They are orthogonal

Question 3

What does a non-trivial null space of A indicate?

A. A maps no vector to zero

B. The columns of A are linearly dependent

C. A must be square

D. A has a non-zero determinant

Correct Answer: B. The columns of A are linearly dependent

Question 4

For an overdetermined system Ax = b, the least-squares solution minimizes which quantity?

A. ‖x‖₁

B. det(A)

C. ‖Ax − b‖₂²

D. rank(A)

Correct Answer: C. ‖Ax − b‖₂²

Question 5

A data matrix contains one column exactly equal to twice another column. What is the most direct consequence?

A. Its column rank is reduced

B. Its determinant must increase

C. All observations become orthogonal

D. The matrix becomes positive definite

Correct Answer: A. Its column rank is reduced

Question 6

Which operation produces a scalar from two vectors x and y?

A. xyᵀ

B. x + y

C. xᵀy

D. [x y]

Correct Answer: C. xᵀy

Question 7

Why can a model have stable predictions but unstable individual coefficients?

A. The design matrix may contain highly correlated or nearly dependent columns

B. The response variable must be zero

C. All matrix norms are identical

D. The system necessarily has a unique exact solution

Correct Answer: A. The design matrix may contain highly correlated or nearly dependent columns

Question 8

What is the geometric interpretation of the least-squares fitted vector A x̂?

A. It is the reflection of b through the origin

B. It is the orthogonal projection of b onto Col(A)

C. It is always equal to b

D. It is a vector in Null(A)

Correct Answer: B. It is the orthogonal projection of b onto Col(A)

Question 9

Which statement about matrix multiplication is correct?

A. It is always commutative

B. It is defined for any two matrices

C. The inner dimensions must agree

D. The result always has the same dimensions as both inputs

Correct Answer: C. The inner dimensions must agree

Question 10

A large condition number primarily indicates that:

A. The matrix contains only integer entries

B. Solutions may be sensitive to small perturbations

C. The matrix has no column space

D. The determinant is necessarily exactly one

Correct Answer: B. Solutions may be sensitive to small perturbations