Matrices, Vectors and Linear Algebra

Introduction

Linear algebra is the mathematics of quantities that have several components and of relationships among those quantities. It provides the language for representing data tables, geometric objects, simultaneous equations, transformations, networks, images and machine-learning models. A spreadsheet of observations can be viewed as a matrix; a customer profile can be represented by a vector; and a recommendation algorithm can be expressed through matrix factorisation.

The central computational tasks of this chapter are to perform matrix operations, solve linear systems and interpret structural properties such as rank, determinant, basis, eigenvalues and orthogonality. These ideas are not isolated procedures. They form a connected framework for modelling and computing with multidimensional information.

Learning Objectives

After completing this chapter, a student should be able to:

1. Represent data and relationships using vectors and matrices.
2. Perform matrix addition, scalar multiplication, multiplication and transposition.
3. Formulate and solve systems of linear equations using elimination and matrix methods.
4. Explain vector spaces, subspaces, linear independence, span, basis and dimension.
5. Calculate and interpret rank, determinants, eigenvalues and eigenvectors.
6. Use inner products, norms, orthogonality and projections.
7. Explain linear transformations and their matrix representations.
8. Apply linear algebra to computer graphics, data science, machine learning and recommendation systems.
9. Recognise numerical, modelling and ethical limitations of linear-algebraic methods.

1. Vectors and Matrices

A scalar is a single number, such as 7 or −2. A vector is an ordered list of numbers. A column vector in ℝⁿ has n real components:

Here xᵢ denotes the i-th component and the superscript T means transpose. A vector may represent a geometric displacement, a student’s marks in several subjects, or the features of an image.

A matrix is a rectangular arrangement of numbers. An m × n matrix has m rows and n columns:

The entry aᵢⱼ is located in row i and column j. A matrix with the same number of rows and columns is square. The identity matrix Iₙ is the n × n matrix with ones on the main diagonal and zeros elsewhere. It satisfies AIₙ = A and IₙA = A whenever the products are defined.

Matrices are useful because their dimensions convey meaning. If A is m × n and x is n × 1, then Ax is m × 1. Thus A can convert n input quantities into m output quantities.

Basic Matrix Operations

If A and B have the same dimensions, their sum is defined element by element:

For a scalar c,

The transpose Aᵀ changes rows into columns: (Aᵀ)ᵢⱼ = aⱼᵢ. A matrix is symmetric when Aᵀ = A.

Matrix multiplication is defined when the number of columns of A equals the number of rows of B. If A is m × n and B is n × p, then AB is m × p, with entries

Matrix multiplication is generally not commutative: AB may exist while BA does not, or both may exist but have different values. It is associative and distributive, however:

Worked Example 1: Matrix Multiplication

Let

Using row-by-column multiplication,

By contrast,

Therefore AB ≠ BA.

2. Linear Systems

A system of m linear equations in n unknowns can be written as

where A is the coefficient matrix, x is the vector of unknowns and b is the right-hand-side vector. The system is consistent if it has at least one solution. It may have one solution, infinitely many solutions or no solution.

Gaussian Elimination

Gaussian elimination uses elementary row operations:

• interchange two rows;
• multiply a row by a nonzero scalar;
• add a multiple of one row to another row.

These operations preserve the solution set. The augmented matrix [A | b] is reduced to row-echelon or reduced row-echelon form. A pivot is the first nonzero entry in a nonzero row after elimination. A variable associated with a pivot is a basic variable; a non-pivot variable is free.

Worked Example 2: Solving a System

Solve

The augmented matrix is

Subtract twice row 1 from row 2, and subtract row 1 from row 3:

Replace row 2 by row 2 + 3(row 3):

Thus z = 18/7. From y − 2z = −3,

Finally, x + y + z = 6 gives

The unique solution is (x, y, z) = (9/7, 15/7, 18/7). Substitution into all three original equations verifies the result.

A row of the form [0 0 … 0 | c], where c ≠ 0, represents a contradiction and proves that the system has no solution. If there are fewer pivots than unknowns and no contradiction, at least one free variable exists and infinitely many solutions result.

Inverse Matrices

For a square matrix A, an inverse A⁻¹ exists if and only if A is nonsingular. It satisfies

If A⁻¹ exists, the system Ax = b has the solution x = A⁻¹b. In computation, directly calculating an inverse is usually less efficient and less numerically stable than solving by elimination or factorisation. The inverse is nevertheless useful conceptually and for small problems.

For a 2 × 2 matrix,

and, when ad − bc ≠ 0,

3. Vector Spaces, Span and Basis

A vector space is a set of objects that can be added and multiplied by scalars while satisfying closure, associativity, distributivity, identity and inverse properties. The familiar space ℝⁿ is a vector space, but polynomials, matrices and functions can also form vector spaces.

A linear combination of vectors v₁, …, vₖ is

where c₁, …, cₖ are scalars. The span of a set is the collection of all its linear combinations. A set is linearly independent if

implies that every coefficient cᵢ is zero. Otherwise it is linearly dependent.

A basis is a linearly independent spanning set. Every vector in a space has a unique coordinate representation relative to a chosen basis. The number of vectors in a basis is the dimension. For example, the standard basis of ℝ³ is e₁ = (1,0,0), e₂ = (0,1,0), e₃ = (0,0,1), so dim(ℝ³) = 3.

The column space of A is the span of its columns. The null space is the set of solutions to Ax = 0. The rank of A is the dimension of its column space, equivalently the number of pivots in its row-reduced form. The rank–nullity theorem for an n-column matrix is

This theorem explains the degrees of freedom in a homogeneous system.

4. Determinants

The determinant is a scalar associated with a square matrix. For a 2 × 2 matrix it is ad − bc. Geometrically, |det(A)| is the factor by which the associated linear transformation scales area in two dimensions, or volume in three dimensions. A negative determinant also indicates orientation reversal.

A square matrix is invertible exactly when det(A) ≠ 0. Equivalent conditions include full rank, linearly independent columns and the equation Ax = 0 having only the zero solution.

For an n × n matrix, the determinant can be defined recursively by cofactors, although elimination is generally more efficient computationally. Important properties include:

• det(AB) = det(A)det(B);
• det(Aᵀ) = det(A);
• swapping two rows changes the sign;
• multiplying one row by c multiplies the determinant by c;
• adding a multiple of one row to another leaves it unchanged.

Determinants are valuable for theory and small calculations, but they should not ordinarily be used as the primary numerical test of invertibility for large, ill-conditioned matrices.

5. Eigenvalues and Eigenvectors

A nonzero vector v is an eigenvector of a square matrix A if applying A changes only its scale or direction sign:

The scalar λ is the corresponding eigenvalue. Rearranging gives

A nonzero solution exists precisely when

This equation is the characteristic equation. Its roots are eigenvalues; for each eigenvalue, solving (A − λI)v = 0 gives the eigenspace.

Worked Example 3: Eigenanalysis

Let

The characteristic equation is

Hence λ = 3 or λ = 1. For λ = 3, (A − 3I)v = 0 gives −x + y = 0, so an eigenvector is (1,1). For λ = 1, x + y = 0, so an eigenvector is (1,−1).

If a matrix has a basis of eigenvectors, it is diagonalizable:

where D contains eigenvalues on its diagonal and P contains corresponding eigenvectors as columns. Then powers can be calculated efficiently as Aᵏ = PDᵏP⁻¹. Symmetric real matrices have especially useful properties: all eigenvalues are real and eigenvectors associated with distinct eigenvalues are orthogonal.

Eigenvalues support dimensionality reduction, stability analysis, image processing, vibration analysis and network ranking. In data science, the eigenvectors of a covariance matrix identify principal directions of variation.

6. Inner Products, Norms and Orthogonality

For vectors x and y in ℝⁿ, the Euclidean inner product is

The Euclidean norm or length is

The vectors are orthogonal if xᵀy = 0. If a nonzero vector is divided by its norm, the result is a unit vector. A set of mutually orthogonal unit vectors is orthonormal.

The angle θ between nonzero vectors satisfies

The formula assumes the standard real inner product and nonzero vectors. The Cauchy–Schwarz inequality ensures that the right-hand side lies between −1 and 1.

The projection of x onto a nonzero vector u is

Projection onto a subspace is central to least squares. Given an overdetermined system Ax ≈ b, the least-squares estimate minimises ||Ax − b||₂². If the columns of A are linearly independent, it satisfies the normal equations

In practical numerical software, QR factorisation is often preferred to forming AᵀA because forming the latter can worsen conditioning.

7. Linear Transformations and Geometry

A function T: ℝⁿ → ℝᵐ is linear if

Every linear transformation is represented by a matrix once bases are selected. Its columns are the images of the basis vectors. For example, in two-dimensional graphics, a scaling transformation is

A rotation through angle θ is

These formulas assume coordinates in a standard Cartesian orthonormal basis and angles measured consistently, usually in radians for software libraries. Matrix composition follows application order: if a point is first transformed by B and then by A, its final vector is ABx.

Translations are not linear in ordinary two-dimensional coordinates because they do not preserve the origin. Computer graphics therefore often use homogeneous coordinates, representing (x,y) as (x,y,1), so translation, rotation and scaling can be combined using 3 × 3 matrices. This is an example of extending a mathematical representation to meet an application requirement.

8. Applications in Computing and Data Analysis

Computer Graphics

Vertices of a model are stored as coordinate vectors. Matrices perform scaling, rotation, reflection, projection and changes of coordinate system. A graphics pipeline repeatedly applies transformations to thousands of vertices, making matrix multiplication highly suitable for parallel hardware such as GPUs. Numerical errors can accumulate, so consistent coordinate conventions and stable transformation order are essential.

Data Science and Machine Learning

A dataset with m observations and n features is commonly stored as an m × n matrix X. A linear prediction model can be written as

where w is an n-dimensional weight vector, b is an intercept, 1 is an m-dimensional all-ones vector and ŷ is the predicted output. Training often minimises a loss function. For ordinary least squares, the objective is

Regularisation can reduce overfitting. Ridge regression uses

where α ≥ 0 controls the penalty. Feature scaling is important because different units can distort distances and the effect of regularisation.

Principal component analysis (PCA) centres the columns of X, forms a covariance matrix such as

and finds eigenvectors corresponding to the largest eigenvalues. The leading eigenvectors define directions retaining the greatest variance under the covariance model. PCA can compress data, but high variance is not automatically the same as predictive importance.

Recommendation Systems

Suppose R is a user–item rating matrix. Most entries may be missing. A latent-factor model approximates observed ratings by

where pᵤ and qᵢ are k-dimensional user and item vectors. A common training objective is

Here Ω is the set of observed ratings, k is the latent dimension and λ controls regularisation. This is a model, not proof that preferences truly have k independent causes. Cold-start users, popularity bias, sparse data and changing preferences remain important limitations.

9. Applied Case: Course-Recommendation Prototype

The following is an illustrative case. A university wants to recommend elective courses to students using prior course ratings.

Background. The system records ratings from 1 to 5, but most students have rated only a small number of courses. Course descriptions are also converted into feature vectors.

Problem. Directly comparing students is unreliable when two students have few common ratings. The university needs recommendations while avoiding unfair exclusion of less popular courses.

Analysis. A rating matrix R is constructed with students as rows and courses as columns. Missing values are not automatically treated as zero, because zero would mean a genuine absence of preference rather than an unobserved rating. The observed set Ω is identified, and user and course latent vectors are learned by minimising the regularised matrix-factorisation objective above.

Approach. The data is divided into training and validation sets by time, rather than randomly only, to reflect future recommendation. The model is compared with a popularity baseline. Regularisation and latent dimension k are selected using validation performance. Recommendations are filtered by prerequisites and timetable constraints. A diversity rule prevents every student from receiving only the most popular courses.

Outcome. In an appropriate deployment, success would be measured using ranking metrics such as precision at k, coverage and calibration, together with student feedback and enrolment outcomes. A mathematically accurate model can still fail if evaluation leaks future information or if recommendations reinforce historical inequalities.

Learning points. Matrix factorisation provides a compact representation of sparse relationships; regularisation controls overfitting; domain constraints are necessary; and governance must address privacy, transparency and fairness.

10. Numerical and Professional Considerations

Exact arithmetic in textbooks differs from floating-point arithmetic in computers. A matrix can be theoretically invertible but numerically ill-conditioned, meaning that small input errors produce large output changes. The condition number, informally, measures sensitivity; a large condition number warns that results may be unreliable. Scaling, pivoting and QR or singular value decomposition (SVD) improve robustness.

SVD factors a real m × n matrix as

where U and V have orthonormal columns (with appropriate dimensions) and Σ contains nonnegative singular values. SVD supports least squares, pseudoinverses, compression and rank approximation. It is often more generally applicable than eigen-decomposition, which is defined for square matrices but may not provide a complete eigenbasis.

Professionals should document dimensions, units, missing-value treatment, scaling, numerical precision and validation procedures. In machine-learning applications, personal data must be handled according to applicable privacy and data-protection requirements. Recommendations and automated decisions should be auditable, tested for disparate effects and accompanied by suitable human oversight. Linear algebra does not remove responsibility for the assumptions encoded in data and models.

Chapter Summary

Vectors represent ordered quantities, while matrices represent rectangular data and linear mappings. Matrix operations are dimension-dependent, and multiplication is generally non-commutative. Linear systems are represented as Ax = b and solved reliably through elementary row operations. Rank identifies independent directions, while nullity measures free dimensions through rank–nullity.

Determinants characterise invertibility and geometric scaling. Eigenvalues and eigenvectors identify directions preserved up to scale and support diagonalisation, PCA and stability analysis. Inner products define length, angle and orthogonality; projections and least squares provide principled approximations. Linear transformations underpin computer graphics and coordinate changes. Matrix models also support regression, dimensionality reduction and recommendation systems, but their quality depends on data, numerical stability, validation and ethical governance.

Key Terms

Scalar: A single numerical quantity.

Vector: An ordered list of components.

Matrix: A rectangular array of numbers.

Transpose: The operation that exchanges rows and columns.

Identity matrix: A square matrix acting as the multiplicative identity.

Linear system: A collection of equations expressible as Ax = b.

Pivot: A leading nonzero entry used in elimination.

Rank: The dimension of a matrix’s column space.

Nullity: The dimension of its null space.

Vector space: A set closed under vector addition and scalar multiplication.

Span: All linear combinations of a given set of vectors.

Basis: A linearly independent spanning set.

Determinant: A scalar indicating invertibility and signed scale change.

Eigenvalue: A scalar satisfying Av = λv for a nonzero v.

Eigenvector: A nonzero vector whose direction is preserved by a matrix.

Inner product: A product producing a scalar measure of alignment.

Orthogonality: Perpendicularity under an inner product.

Projection: The component of one vector in another direction or subspace.

Conditioning: Sensitivity of a numerical problem to input perturbations.

Singular value decomposition: A factorisation A = UΣVᵀ using orthogonal directions.

Review Questions

Short Answer Questions

1. What dimensions must A and B have for the product AB to be defined?
2. What is the difference between the span of vectors and their basis?
3. State the condition under which a square matrix is invertible.
4. Define an eigenvector and its associated eigenvalue.
5. What does it mean for two vectors to be orthogonal?

Long Answer Questions

1. Explain Gaussian elimination and illustrate how pivots determine the nature of a linear system’s solution.
2. Discuss rank, nullity, column space and null space, including the rank–nullity theorem.
3. Derive the 2 × 2 determinant and explain its geometric interpretation.
4. Explain projection and least squares, including the normal equations and their numerical limitation.
5. Compare the uses of eigen-decomposition and SVD in data analysis.

Analytical / Application Questions

1. A graphics program first rotates a point and then scales it. Explain which matrix product represents the combined transformation and why order matters.
2. A dataset contains features measured in rupees, years and counts. Analyse why feature scaling may be required before distance-based modelling or regularised regression.
3. Design an evaluation plan for a course-recommendation system that addresses sparse ratings, temporal validation, coverage, privacy and fairness.

Multiple Choice Questions with Answers

Question 1

If A is a 3 × 2 matrix and B is a 2 × 4 matrix, what is the dimension of AB?

A. 2 × 3

B. 3 × 4

C. 2 × 4

D. 4 × 3

Correct Answer: B. 3 × 4

Question 2

Which condition guarantees that a square matrix A is invertible?

A. det(A) = 0

B. A has a zero column

C. det(A) ≠ 0

D. A is rectangular

Correct Answer: C. det(A) ≠ 0

Question 3

For the matrix [[2,1],[1,2]], which is an eigenvalue?

A. 0

B. 1

C. 2

D. 4

Correct Answer: B. 1

Question 4

Two nonzero vectors x and y are orthogonal when:

A. xᵀy = 0

B. xᵀy = 1

C. ||x||₂ = ||y||₂

D. x = y

Correct Answer: A. xᵀy = 0

Question 5

What does a row [0 0 0 | 5] in an augmented matrix represent?

A. A unique solution

B. Infinitely many solutions

C. A contradiction and no solution

D. A zero vector

Correct Answer: C. A contradiction and no solution

Question 6

The rank of a matrix is equal to:

A. The number of its columns in every case

B. The number of pivots in row-echelon form

C. Its determinant in every case

D. The sum of all entries

Correct Answer: B. The number of pivots in row-echelon form

Question 7

In a recommendation model, rᵤᵢ ≈ pᵤᵀqᵢ, the vectors pᵤ and qᵢ primarily represent:

A. Latent user and item factors

B. Matrix row numbers only

C. Missing values coded as zero

D. Determinants of the rating matrix

Correct Answer: A. Latent user and item factors

Question 8

Which factorisation is especially useful for least squares and rank approximation of a general rectangular matrix?

A. SVD

B. Only a determinant

C. A scalar inverse

D. A transpose without multiplication

Correct Answer: A. SVD

Question 9

If A is 2 × 2 with entries [[1,2],[3,4]], what is det(A)?

A. −2

B. 2

C. 5

D. 10

Correct Answer: A. −2

Question 10

If a homogeneous system Ax = 0 has fewer pivots than unknowns, then it has:

A. No solution

B. Only one nonzero solution

C. Infinitely many solutions

D. A necessarily nonsquare matrix

Correct Answer: C. Infinitely many solutions