4. Analytic Geometry, Projections and High-Dimensional SpacesIntroduction and contextMachine-learning models operate on data represented as points, vectors, and matrices. A customer described by age, income, and transaction frequency is a point in a three-dimensional feature space; an image represented by pixel intensities is a point in a space with thousands or millions of dimensions. Although such spaces are often difficult to visualize, their geometry determines how learning algorithms measure similarity, separate classes, reduce dimension, and construct useful representations. Analytic geometry provides the language for this reasoning. It describes points through coordinates, lines and planes through equations, and relationships through distances, angles, and projections. These ideas are not merely mathematical preliminaries. Linear classifiers use hyperplanes as decision boundaries, nearest-neighbour methods use distances, principal component analysis uses projections, and least-squares estimation finds the point in a subspace that is closest to observed data. This chapter develops a geometric view of machine learning. It begins with affine spaces and hyperplanes, then examines distance and angle, orthogonal projection and least squares, convexity, and the special behaviour of high-dimensional spaces. The final sections connect these ideas to classification, clustering, representation learning, and professional data-science practice. Learning objectivesAfter studying this chapter, the reader should be able to: 1. Distinguish vector spaces, affine spaces, subspaces, and hyperplanes. 2. Represent lines, planes, and hyperplanes using parametric and implicit equations. 3. Compute and interpret distances between points, subspaces, and hyperplanes. 4. Explain angles, orthogonality, projections, and Gram matrices. 5. Derive the least-squares solution geometrically and algebraically. 6. Explain convex sets and their relevance to optimization and classification. 7. Analyze why distance and similarity behave differently in high-dimensional spaces. 8. Select appropriate geometric representations and metrics for machine-learning applications. 9. Recognize numerical, statistical, ethical, and practical risks associated with geometric modelling. From coordinates to affine structureA real vector space is a collection of objects that can be added and multiplied by scalars while satisfying the usual vector-space rules. In machine learning, the most common example is \(\mathbb{R}^d\), the set of real-valued vectors with \(d\) coordinates. A vector may represent either a displacement or a feature record, but these interpretations should not be confused. A displacement has a natural origin; a feature record often does not. A subspace \(S\) of \(\mathbb{R}^d\) contains the zero vector and is closed under linear combinations. If \(u_1,\ldots,u_k\) are vectors, their span is \[ \operatorname{span}(u_1,\ldots,u_k)=\left\{\sum_{i=1}^{k}\alpha_i u_i:\alpha_i\in\mathbb{R}\right\}. \] A line through the origin is a one-dimensional subspace, and a plane through the origin is a two-dimensional subspace. In applications, however, important structures frequently do not pass through the origin. An affine set is a translated subspace: \[ A=a+S=\{a+s:s\in S\}. \] For example, the line through point \(p\) in direction \(v\neq 0\) is \[ x=p+tv,\qquad t\in\mathbb{R}. \] This line is affine unless it happens to pass through the origin. Affine combinations preserve coefficients that sum to one, whereas arbitrary linear combinations need not. This distinction matters because centering data changes the origin and can convert affine relationships into linear ones without changing their intrinsic geometry. A set of points \(x_1,\ldots,x_n\) is affinely dependent if there are coefficients \(\alpha_i\), not all zero, satisfying \(\sum_i\alpha_i=0\) and \(\sum_i\alpha_i x_i=0\). Affine independence is useful when determining whether a collection defines a unique line, plane, or higher-dimensional flat. Hyperplanes and geometric decision boundariesA hyperplane in \(\mathbb{R}^d\) is an affine set of dimension \(d-1\). Its implicit equation is \[ w^\top x+b=0, \] where \(w\neq 0\) is a normal vector and \(b\) is an offset. The vector \(w\) is perpendicular to every direction lying within the hyperplane. The two half-spaces determined by the hyperplane are \[ w^\top x+b>0 \quad\text{and}\quad w^\top x+b<0. \] A linear binary classifier assigns a label according to the sign of \(w^\top x+b\). The score is sometimes called a discriminant or decision function. Its magnitude indicates algebraic separation, but it is not a geometric distance unless it is normalized by \(\|w\|\). The signed perpendicular distance from a point \(x_0\) to the hyperplane is \[ d_{\text{signed}}(x_0,H)=\frac{w^\top x_0+b}{\|w\|_2}, \] and the ordinary distance is its absolute value. This formula is central to margin-based classification. If two parallel hyperplanes are \(w^\top x+b_1=0\) and \(w^\top x+b_2=0\), their distance is \(|b_1-b_2|/\|w\|_2\). A hyperplane can be represented by many parameter pairs: multiplying both \(w\) and \(b\) by the same nonzero scalar leaves the geometric set unchanged. Consequently, model coefficients should not be interpreted without considering scale and feature standardization. Metrics: distance, angle, and similarityThe Euclidean inner product of vectors \(x\) and \(y\) is \(x^\top y\). It induces the Euclidean norm \[ \|x\|_2=\sqrt{x^\top x} \] and Euclidean distance \(\|x-y\|_2\). Two nonzero vectors are orthogonal when \(x^\top y=0\). The angle \(\theta\) between them satisfies \[ \cos\theta=\frac{x^\top y}{\|x\|_2\|y\|_2}. \] Cosine similarity is therefore insensitive to vector magnitude and measures orientation. This can be useful for text representations, where document length may be less meaningful than word-distribution direction. Euclidean distance, by contrast, responds to both direction and scale. The choice of metric encodes an assumption about what counts as similarity. The Euclidean metric assumes that coordinate differences are commensurable and that spherical neighbourhoods are appropriate. A weighted quadratic metric has the form \[ d_M(x,y)=\sqrt{(x-y)^\top M(x-y)}, \] where \(M\) is positive semidefinite. If \(M\) is positive definite, it defines a genuine norm after a suitable linear transformation. The Mahalanobis distance uses an inverse covariance matrix, when reliably estimated, to account for scale and correlation: \[ d_M(x,y)=\sqrt{(x-y)^\top\Sigma^{-1}(x-y)}. \] Poorly scaled features can cause one variable to dominate Euclidean distance. Standardization, whitening, domain-specific weighting, or learned metrics may be appropriate, but each changes the geometry and must be justified by the application. Orthogonal projection and the geometry of approximationGiven a nonzero vector \(u\), the orthogonal projection of \(x\) onto the line spanned by \(u\) is \[ \operatorname{proj}_{u}(x)=\frac{u^\top x}{u^\top u}u. \] The residual \(r=x-\operatorname{proj}_{u}(x)\) is orthogonal to \(u\). Thus projection decomposes a vector into a component retained by the subspace and a perpendicular component discarded by the approximation. For a subspace with basis columns in a matrix \(A\), the projection of \(x\) onto the column space of \(A\) is \[ \hat{x}=A(A^\top A)^{-1}A^\top x, \] provided the columns of \(A\) are linearly independent. The matrix \[ P=A(A^\top A)^{-1}A^\top \] is an orthogonal projection matrix. It satisfies \(P^2=P\) (idempotence) and \(P^\top=P\) (symmetry). If the basis is orthonormal, collected in \(Q\), the expression simplifies to \(P=QQ^\top\). For an affine set \(a+\operatorname{col}(A)\), first translate the point: project \(x-a\), then translate back: \[ \hat{x}=a+A(A^\top A)^{-1}A^\top(x-a). \] Projection is the closest-point operation under Euclidean distance. It is unique for a closed linear or affine subspace. This uniqueness makes projection useful in dimensionality reduction, signal reconstruction, geometric data cleaning, and constrained optimization. The Gram matrix \(G=A^\top A\) records pairwise inner products among basis vectors. If \(G\) is ill-conditioned, the basis contains nearly dependent directions, and projection calculations become numerically unstable. QR factorization or singular-value decomposition is generally safer than explicitly forming \((A^\top A)^{-1}\). Least squares as a projection problemConsider the overdetermined system \(Ax\approx b\), where \(A\in\mathbb{R}^{n\times p}\). Usually, no exact solution exists because \(b\) does not lie in the column space of \(A\). Least squares chooses \(\hat{x}\) to minimize \[ \|A\hat{x}-b\|_2^2. \] Geometrically, \(A\hat{x}\) is the orthogonal projection of \(b\) onto \(\operatorname{col}(A)\). The residual \(r=b-A\hat{x}\) is perpendicular to every column of \(A\), giving the normal equations \[ A^\top A\hat{x}=A^\top b. \] When \(A\) has full column rank, \(\hat{x}=(A^\top A)^{-1}A^\top b\). In numerical practice, QR decomposition or SVD is preferred, especially when predictors are correlated. With rank deficiency, the solution may not be unique; the Moore–Penrose pseudoinverse gives the minimum-norm least-squares solution. In linear regression, the columns of \(A\) may include a column of ones for the intercept. The fitted response \(A\hat{x}\) is a projection of the observed response vector onto the model subspace. This interpretation explains residual orthogonality, normal equations, and the effect of adding predictors. It also clarifies why a high training fit does not necessarily imply good generalization: a large or flexible subspace can project noise as well as signal. Regularization modifies this geometry. Ridge regression minimizes \[ \|Ax-b\|_2^2+\lambda\|x\|_2^2, \] which shrinks coefficients and improves stability when directions are poorly determined. In singular-vector coordinates, ridge reduces the influence of small singular values rather than allowing them to amplify noise. Convex sets and separating geometryA set \(C\) is convex if, for any \(x,y\in C\) and \(t\in[0,1]\), \[ tx+(1-t)y\in C. \] The line segment joining any two points in a convex set remains inside the set. Hyperplanes, half-spaces, Euclidean balls, norm balls, affine subspaces, and intersections of convex sets are common examples. Convexity is important because a local minimum of a differentiable convex objective is global, and many optimization algorithms have reliable convergence behaviour under suitable conditions. The convex hull of points \(x_1,\ldots,x_n\) is the set of all convex combinations \(\sum_i\alpha_i x_i\), where \(\alpha_i\geq0\) and \(\sum_i\alpha_i=1\). It represents the smallest convex set containing the points. In classification, if the convex hulls of two classes are disjoint, a separating hyperplane may exist under appropriate conditions. The maximum-margin principle used in support vector machines seeks a hyperplane that separates classes while maximizing the distance to the closest training observations. Convexity does not guarantee that the data are linearly separable. Interlocking classes, concentric structures, and many real decision boundaries require nonlinear features or kernels. A nonlinear mapping can make a problem more linearly separable in a transformed feature space, but it may also increase computational cost and risk overfitting. High-dimensional geometry: what changes as dimension grows?High-dimensional spaces are not simply large versions of three-dimensional space. Several geometric effects become statistically important. First, volume concentrates near the boundary of a ball. For a unit ball in \(d\) dimensions, the ratio of the volume of the inner ball of radius \(1-\varepsilon\) to the unit ball is \((1-\varepsilon)^d\), which decreases rapidly as \(d\) increases. Random points are therefore often located near the boundary rather than near the centre. Second, distances can concentrate. For independent coordinates with comparable variance, the relative difference between the largest and smallest pairwise distances may shrink as dimension increases. Nearest and farthest neighbours can become less distinguishable, weakening methods based on neighbourhood ranking. The precise behaviour depends on distributions, dependence, scaling, and effective rather than nominal dimension, but the practical warning is general: distance-based algorithms require validation in high-dimensional settings. Third, random vectors are often nearly orthogonal. If vectors have independent, centred coordinates, their normalized inner product tends to be close to zero as dimension grows. This helps explain why random projections can preserve certain pairwise distances approximately, as stated by the Johnson–Lindenstrauss principle: a finite collection of points can often be embedded into a substantially lower-dimensional space with controlled distortion, with target dimension growing logarithmically with the number of points and inversely with the square of the tolerated distortion. Fourth, the curse of dimensionality affects data coverage. The number of observations required to populate a region adequately can grow dramatically with dimension. A model may have many available coordinates but only a small effective sample size. This can produce unstable covariance estimates, misleading clusters, spurious separations, and severe overfitting. These phenomena motivate feature selection, dimensionality reduction, regularization, metric learning, sparse modelling, and domain-informed representations. Principal component analysis identifies directions of maximum variance, but high variance is not automatically predictive. Autoencoders learn lower-dimensional representations through reconstruction objectives, while supervised representation learning may preserve task-relevant rather than merely high-variance information. Applications in machine learningIn classification, the geometry of a decision boundary determines which observations are assigned to each class. Logistic regression produces a linear boundary in the original feature space; support vector machines emphasize margin geometry; neural networks compose affine transformations with nonlinearities to construct curved boundaries in the input space. In clustering, distances and neighbourhoods define the result. K-means minimizes within-cluster squared Euclidean distance and is consequently sensitive to scale, outliers, and non-spherical cluster structure. Hierarchical clustering depends strongly on the chosen linkage and metric. Density-based methods can identify irregular shapes, but density estimation becomes difficult as dimension increases. In dimensionality reduction, projection seeks a compact representation. PCA projects data onto an orthogonal subspace that maximizes retained variance, equivalently minimizing squared reconstruction error among linear subspaces of a fixed dimension. The method is useful for visualization, compression, denoising, and preprocessing, but principal components may be difficult to interpret and may discard low-variance yet decision-critical directions. In recommendation, information retrieval, and natural-language processing, cosine similarity is often preferred when orientation matters more than magnitude. In anomaly detection, distance from a learned centre or subspace can be useful, but a point may appear anomalous merely because of poor scaling or an unrepresented subgroup. Geometry must therefore be evaluated alongside data quality and domain context. Illustrative case study: reducing industrial sensor dataBackgroundConsider an illustrative manufacturing company monitoring a production line with 120 sensors. Each time window produces a 120-dimensional vector containing temperatures, pressures, vibration summaries, flow rates, and electrical measurements. The engineering team wants to detect developing faults before an unplanned shutdown. Situation and problemAn initial nearest-neighbour detector uses raw Euclidean distance. It reports many alarms after a change in operating speed, even though engineers identify no faults. Conversely, several known failures are not detected because the relevant vibration variables have small numerical scales relative to temperature variables. AnalysisThe raw coordinates impose an unjustified metric. Correlated sensors create redundant directions, and the number of dimensions makes neighbourhood distances similar. Operating speed also shifts the normal data manifold, so comparing observations without conditioning on regime creates false anomalies. A covariance estimate is unstable because the number of normal historical windows is not much larger than 120. ApproachThe team first standardizes measurements using training-period statistics and removes sensor channels with excessive missingness. It then uses SVD-based PCA on normal-operation data, retaining a compact subspace selected by cross-validation and engineering review. Each new observation is decomposed into a score component within the normal subspace and a residual component perpendicular to it. Monitoring both score distance and residual distance distinguishes ordinary changes in operating regime from patterns not represented by the normal model. Separate models are fitted for major operating regimes, and alarms are reviewed using sensor-level reconstruction contributions. OutcomeThe revised system produces fewer regime-change alarms and identifies abnormal residual patterns earlier in controlled validation. The company does not treat the geometric score as an automatic diagnosis; it is used to prioritize inspection and is combined with maintenance logs and engineering rules. Learning pointsThe case demonstrates that a distance is meaningful only relative to a justified metric and reference distribution. Projection can separate variation explained by a model from unexplained variation, while high-dimensional effects make naive neighbourhood methods unreliable. It also shows the value of combining mathematical structure with domain expertise rather than allowing an algorithmic score to become an unquestioned decision. Limitations, risks, and professional considerationsA geometric model can be mathematically correct yet operationally misleading. Feature scaling, missing-value treatment, outliers, measurement error, and sampling bias all alter distances and projections. If a protected attribute or its proxy dominates a metric, nearest-neighbour retrieval or anomaly scoring may reproduce discriminatory patterns. A classifier with a visually appealing margin may still have unequal error rates across groups. Data leakage is another common mistake. Scaling parameters, PCA directions, or covariance matrices must be estimated from training data only and then applied to validation and test data. Otherwise, information from the evaluation set changes the geometry used during training. Analysts should also distinguish correlation-based directions from causal mechanisms; PCA, for example, describes variation without establishing why it occurs. Numerical conditioning deserves attention. Explicit matrix inversion can magnify error, especially when features are nearly collinear. Stable factorizations, regularization, rank diagnostics, and sensitivity analyses are preferable. Distance metrics should be documented, monitored after deployment, and reassessed when the data-generating process changes. Interpretability is not automatic. A low-dimensional projection may be useful for visualization while obscuring important variables. Stakeholders should be told what the representation preserves, what it discards, and how uncertainty affects decisions. In high-impact applications, human review, audit trails, subgroup validation, privacy protection, and appropriate regulatory controls are essential. Emerging directions and best practiceModern representation learning increasingly treats geometry as learnable. Siamese networks and contrastive learning seek embeddings in which related examples are close and unrelated examples are separated. Manifold-learning methods model data as lying near a curved, lower-dimensional structure. Graph neural networks replace ordinary Euclidean neighbourhoods with relationships encoded by graph connectivity. Hyperbolic and other non-Euclidean embeddings are being investigated for hierarchical data. These developments do not eliminate the need for geometric discipline. A sound workflow should: - define the application-specific notion of similarity;
- inspect scale, dependence, missingness, and outliers;
- choose a metric and representation that reflect the task;
- fit transformations only on training data;
- use stable numerical algorithms;
- validate neighbourhoods, margins, projections, and errors across relevant subgroups;
- test robustness under distribution shift; and
- document assumptions, limitations, and operational consequences.
The central principle is that geometry is part of the model, not a neutral container for data. Choosing coordinates, a metric, a subspace, or a projection determines what the learning system can regard as close, separate, typical, or anomalous. Chapter SummaryAnalytic geometry gives machine learning a precise language for points, affine sets, subspaces, hyperplanes, distances, angles, and projections. Hyperplanes define linear decision boundaries, while normalized signed distance measures position relative to those boundaries. Inner products generate norms and angles, but alternative metrics such as Mahalanobis distance encode different assumptions about scale and correlation. Orthogonal projection finds the closest point in a subspace or affine set. Least squares is therefore a projection problem: fitted values are the projection of observations onto the model space, and residuals are orthogonal to that space. Convex sets support reliable optimization and provide the geometric basis for separation and maximum-margin methods. High-dimensional spaces exhibit boundary concentration, distance concentration, near-orthogonality, and sparse data coverage. These effects influence classification, clustering, anomaly detection, and representation learning. Feature scaling, dimensionality reduction, regularization, stable factorization, careful validation, and fairness-aware evaluation are necessary for responsible application. Geometry should always be treated as an explicit modelling choice. Key TermsAffine set: A translation of a vector subspace. Affine combination: A linear combination whose coefficients sum to one. Convex set: A set containing the line segment between every pair of its points. Convex hull: The smallest convex set containing a specified collection of points. Distance metric: A rule that quantifies separation between objects while satisfying metric properties. Gram matrix: A matrix of pairwise inner products, commonly written \(A^\top A\). Hyperplane: An affine set of dimension one less than its ambient space. Inner product: A bilinear operation that generalizes the dot product and defines angles and lengths. Least squares: Estimation by minimizing the squared Euclidean residual norm. Mahalanobis distance: A covariance-adjusted distance based on \(\Sigma^{-1}\). Margin: The distance between a decision boundary and nearby observations or class constraints. Normal vector: A vector perpendicular to a line, plane, or hyperplane. Orthogonal projection: The closest point in a subspace under Euclidean distance. Pseudoinverse: A generalized inverse that provides minimum-norm solutions for rank-deficient problems. Residual: The component remaining after a model approximation or projection. Subspace: A vector space contained within another vector space. Support vector: An observation that constrains a maximum-margin decision boundary. Whitening: A transformation that rescales and decorrelates variables, often using covariance structure. Curse of dimensionality: Difficulties in estimation, coverage, and neighbourhood analysis caused by increasing dimension. Review QuestionsShort Answer Questions1. What distinguishes an affine set from a vector subspace? 2. Why is \(|w^\top x+b|/\|w\|_2\) a distance to a hyperplane? 3. State two defining properties of an orthogonal projection matrix. 4. Why are residuals orthogonal to the columns of a least-squares design matrix? 5. What is the practical significance of distance concentration in high-dimensional data? Long Answer Questions1. Derive the orthogonal projection of a vector onto the column space of a full-rank matrix and explain its geometric meaning. 2. Explain least squares through both the normal equations and the projection viewpoint. 3. Compare Euclidean, cosine, and Mahalanobis similarity, including situations in which each may be appropriate. 4. Discuss the role of convexity in linear classification and optimization. 5. Explain how high-dimensional geometry affects clustering and dimensionality reduction. Analytical / Application Questions1. A classifier uses \(w=(3,4)\) and \(b=-10\). Compute the signed distance of \(x=(4,3)\) from its decision hyperplane and interpret the result. 2. A dataset contains highly correlated features measured in different units. Propose a geometrically justified preprocessing and modelling strategy, including safeguards against data leakage. 3. Design an anomaly-detection approach for observations believed to lie near a low-dimensional subspace. Explain how projection residuals, regime changes, validation, and fairness or governance concerns should be handled. 22. Multiple Choice Questions with AnswersQuestion 1Which expression represents the signed perpendicular distance from a point x to the hyperplane wᵀx + b = 0? A. wᵀx + b B. |wᵀx + b| C. (wᵀx + b)/||w||₂ D. (wᵀx + b)/||x||₂ Correct Answer: C. (wᵀx + b)/||w||₂ Question 2For a full-column-rank matrix A, which matrix gives the orthogonal projection onto col(A)? A. AᵀA B. A(AᵀA)⁻¹Aᵀ C. (AᵀA)A D. A⁻¹Aᵀ Correct Answer: B. A(AᵀA)⁻¹Aᵀ Question 3In least squares, the residual b − A x̂ is orthogonal to: A. The null space of A only B. Every row of A C. The column space of A D. The vector x̂ only Correct Answer: C. The column space of A Question 4Which property characterizes a convex set C? A. It must contain the origin B. It must be bounded C. Every line segment joining two points in C lies in C D. All points must be mutually orthogonal Correct Answer: C. Every line segment joining two points in C lies in C Question 5Two vectors have cosine similarity zero. Under the usual Euclidean inner product, they are: A. Parallel B. Orthogonal C. Identical in magnitude D. At Euclidean distance zero Correct Answer: B. Orthogonal Question 6Why can raw Euclidean distance be inappropriate when features have very different units? A. Euclidean distance ignores all coordinate differences B. The feature with the largest numerical scale can dominate the distance C. Euclidean distance is always invariant to rescaling D. Different units make vectors nonlinear Correct Answer: B. The feature with the largest numerical scale can dominate the distance Question 7Which statement about a projection matrix P for an orthogonal projection is correct? A. P² = 0 and Pᵀ = −P B. P² = P and Pᵀ = P C. P² = I and Pᵀ = −P D. P has no eigenvalues Correct Answer: B. P² = P and Pᵀ = P Question 8A major consequence of distance concentration in high-dimensional spaces is that: A. All observations become exactly identical B. Nearest and farthest neighbours may become less distinguishable C. Every classifier becomes perfectly accurate D. The dimension automatically decreases Correct Answer: B. Nearest and farthest neighbours may become less distinguishable Question 9Why is QR factorization or SVD often preferred to explicitly computing (AᵀA)⁻¹? A. It changes a linear problem into a classification problem B. It avoids or reduces numerical instability caused by ill-conditioning C. It guarantees that the data are linearly separable D. It removes the need for a target variable Correct Answer: B. It avoids or reduces numerical instability caused by ill-conditioning Question 10In the industrial sensor case, monitoring both subspace scores and perpendicular residuals helps distinguish: A. Training labels from test labels B. Explained operating variation from patterns not represented by the normal model C. Euclidean distance from cosine similarity only D. Convex sets from affine sets Correct Answer: B. Explained operating variation from patterns not represented by the normal model
|