Dimensionality Reduction and Representation Learning

Introduction and Context

Modern machine-learning datasets often contain hundreds, thousands, or millions of measured variables. Images contain pixel intensities, text is represented by vocabulary dimensions, and industrial systems may record many correlated sensor streams. Although a large number of variables can contain useful information, it also creates computational, statistical and interpretive difficulties.

Dimensionality reduction transforms data from a high-dimensional space into a lower-dimensional representation while attempting to preserve information relevant to a task. The reduction may be linear, as in principal component analysis (PCA), or nonlinear, as in manifold learning and neural autoencoders. Representation learning goes further: rather than relying entirely on manually designed variables, it learns a useful encoding directly from data.

Dimensionality reduction supports:

• visualization in two or three dimensions;
• compression and storage efficiency;
• removal of noise and redundant variables;
• faster model training and inference;
• mitigation of multicollinearity;
• discovery of latent structure; and
• construction of representations for downstream prediction, retrieval or clustering.

Reduction is not automatically beneficial. Eliminating dimensions can discard rare but important signals, obscure causal meaning, or introduce misleading visual patterns. The correct representation depends on the objective, data-generating process and evaluation criterion.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. distinguish feature selection from feature extraction;
2. explain the curse of dimensionality and the bias–variance implications of dimensionality;
3. standardize data appropriately before applying PCA;
4. derive and interpret PCA using covariance matrices and singular value decomposition;
5. compare matrix factorization, manifold learning and autoencoders;
6. select dimensionality-reduction methods for visualization, compression, denoising and efficient modelling;
7. evaluate representations without data leakage;
8. identify limitations, ethical risks and governance requirements; and
9. implement a defensible dimensionality-reduction workflow in professional practice.

1. The Dimensionality Problem

Let a dataset be represented by a matrix X ∈ ℝ^(n×p), where n is the number of observations and p is the number of measured features. A reduction produces Z ∈ ℝ^(n×k), where k < p. The mapping may be written as:

Here, f is a learned or designed transformation. If the transformation is linear, it can often be expressed as Z = XW, where W ∈ ℝ^(p×k) contains directions or loadings.

1.1 Curse of dimensionality

As p increases, observations become sparse relative to the volume of the feature space. Distances become less discriminative: in many high-dimensional distributions, the nearest and farthest neighbour distances become increasingly similar. Consequently, methods based on distance, density or local neighbourhoods require much more data as dimensionality grows.

High dimensionality also increases the number of potential relationships that a model can fit. If irrelevant variables are included, a flexible model may fit noise rather than generalizable structure. This is one reason dimensionality reduction can improve generalization, although it is not a substitute for appropriate regularization or adequate sample size.

1.2 Feature selection versus feature extraction

Feature selection retains a subset of the original variables. If the selected index set is S, then the resulting representation is X_S. The variables remain interpretable, such as “age,” “temperature” or “blood pressure.” Selection may be filter-based, wrapper-based or embedded in a predictive model.

Feature extraction creates new variables from the originals. PCA components, latent factors and neural embeddings are examples. The new variables may be compact and predictive but are often less directly interpretable.

A supervised task may favour selection when auditability is essential. An unsupervised compression task may favour extraction.

2. Preparing Data for Reduction

The quality of a reduced representation depends strongly on preprocessing.

2.1 Centering and scaling

For feature j, the sample mean is:

The sample standard deviation is:

A standardized value is:

Centering makes each feature have sample mean approximately zero. Scaling gives features with different units comparable variance. PCA applied to an unscaled matrix is covariance-based; PCA applied after standardization is effectively correlation-based. Neither is universally correct. For example, if physical units and measurement precision are meaningful, scaling may remove useful information. If one variable is measured in euros and another in millimetres, scaling is often appropriate.

A feature with zero variance cannot be standardized using this formula and should normally be removed or handled separately.

2.2 Missing values, outliers and categorical variables

Many algorithms require a complete numeric matrix. Missing values should be imputed using a procedure fitted only on the training data. Median imputation is robust for skewed numeric variables, while model-based imputation may preserve relationships at greater complexity.

PCA is sensitive to outliers because covariance uses squared deviations. Robust scaling, winsorization, robust covariance or an outlier-aware model may be appropriate, but removing observations must be justified rather than used merely to improve a plot.

Categorical variables require suitable encoding. One-hot encoding can create sparse, high-dimensional matrices; numerical labels should not be used when they falsely imply order. For mixed data, methods designed for mixed-type variables may be preferable.

2.3 Preventing leakage

The means, standard deviations, selected features and reduction directions must be learned from the training partition only. Applying PCA to the entire dataset before cross-validation allows information from validation observations to influence the components. A valid workflow fits preprocessing and reduction inside each training fold and applies the fitted transformation to the corresponding validation fold.

3. Principal Component Analysis

PCA finds orthogonal directions that successively capture maximum variance. It is one of the most important linear dimensionality-reduction methods.

Assume X has been centered, with n observations and p features. The sample covariance matrix is:

C ∈ ℝ^(p×p), and C_{jk} measures the sample covariance between features j and k. The first principal direction w₁ solves:

The constraint ||w||₂ = 1 prevents the objective from becoming arbitrarily large by scaling w. The scalar wᵀCw is the variance of the projection Xw. Subsequent directions maximize the same quantity subject to orthogonality with earlier directions.

Using a Lagrange multiplier λ, maximize wᵀCw subject to wᵀw = 1:

Setting the derivative with respect to w to zero gives:

Thus, the principal directions are eigenvectors of C, and their associated eigenvalues are the variances of the corresponding principal-component scores. Eigenvalues are conventionally ordered λ₁ ≥ λ₂ ≥ ⋯ ≥ λ_p.

The reduced representation using k components is:

where W_k = [w₁, …, w_k] and W_kᵀW_k = I_k.

3.1 Reconstruction and optimality

A rank-k reconstruction is:

For centered data and a fixed k, PCA gives the orthogonal rank-k approximation that minimizes the squared Frobenius reconstruction error:

The squared Frobenius norm is the sum of squared reconstruction errors over all observations and features. Under the stated centered, Euclidean-loss assumptions, the minimum error equals the sum of discarded eigenvalues:

This result does not mean PCA preserves predictive information. It preserves variance, which may be unrelated to the target variable.

3.2 Explained variance

The proportion of variance explained by component j is:

Cumulative explained variance for k components is:

A threshold such as 90% or 95% is a heuristic, not a universal rule. The number of components should also consider validation performance, computational cost, interpretability and the consequences of reconstruction error.

3.3 Singular value decomposition

Computing the covariance matrix can be inefficient or numerically undesirable when p is very large. Singular value decomposition (SVD) expresses centered X as:

where U and V have orthonormal columns and Σ contains non-negative singular values σ_j. The principal directions are the columns of V, and the component scores are UΣ. The covariance eigenvalues satisfy:

SVD is especially useful for sparse or high-dimensional data. Randomized SVD can efficiently approximate leading components when only a small k is required.

3.4 Worked example: two correlated variables

Suppose centered observations have covariance matrix:

The eigenvalues solve det(C − λI) = 0:

Therefore λ₁ = 7 and λ₂ = 1. The total variance is 8, so the first component explains 7/8 = 87.5% of the variance. A corresponding unit eigenvector is w₁ = (1/√2, 1/√2)ᵀ. Hence the first score for an observation (x₁, x₂) is:

The direction combines the two correlated variables. The second direction, proportional to (1/√2, −1/√2)ᵀ, captures their contrast and explains 12.5%. This example illustrates that PCA rotates the coordinate system rather than simply deleting one variable.

3.5 Interpretation and limitations of PCA

Loadings indicate how strongly original variables contribute to a component, but signs are arbitrary: multiplying an eigenvector by −1 produces the same subspace. A large loading does not prove causation. Components may mix unrelated mechanisms, and linear PCA cannot represent curved structure efficiently.

PCA is also affected by scaling, outliers, missing-value treatment and feature engineering. For supervised modelling, selecting k solely by explained variance can be inferior to selecting k by nested cross-validation on the actual prediction objective.

4. Matrix Factorization and Latent-Factor Models

Matrix factorization approximates a data matrix using lower-dimensional factors. A rank-k approximation can be written:

where W ∈ ℝ^(n×k) contains observation factors and H ∈ ℝ^(k×p) contains feature factors. The factors are generally not unique: for an invertible matrix Q, WH = (WQ)(Q⁻¹H). Constraints or regularization are therefore needed for identifiability or useful interpretation.

4.1 Low-rank approximation and SVD

Truncated SVD retains the k largest singular values:

It is optimal for squared-error rank-k approximation under the conditions stated for the Eckart–Young theorem. In information retrieval, latent semantic analysis applies truncated SVD to a term-document matrix, reducing vocabulary co-occurrence patterns to latent topics.

4.2 Non-negative matrix factorization

When X contains non-negative quantities such as counts, intensities or spending, non-negative matrix factorization (NMF) imposes W ≥ 0 and H ≥ 0 elementwise:

NMF often produces parts-based representations because additive combinations are allowed but cancellation through negative coefficients is not. Unlike SVD, NMF generally has no single closed-form global solution; alternating optimization is commonly used, and results may depend on initialization.

4.3 Probabilistic latent factor models

A basic factor model represents an observation x_i as:

where μ is a mean vector, Λ is a loading matrix, f_i is a latent factor vector and ε_i is noise. A common assumption is f_i ∼ N(0, I) and ε_i ∼ N(0, Ψ), with Ψ often diagonal. The model separates shared latent variation from feature-specific noise, but Gaussian assumptions and rotational ambiguity must be assessed.

4.4 Recommendation systems

In a user–item matrix, missing entries can be predicted by latent factors:

Here r̂_{ui} is the predicted rating of user u for item i, μ is a global mean, b_u and b_i are user and item biases, and p_u and q_i are k-dimensional latent vectors. Training often minimizes observed-entry squared error plus regularization:

Ω is the set of observed ratings and λ ≥ 0 controls shrinkage. Evaluation must account for temporal order and exposure bias; a recommendation model should not be judged only by reconstructing ratings that users chose to provide.

5. Manifold Learning

Manifold learning assumes high-dimensional observations lie near a lower-dimensional smooth manifold embedded in the ambient space. For example, images of an object may vary according to a small number of factors such as pose, illumination and expression, despite having many pixels.

5.1 Locally linear embedding

Locally linear embedding (LLE) first represents each point x_i as a weighted combination of its neighbours:

It then finds low-dimensional points z_i preserving the same weights:

The reconstruction-weight constraint makes the representation invariant to translations. LLE depends strongly on neighbourhood size and can fail when the neighbourhood graph is disconnected or noisy.

5.2 Isomap

Isomap constructs a neighbourhood graph, estimates geodesic distances as shortest-path distances on that graph, and applies classical multidimensional scaling. It attempts to preserve distances along the manifold rather than straight-line Euclidean distances through the ambient space.

5.3 t-SNE and UMAP

t-SNE emphasizes local neighbourhood probabilities and is primarily a visualization tool. It can separate clusters visually even when global distances and cluster sizes are not faithfully represented. Results depend on perplexity, initialization and random seed.

UMAP constructs a graph based on local connectivity and optimizes a low-dimensional fuzzy-topological representation. It is often faster and can preserve more global structure than t-SNE in some settings, but its geometry still depends on hyperparameters and implementation choices. Neither method should be treated as proof that visually separated groups are objectively distinct populations.

For visualization, fit the method using a documented pipeline, repeat with multiple seeds, inspect sensitivity to hyperparameters, and avoid interpreting arbitrary axis scales as scientifically meaningful.

6. Autoencoders and Neural Representation Learning

An autoencoder consists of an encoder f_θ and decoder g_φ. Given input x, the encoded representation is:

The bottleneck z has fewer dimensions than x. A standard reconstruction objective is:

For continuous data, L is commonly squared error, L(x, x̂) = ||x − x̂||₂². For binary observations, binary cross-entropy may be more suitable, provided the output represents probabilities.

6.1 Undercomplete and denoising autoencoders

An undercomplete autoencoder has a bottleneck smaller than the input and must learn a compressed representation. Without sufficient restriction, an overcomplete network can learn an identity mapping without useful abstraction.

A denoising autoencoder receives a corrupted input x̃ but reconstructs the clean x:

q(x̃|x) is the chosen corruption process. The model learns features stable to the specified corruption, not necessarily to every real-world disturbance.

6.2 Variational autoencoders

A variational autoencoder (VAE) represents an approximate posterior q_θ(z|x) and a decoder distribution p_φ(x|z). It maximizes the evidence lower bound (ELBO):

The first term rewards reconstruction likelihood and the Kullback–Leibler term encourages the latent distribution to remain close to a prior p(z), commonly a standard normal. VAEs provide a structured probabilistic latent space but may produce blurrier reconstructions and require careful balance between reconstruction and regularization.

6.3 Convolutional and sequence autoencoders

For images, convolutional layers exploit local spatial structure and share parameters. For time series, recurrent, temporal-convolutional or transformer-based encoders can represent temporal dependencies. Architecture should reflect the data structure; flattening a large image into a generic vector may waste useful inductive bias.

6.4 Autoencoder cautions

Neural representations can encode sensitive attributes, shortcuts and dataset artefacts. A low reconstruction error does not establish that the representation is useful for a target task. Latent dimensions may not be identifiable: several transformations of the latent space can produce equivalent reconstructions. Regularization, hold-out evaluation, probing tasks and domain review are required.

7. Choosing and Evaluating a Method

The selection criterion should follow the intended use.

Useful metrics include reconstruction mean squared error:

For a supervised task, compare the complete baseline with the reduced model using a held-out test set. For clustering, use internal measures such as silhouette score cautiously and, where justified, external labels. For embeddings, assess trustworthiness, continuity, neighbourhood preservation and stability across seeds.

A robust workflow is:

1. define the operational objective;
2. identify the data type and measurement scale;
3. split data according to deployment conditions;
4. fit imputation, scaling and reduction on training data only;
5. compare against a no-reduction baseline;
6. tune k or other hyperparameters using validation data;
7. evaluate on untouched test data;
8. inspect stability, subgroup behaviour and failure cases; and
9. document the transformation for deployment and audit.

8. Illustrative Case: Predictive Maintenance in a Manufacturing Plant

Background

A plant monitors motors using 240 sensor variables, including vibration summaries, temperatures, currents and frequency-domain measurements. The engineering team wants earlier detection of bearing failure while reducing model latency.

Problem

Many sensor variables are strongly correlated, measurements have different units, and the failure class is rare. A two-dimensional plot is also requested to investigate operating regimes. The plant cannot remove sensors physically because they support safety and diagnostics, but it can reduce the representation supplied to a predictive model.

Analysis

The data are divided chronologically: earlier records are used for training, later records for validation and testing. This prevents future operating conditions from leaking into the past. Missing values are imputed using training-set medians, and numeric features are standardized using training-set means and standard deviations. Duplicate and constant variables are removed.

PCA is fitted to the training data. A scree plot shows that 18 components explain 92% of standardized variance, but validation experiments reveal that a classifier using 12 components has the best failure-detection trade-off. The team therefore does not choose 18 solely because of the 92% threshold.

For visualization, UMAP is fitted only to training data and applied cautiously to validation records. Repeated runs show two stable operating regimes, but a small apparent cluster disappears when the neighbourhood parameter changes. Engineers conclude that the small cluster is not sufficiently reliable for a maintenance policy.

A denoising autoencoder is also trained by masking selected sensor values and reconstructing the original values. It reduces reconstruction error on artificially corrupted validation data, but it does not improve failure detection over PCA. The autoencoder is rejected for production because its complexity is not justified by measurable operational benefit.

Outcome

The final classifier uses 12 PCA scores, operating regime indicators and a small number of domain-selected raw features. It achieves lower inference latency and similar test recall to the full-feature model. The retained raw features help engineers explain alarms, while the PCA scores capture correlated vibration and thermal patterns. The system reports uncertainty and requires human review before a shutdown decision.

Learning points

1. Variance preservation and predictive usefulness are different objectives.
2. Chronological splitting is essential when deployment involves future data.
3. Visualization embeddings are exploratory evidence, not automatic operating rules.
4. A more complex neural representation is not justified unless it improves the defined objective.
5. Hybrid representations can balance compactness and interpretability.

9. Professional, Ethical and Governance Considerations

Dimensionality reduction changes what a downstream decision-maker can observe. In high-stakes applications, a compact representation may hide evidence needed for contestability or clinical interpretation. Latent variables can also encode protected characteristics even when those variables are removed explicitly.

Important controls include:

• documenting features, preprocessing, algorithm, hyperparameters and random seeds;
• recording the training population and intended use;
• testing performance across relevant demographic and operational subgroups;
• checking whether missingness or measurement quality differs across groups;
• preserving access to original records where lawful and necessary;
• monitoring drift in component scores and reconstruction error;
• restricting use of visualization clusters as automated decisions; and
• conducting privacy review before releasing embeddings.

Embeddings can create privacy risks because membership, identity or sensitive attributes may be inferred from them. Access control, encryption, retention limits and attack testing are appropriate safeguards. In regulated environments, the organization should determine whether the transformation affects explainability, validation, record keeping or change-control obligations.

10. Contemporary Developments and Best Practices

Self-supervised learning learns representations from pretext tasks such as masked-feature reconstruction, contrastive alignment or next-element prediction. Foundation models generate reusable embeddings for text, images, audio and multimodal data. These representations can be powerful, but transfer performance depends on domain shift, licensing, training-data provenance and hidden biases.

For advanced practice:

• use PCA or randomized SVD for transparent linear baselines;
• use sparse or structured methods when data are high-dimensional and sparse;
• choose nonlinear embeddings for exploration, not as unquestioned measurements;
• evaluate learned representations on the actual downstream task;
• use nested validation when reduction and model selection are both tuned;
• test sensitivity to scaling, outliers, seeds and hyperparameters;
• retain an interpretable baseline;
• monitor representation drift after deployment; and
• treat latent features as model artefacts requiring governance, not as inherently objective discoveries.

Chapter Summary

Dimensionality reduction maps high-dimensional observations into a smaller representation for visualization, compression, denoising or efficient modelling. Feature selection retains original variables, whereas feature extraction constructs new variables. Preprocessing, especially scaling and leakage prevention, is fundamental.

PCA finds orthogonal directions of maximum variance through covariance eigenvectors or SVD. It provides an optimal linear rank-k approximation under squared reconstruction loss, but explained variance does not guarantee predictive value. Matrix factorization represents data through latent factors; SVD, NMF and probabilistic factor models differ in constraints and interpretation. Manifold-learning methods preserve selected local or geodesic relationships and are valuable for exploration, although their visual geometry can be unstable. Autoencoders learn nonlinear encoders and decoders, while denoising and variational objectives impose additional structure.

Professional selection requires a defined objective, leakage-free validation, comparison with a baseline, stability analysis and governance. Compact representations can improve efficiency, but they may reduce interpretability, encode bias or expose sensitive information. The most defensible solution is not necessarily the most sophisticated method; it is the method whose assumptions, benefits and limitations are demonstrated for the intended application.

Key Terms

Dimensionality reduction: Transformation of data into fewer dimensions while attempting to preserve relevant information.

Feature selection: Choosing a subset of the original variables.

Feature extraction: Constructing new variables from existing variables.

Curse of dimensionality: Difficulties caused by sparsity and unreliable distance or density relationships in high-dimensional spaces.

Standardization: Rescaling a variable by subtracting its mean and dividing by its standard deviation.

Covariance matrix: Matrix describing pairwise covariation among features.

Principal component: An orthogonal direction in feature space capturing a specified amount of variance.

Loading: Coefficient describing the contribution of an original feature to a component.

Explained variance: Variance captured by a component or group of components.

Singular value decomposition: Factorization X = UΣVᵀ into orthogonal matrices and singular values.

Low-rank approximation: Approximation of a matrix using factors with rank smaller than the original matrix.

Non-negative matrix factorization: Factorization constrained so that factor entries are non-negative.

Latent factor: Unobserved variable used to explain relationships among observed variables.

Manifold: A lower-dimensional structure embedded in a higher-dimensional space.

Geodesic distance: Distance measured along a manifold or its neighbourhood graph.

Embedding: Representation of objects as vectors in a learned or constructed space.

Autoencoder: Neural network that encodes input into a representation and decodes it for reconstruction.

Denoising autoencoder: Autoencoder trained to reconstruct clean data from corrupted inputs.

Variational autoencoder: Probabilistic autoencoder trained using a reconstruction objective and a KL-divergence regularizer.

Data leakage: Use of information during training that would not be available at the intended prediction time.

Review Questions

Short Answer Questions

1. Distinguish feature selection from feature extraction and give one example of each.
2. Why might standardization be important before applying PCA?
3. What does an eigenvalue represent in PCA?
4. State one reason t-SNE or UMAP plots should not be interpreted as proof of clusters.
5. What is the purpose of the bottleneck in an undercomplete autoencoder?

Long Answer Questions

1. Explain the derivation of PCA from the constrained variance-maximization problem. Define the role of eigenvectors and eigenvalues.
2. Compare truncated SVD, NMF and probabilistic factor models in terms of assumptions, constraints and interpretability.
3. Explain how data leakage can occur in a dimensionality-reduction pipeline and design a leakage-free validation procedure.
4. Compare PCA, manifold learning and autoencoders for visualization, compression and denoising.
5. Discuss the ethical and governance implications of using latent representations in a high-stakes decision system.

Analytical / Application Questions

1. A standardized dataset has PCA eigenvalues 6, 2, 1 and 1. Calculate the cumulative explained variance using the first two components and explain whether that alone is sufficient to choose k = 2.
2. Design a dimensionality-reduction workflow for a text-document matrix containing 100,000 terms, including preprocessing, candidate methods, evaluation and leakage controls.
3. A nonlinear embedding shows two well-separated groups, but the separation disappears under a different random seed and neighbourhood parameter. How should an analyst report and investigate this result?

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes feature selection from feature extraction?

A. Feature selection creates latent variables, whereas extraction retains original variables.

B. Feature selection retains chosen original variables, whereas extraction constructs new variables.

C. Feature selection can be used only for unsupervised learning.

D. Feature extraction always produces more variables than the original dataset.

Correct Answer: B. Feature selection retains chosen original variables, whereas extraction constructs new variables.

Question 2

For centered data, the first PCA direction is the unit vector that maximizes which quantity?

A. The number of observations

B. The classification accuracy on a test set

C. The variance of the projected observations

D. The number of missing values

Correct Answer: C. The variance of the projected observations

Question 3

A PCA covariance matrix has eigenvalues 7 and 1. What proportion of variance is explained by the first component?

A. 12.5%

B. 50%

C. 75%

D. 87.5%

Correct Answer: D. 87.5%

Question 4

Which condition is required for the standard PCA reconstruction-error optimality statement?

A. The data must be centered and the loss is squared Euclidean reconstruction error.

B. All features must be categorical.

C. The target variable must be normally distributed.

D. The data must contain no correlated variables.

Correct Answer: A. The data must be centered and the loss is squared Euclidean reconstruction error.

Question 5

What is a characteristic constraint of non-negative matrix factorization?

A. All factor entries must be orthogonal.

B. All factor entries must be non-negative.

C. The number of observations must equal the number of features.

D. The input must have zero variance.

Correct Answer: B. All factor entries must be non-negative.

Question 6

Why should PCA fitting normally occur inside each training fold during cross-validation?

A. To increase the number of features in each fold

B. To ensure validation observations influence the learned components

C. To prevent information from validation data leaking into training

D. To guarantee that every component explains equal variance

Correct Answer: C. To prevent information from validation data leaking into training

Question 7

What is the principal purpose of an undercomplete autoencoder bottleneck?

A. To force the network to learn a compressed representation

B. To ensure every input is reconstructed perfectly without training

C. To convert all variables into categorical values

D. To remove the need for a decoder

Correct Answer: A. To force the network to learn a compressed representation

Question 8

Which method is primarily designed to preserve local neighbourhood relationships for visual exploration?

A. Ordinary least squares regression

B. t-SNE

C. Naive Bayes classification

D. One-hot encoding

Correct Answer: B. t-SNE

Question 9

A dataset has total PCA variance 10, with the first two eigenvalues equal to 6 and 2. What cumulative variance do the first two components explain?

A. 20%

B. 40%

C. 60%

D. 80%

Correct Answer: D. 80%

Question 10

Which evaluation practice is most appropriate when selecting a representation for a predictive maintenance model?

A. Choose the representation only from the most attractive two-dimensional plot.

B. Use future records to fit preprocessing and then test on past records.

C. Compare reduced and unreduced baselines using deployment-appropriate held-out data.

D. Assume that the largest explained-variance threshold always gives the best classifier.

Correct Answer: C. Compare reduced and unreduced baselines using deployment-appropriate held-out data.