6. Feature Selection, Extraction and Dimensionality Reduction

Introduction and Context

A data set may contain thousands of measured variables, yet only a fraction may be useful for predicting an outcome or discovering structure. Some variables are redundant, some are noisy, some are expensive to collect, and others may encode information that is unavailable at the time of deployment. The central problem is therefore not simply how to fit a powerful model, but how to construct a representation of the data that is informative, efficient, stable, and defensible.

Feature selection retains a subset of the original variables. Feature extraction transforms the original variables into a new, usually smaller, set of variables. Dimensionality reduction is the broader objective of representing data in fewer dimensions while preserving information relevant to a task. These activities are closely related to data preprocessing, predictive modelling, interpretability, computational efficiency, and governance. They are particularly important in text mining, genomics, sensor analytics, image analysis, finance, and high-dimensional business data.

A well-designed reduction strategy can improve generalisation by reducing variance, shorten training time, simplify deployment, and make patterns easier to inspect. However, reduction can also remove weak but collectively important signals, conceal meaningful variables, or introduce leakage if it is performed before the training and test data are properly separated. The appropriate method consequently depends on the modelling objective, the data-generating process, and the cost of errors.

Learning Objectives

By the end of this chapter, a student should be able to:

• distinguish feature selection from feature extraction and dimensionality reduction;
• compare filter, wrapper, and embedded selection methods;
• apply correlation analysis, mutual information, recursive feature elimination, regularisation, and importance-based methods;
• explain the mathematics and interpretation of principal component analysis and matrix factorisation;
• select and evaluate manifold-learning methods for visualisation or representation learning;
• assess information loss, stability, interpretability, computational cost, and leakage risk;
• design a defensible feature-reduction workflow for a practical data-mining problem.

The Feature-Reduction Problem

Let the input matrix be \(X \in \mathbb{R}^{n \times p}\), with \(n\) observations and \(p\) features. In supervised learning, the target is represented by \(y\). Feature selection chooses an index set \(S \subseteq \{1,\ldots,p\}\) and forms \(X_S\), while feature extraction creates a transformed matrix \(Z = f(X)\), where \(Z\) has fewer columns than \(X\).

Selection preserves the original meaning of variables. If a model retains income, age, and transaction frequency, a business user can still interpret each variable. Extraction may instead produce components such as \(z_1 = 0.42x_1 - 0.18x_2 + \cdots\), whose meaning is distributed across many original variables. Extraction can be highly effective for compression, but it often weakens direct interpretability.

Reduction has several possible objectives:

1. Prediction: retain information that improves performance on unseen data.

2. Description: identify variables or latent dimensions that explain structure.

3. Visualisation: represent complex data in two or three dimensions.

4. Efficiency: reduce storage, training time, and inference cost.

5. Measurement design: identify a smaller set of variables for future collection.

These objectives need not produce the same solution. A feature that is useful for prediction may be difficult to interpret; a principal component that captures large variance may have little relationship with a target; and a compact representation used for visualisation may be unsuitable for production prediction.

Three Families of Feature Selection

Filter methods: selection before modelling

Filter methods score variables using properties of the data, generally without training a particular predictive model. They are fast, scalable, and comparatively model-agnostic. Common filters include variance thresholds, correlation, chi-square tests, analysis of variance, and mutual information.

A variance threshold removes features whose values barely change. It is useful for eliminating constants or near-constants, but low variance does not necessarily imply low predictive value. A rare binary indicator may be highly informative for a small but important group.

Correlation measures linear association. For variables \(X_j\) and \(X_k\), Pearson correlation is

\[

r_{jk}=\frac{\operatorname{cov}(X_j,X_k)}{\sigma_j\sigma_k}.

\]

Highly correlated predictors may be redundant, particularly for linear models. Yet correlation is not a complete dependence measure: nonlinear relationships can have low Pearson correlation, and correlation with the target can miss interactions.

Mutual information measures statistical dependence. For random variables \(X\) and \(Y\),

\[

I(X;Y)=\sum_{x,y}p(x,y)\log\frac{p(x,y)}{p(x)p(y)},

\]

with an analogous integral for continuous variables. It is zero when variables are independent and positive when knowing one reduces uncertainty about the other. Mutual information can detect nonlinear association, but its estimate depends on sample size, discretisation or nearest-neighbour procedures, and the treatment of missing values.

Filter methods are appropriate when the feature space is very large, as in a document-term matrix. They can be used as an initial screening step before a more expensive model-based procedure. Their weakness is that they usually evaluate variables individually and may fail to recognise variables that are weak alone but valuable jointly.

Wrapper methods: search guided by a model

Wrapper methods evaluate subsets by repeatedly fitting a chosen learning algorithm. The model’s validation performance becomes the criterion for selecting features. Recursive feature elimination (RFE) is a widely used example. It fits a model, ranks features using coefficients or importance scores, removes the least important variables, and repeats until the desired number remains.

Forward selection begins with no variables and adds the one producing the greatest improvement. Backward elimination begins with all variables and removes one at a time. Exhaustive subset search is usually impractical because the number of subsets is \(2^p\). Greedy procedures are more feasible but may become trapped in a locally good subset.

Wrappers can account for interactions and model-specific behaviour better than filters. Their cost is substantial, especially when the estimator is complex or cross-validation is repeated many times. The selection criterion must be evaluated inside the resampling process. If RFE or forward selection uses the complete data before cross-validation, information from validation folds influences the selected subset and produces optimistic performance estimates.

Embedded methods: selection during model fitting

Embedded methods select or shrink features as part of the estimation procedure. They generally provide a useful compromise between the speed of filters and the task sensitivity of wrappers.

In linear regression, ordinary least squares minimises residual error. Ridge regression adds an \(L_2\) penalty:

\[

\min_{\beta}\left\{\frac{1}{n}\|y-X\beta\|_2^2+\lambda\|\beta\|_2^2\right\}.

\]

Ridge shrinks coefficients toward zero, which is valuable when predictors are correlated, but normally does not set them exactly to zero. Lasso regression uses an \(L_1\) penalty:

\[

\min_{\beta}\left\{\frac{1}{n}\|y-X\beta\|_2^2+\lambda\|\beta\|_1\right\}.

\]

The lasso can set coefficients exactly to zero and therefore performs selection. Elastic net combines both penalties and is often preferable when groups of correlated variables should be retained:

\[

\min_{\beta}\left\{\frac{1}{n}\|y-X\beta\|_2^2+\lambda\left[\alpha\|\beta\|_1+(1-\alpha)\|\beta\|_2^2\right]\right\}.

\]

The penalty parameter must be selected through cross-validation. Features should generally be standardised before penalised linear modelling, because the penalty acts on coefficient size and otherwise depends on measurement units.

Tree-based models provide another embedded or model-based approach. Decision trees and ensembles can estimate impurity-based or permutation importance. Impurity importance may favour continuous variables or categorical variables with many possible splits. Permutation importance measures the decrease in validation performance after a feature is randomly permuted, and is often more meaningful, although correlated predictors can share importance and make individual scores appear small.

A Practical Selection Workflow

A defensible workflow starts with the analytical question rather than with an algorithm. First define whether the purpose is prediction, explanation, measurement reduction, or visualisation. Next remove obvious identifiers, post-outcome variables, duplicated records, and features unavailable at prediction time. Handle missingness, encode categorical variables, and establish a baseline model.

The data should then be divided into training, validation, and test partitions, or assessed through nested cross-validation. All operations that learn from data—including imputation, scaling, feature ranking, selection, and component construction—must be fitted only on the training portion. In software, a pipeline is an effective way to enforce this order.

A typical supervised workflow may be:

• apply inexpensive domain and quality screens;
• remove one of each highly redundant feature pair when justified;
• use mutual information or a statistical test for initial screening;
• apply lasso, elastic net, RFE, or permutation importance;
• tune the number of features within cross-validation;
• compare against the full-feature baseline;
• inspect stability across folds, time periods, and subgroups;
• evaluate the final locked pipeline on untouched test data.

The number of selected features should not be determined solely by a conventional threshold. It should reflect the performance curve, operational cost, measurement availability, interpretability requirements, and stability of the selected set.

Extraction Through Principal Component Analysis

Principal component analysis (PCA) is an unsupervised linear transformation that creates orthogonal directions of maximum variance. After centring, and usually scaling, the covariance matrix is

\[

S=\frac{1}{n-1}X^TX.

\]

PCA finds eigenvectors \(v_k\) and eigenvalues \(\lambda_k\) satisfying \(Sv_k=\lambda_kv_k\). The first component is the direction \(v_1\) that maximises the variance of \(Xv_1\), subject to \(\|v_1\|=1\). Subsequent components are orthogonal to earlier ones and capture the largest remaining variance. The transformed observations are called scores, while the eigenvectors are often called loadings.

If the first \(q\) components are retained, the proportion of variance explained is

\[

\frac{\sum_{k=1}^{q}\lambda_k}{\sum_{k=1}^{p}\lambda_k}.

\]

This ratio is useful but not sufficient. High variance is not the same as predictive relevance. A low-variance direction may distinguish an important class. The number of components can be chosen using a scree plot, a cumulative variance target, cross-validated predictive performance, reconstruction error, or domain constraints.

PCA is sensitive to scale and outliers. If age is measured in years and income in currency units, income may dominate the covariance matrix unless standardisation is appropriate. Robust scaling or outlier treatment may be necessary. PCA components can also change substantially when the sample changes, especially when eigenvalues are similar. Loadings should therefore be examined, and component interpretation should be stated cautiously.

PCA is useful for compressing sensor measurements, reducing collinearity, denoising, and creating inputs for algorithms affected by high dimensionality. It is less suitable when the application requires direct variable-level explanations or when the data contain important nonlinear structure.

Matrix Factorisation and Latent Representations

Matrix factorisation represents a data matrix approximately as a product of lower-dimensional matrices. Singular value decomposition (SVD) gives

\[

X=U\Sigma V^T.

\]

A rank-\(q\) approximation retains only the largest singular values:

\[

X_q=U_q\Sigma_qV_q^T.

\]

For centred data, truncated SVD is closely related to PCA. SVD is especially useful for sparse high-dimensional matrices, such as document-term matrices, where explicitly forming a covariance matrix may be inefficient.

Non-negative matrix factorisation (NMF) constrains the factors to be non-negative:

\[

X\approx WH, \qquad W,H\geq 0.

\]

Because additive parts are combined rather than cancelled by positive and negative weights, NMF can produce interpretable topics, image parts, or customer profiles. Its solution is not generally unique and depends on the chosen rank and optimisation procedure.

Latent factor models are also common in recommender systems. A user–item rating matrix may be approximated by the product of user and item factor matrices. The factors represent hidden preferences rather than directly observed attributes. Missing-not-at-random ratings, popularity effects, and cold-start problems must be considered when interpreting such representations.

Manifold Learning: Nonlinear Geometry

Manifold methods assume that high-dimensional observations may lie near a lower-dimensional nonlinear surface. They attempt to preserve selected aspects of local or global geometry.

t-distributed stochastic neighbour embedding (t-SNE) is designed primarily for visualisation. It preserves local neighbourhood relationships by mapping similar points close together and dissimilar points farther apart in a low-dimensional display. Its appearance depends on perplexity, learning rate, initialisation, and random seed. Distances between separated clusters and cluster sizes should not be interpreted naively.

Uniform Manifold Approximation and Projection (UMAP) constructs a neighbourhood graph and seeks a lower-dimensional representation that preserves its structure. It is often faster and can preserve more global organisation than t-SNE in some settings, but its output remains parameter-dependent and is not automatically a reliable predictive representation.

Isomap preserves approximate geodesic distances along a neighbourhood graph, while locally linear embedding preserves local reconstruction relationships. These methods are valuable for exploratory analysis, but they can fail when the neighbourhood graph is disconnected, noisy, sparsely sampled, or incorrectly parameterised. For production features, transformations should be fitted on training data and applied consistently; not every visualisation algorithm provides a stable out-of-sample transformation.

Interpreting Information Loss and Model Quality

Reduction should be evaluated using more than one criterion. Predictive evaluation compares models trained on the full and reduced representations using the same resampling design. Reconstruction error assesses how accurately the original data can be approximated, but a low reconstruction error does not guarantee good prediction. Stability examines whether the same features or components are selected across samples, folds, time windows, and reasonable hyperparameter changes.

Interpretability has several dimensions. A selected original variable has semantic interpretability, whereas a component may have structural interpretability if its loadings form a coherent theme. Operational interpretability asks whether a decision can be explained to a customer, auditor, clinician, or manager. Sparse loadings, grouped features, monotonic constraints, and domain-informed transformations can improve interpretation.

Information loss may be technical, predictive, or organisational. Technical loss concerns reconstruction; predictive loss concerns the target; organisational loss concerns the disappearance of variables needed for fairness monitoring, audit, or intervention design. A feature may be removed from the prediction model but retained in a governance dataset for monitoring disparate outcomes.

Applied Case: Reducing Features in Credit-Card Fraud Detection

Background

A payment company receives transaction records containing merchant, amount, time, device, location, account history, and derived behavioural variables. The original table contains 420 candidate features, including highly correlated velocity measures and several variables created after a transaction has been reviewed.

Situation and problem

The fraud team’s gradient-boosting model has strong training performance but unstable validation results. Inference latency is increasing, data-quality failures are common, and the compliance team cannot explain why some variables are used. A key risk is leakage: chargeback status and investigation outcomes are not available when a transaction is authorised.

Analysis

The team first defines the deployment time as the moment of authorisation. Post-authorisation variables are removed. Features are divided into groups representing transaction context, customer history, device behaviour, and merchant behaviour. Missingness and availability are assessed separately for each group. A correlation analysis identifies redundant velocity measures, but the team avoids automatic removal where two measures have different operational meanings.

A time-based training and validation split is used because fraud patterns evolve. Mutual information provides an initial screen, followed by a model-based comparison of elastic net logistic regression, gradient boosting, and RFE. Permutation importance is calculated on future validation periods rather than on the training data. The selected set is compared with a PCA representation.

Approach and solution

The final design retains 68 original features. Elastic net is used as a transparent benchmark, while the production model uses a constrained tree ensemble. PCA is rejected for the primary decision model because its components cannot be explained adequately to investigators and because the modest predictive gain does not justify the interpretability cost. A small PCA representation is nevertheless used for exploratory visualisation of changing fraud patterns.

Selection, imputation, and scaling are implemented in a single training pipeline. The test period is held back until the selection strategy is fixed. The governance process records feature definitions, availability time, selection rationale, subgroup performance, and drift indicators.

Outcome

The reduced model lowers inference cost and improves operational reliability while maintaining validation performance close to the full model. More importantly, investigators can relate alerts to transaction velocity, device novelty, geographic inconsistency, and account behaviour. The team continues to monitor whether correlated variables are being replaced by new proxies for protected characteristics.

Learning points

The case demonstrates that the best reduction method is not determined by predictive accuracy alone. Leakage control, temporal validation, stability, explanation, data availability, and governance are part of feature engineering. It also illustrates that selection and extraction can coexist: original variables may serve the production model while latent representations support exploration.

Limitations, Risks, and Professional Responsibilities

A common mistake is to rank features on the complete data before splitting it. This causes leakage even when the target is not directly used, because distributional information from the test set influences the transformation. Another mistake is to assume that a zero coefficient proves a variable is useless. Under correlation, a penalised model may select one variable from a group arbitrarily.

Importance is not causality. A feature can be highly predictive because it is a proxy, a consequence of the target, or an artefact of data collection. Removing protected attributes does not necessarily remove discrimination because other variables may encode them indirectly. Feature reduction should therefore include fairness analysis, subgroup error measurement, privacy assessment, and documentation of data provenance.

Unsupervised methods present distinct risks. PCA may hide rare but consequential variation. t-SNE and UMAP can create visually persuasive clusters from parameter choices. Matrix factorisation can reproduce historical popularity or institutional bias. Reproducible random seeds, sensitivity analysis, and domain review are essential.

In regulated applications, a compact model is not automatically a compliant model. Organisations may need to retain audit variables, explain automated decisions, demonstrate data minimisation, and provide mechanisms for correction. Feature selection should be reviewed as part of model risk management rather than treated as a purely technical optimisation.

Current Directions and Best Practice

Modern workflows increasingly combine domain knowledge with automated search. Stability selection, bootstrap-based inclusion frequencies, group lasso, and knockoff methods attempt to control false discoveries or identify reliable variables. AutoML systems can search transformations and subsets, but their outputs still require leakage control, external validation, interpretability assessment, and human review.

Representation learning with autoencoders and supervised deep embeddings offers nonlinear compression for images, language, and complex signals. These representations can be powerful but may be difficult to audit and can encode unwanted shortcuts. In high-stakes settings, simpler selection methods may be preferable even when a black-box representation produces a small performance gain.

Best practice is to treat reduction as a modelling decision with an explicit purpose. Establish a full-feature baseline, use leakage-safe pipelines, evaluate on realistic future-like data, inspect stability, preserve governance-relevant variables, and report both what was removed and why. A reduced representation is successful when it delivers an appropriate balance among predictive value, cost, robustness, interpretability, and responsibility.

Chapter Summary

Feature selection retains original variables, whereas feature extraction creates new representations; both seek to reduce effective dimensionality. Filter methods are fast and model-independent but may miss interactions. Wrapper methods use model performance to search subsets and can be effective but computationally expensive. Embedded methods, including lasso, elastic net, and model-based importance, integrate selection with fitting.

PCA and SVD provide linear low-dimensional representations based on variance or singular values. NMF offers additive latent factors, while t-SNE, UMAP, Isomap, and locally linear embedding explore nonlinear geometry. Reduction must be evaluated through predictive performance, reconstruction, stability, interpretability, operational cost, and information loss. Leakage prevention, fairness, documentation, and temporal validation are essential. The appropriate method is determined by the purpose and risks of the application, not by dimensionality reduction alone.

Key Terms

Feature selection: Choosing a subset of the original variables.

Feature extraction: Transforming original variables into new features.

Dimensionality reduction: Representing data with fewer dimensions while retaining useful information.

Filter method: A selection method that scores features independently of a fitted predictive model.

Wrapper method: A method that evaluates feature subsets using a learning algorithm.

Embedded method: A method that performs selection during model estimation.

Mutual information: A measure of statistical dependence based on reduction in uncertainty.

Recursive feature elimination: Iterative removal of the least important features according to a fitted model.

Regularisation: Penalisation added to an objective function to control model complexity.

Lasso: Regression using an L1 penalty that can produce zero coefficients.

Ridge: Regression using an L2 penalty that shrinks coefficients toward zero.

Elastic net: A model combining L1 and L2 regularisation.

Principal component: An orthogonal direction capturing ordered variance in PCA.

Loading: The weight assigned to an original variable in a component.

Score: An observation’s coordinate on a component or latent dimension.

Singular value decomposition: Matrix factorisation into orthogonal matrices and singular values.

Non-negative matrix factorisation: Factorisation constrained to non-negative latent factors.

Manifold learning: Nonlinear dimensionality reduction based on assumed low-dimensional geometry.

Permutation importance: Performance decrease caused by randomly permuting a feature.

Data leakage: Use of information during training that would not be available at deployment or that originates from evaluation data.

Review Questions

Short Answer Questions

1. Distinguish feature selection from feature extraction.

2. Why can mutual information identify relationships that Pearson correlation misses?

3. What is the principal purpose of the L1 penalty in lasso regression?

4. Define explained variance in the context of PCA.

5. Why must feature-selection operations be performed within cross-validation?

Long Answer Questions

1. Compare filter, wrapper, and embedded methods in terms of model dependence, computational cost, ability to capture interactions, and interpretability.

2. Explain PCA from covariance construction through component selection, and discuss why high explained variance may not imply high predictive value.

3. Discuss how elastic net addresses limitations that arise when predictors are strongly correlated.

4. Compare SVD, NMF, t-SNE, and UMAP as dimensionality-reduction approaches.

5. Design a complete leakage-safe feature-reduction workflow for a supervised classification problem.

Analytical / Application Questions

1. A data scientist ranks 2,000 features using the full data, retains the top 100, and then reports cross-validated accuracy. Identify the methodological problem and propose a corrected procedure.

2. A PCA model explains 95% of input variance but reduces fraud-detection recall substantially. Analyse why this may occur and recommend an alternative evaluation and modelling strategy.

3. An organisation removes a protected attribute from a model but retains several geographic and purchasing variables. Explain how proxy discrimination may remain and identify governance checks that should accompany feature selection.

22. Multiple Choice Questions with Answers

Question 1

Which method is most directly designed to detect nonlinear dependence between a feature and a target without fitting a specific predictive model?

A. Pearson correlation

B. Mutual information

C. Ridge regression

D. Recursive feature elimination

Correct Answer: B. Mutual information

Question 2

A major methodological risk occurs when feature ranking is performed before cross-validation because it can:

A. Increase the number of observations

B. Guarantee model underfitting

C. Allow validation information to influence the selected features

D. Make all features normally distributed

Correct Answer: C. Allow validation information to influence the selected features

Question 3

Which regularisation method can set some linear-model coefficients exactly to zero?

A. Ridge

B. Lasso

C. Ordinary least squares

D. PCA

Correct Answer: B. Lasso

Question 4

Why is elastic net often useful with strongly correlated predictors?

A. It removes the need for validation

B. It combines sparsity from L1 regularisation with stabilisation from L2 regularisation

C. It guarantees causal interpretation

D. It operates without numerical optimisation

Correct Answer: B. It combines sparsity from L1 regularisation with stabilisation from L2 regularisation

Question 5

In PCA, the first component is the direction that:

A. Maximises projected variance subject to a unit-length constraint

B. Has the smallest eigenvalue

C. Has the highest correlation with the target in every case

D. Contains only one original variable

Correct Answer: A. Maximises projected variance subject to a unit-length constraint

Question 6

Which statement about t-SNE is most appropriate?

A. It guarantees that distances between all clusters are globally meaningful

B. It is primarily a visualisation method sensitive to parameter settings

C. It always produces a unique representation

D. It preserves original feature meanings

Correct Answer: B. It is primarily a visualisation method sensitive to parameter settings

Question 7

A model’s permutation importance is low for two highly correlated variables. The most plausible explanation is that:

A. Neither variable can contain information

B. The variables may share predictive information, so permuting one leaves the other available

C. Permutation importance measures coefficient magnitude

D. Correlation automatically causes data leakage

Correct Answer: B. The variables may share predictive information, so permuting one leaves the other available

Question 8

Which representation is most likely to retain direct semantic interpretability?

A. A selected subset of original variables

B. A t-SNE coordinate

C. A PCA component with mixed loadings

D. A latent factor from an unconstrained neural network

Correct Answer: A. A selected subset of original variables

Question 9

In the credit-card case, post-authorisation chargeback information was removed primarily because it:

A. Had low variance

B. Was unavailable at the time of deployment and could cause leakage

C. Was necessarily protected information

D. Could not be standardised

Correct Answer: B. Was unavailable at the time of deployment and could cause leakage

Question 10

Which evaluation criterion addresses whether the same features are selected across samples and time periods?

A. Reconstruction error

B. Selection stability

C. Training accuracy

D. Component loading magnitude alone

Correct Answer: B. Selection stability