Dimensionality Reduction and Anomaly Detection

Introduction and Context

Modern datasets often contain hundreds or thousands of variables. Examples include customer transactions, sensor readings, medical measurements, images, text representations and web activity logs. Although many variables may contain useful information, high dimensionality creates practical difficulties: computation becomes slower, visualisation becomes difficult, irrelevant variables increase model complexity, and statistical estimates become less reliable. This problem is commonly called the curse of dimensionality.

Dimensionality reduction addresses this problem by representing data with fewer variables while preserving the information required for analysis. Two broad approaches are feature selection and feature extraction. Feature selection retains a subset of the original variables. Feature extraction creates new variables, such as principal components, from combinations of the original variables.

Anomaly detection has a closely related purpose. It identifies observations that differ substantially from the normal pattern. Anomalies may indicate fraud, equipment failure, cyberattack, data-entry error or a rare but legitimate event. Dimensionality reduction can make anomaly detection more efficient and interpretable, but reducing dimensions may also remove the very information that identifies an unusual observation. Therefore, both tasks require careful preprocessing, validation and domain knowledge.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Explain why high dimensionality creates analytical and computational problems.
2. Distinguish feature selection from feature extraction.
3. Apply filter, wrapper and embedded feature-selection methods.
4. Explain and calculate the main steps of principal component analysis (PCA).
5. Interpret low-dimensional visualisations responsibly.
6. Compare statistical, distance-based and isolation-based anomaly detection.
7. Select thresholds and evaluate anomaly-detection results.
8. Apply dimensionality reduction and anomaly detection to practical risk, fraud and quality-monitoring problems.
9. Recognise leakage, fairness, privacy and governance risks.

1. The Problem of High Dimensionality

Let a dataset contain n observations and p measured features. It can be represented by a matrix X with n rows and p columns. As p increases, several problems arise:

• Sparsity: observations become far apart in the feature space.
• Distance instability: nearest and farthest distances may become increasingly similar, reducing the usefulness of distance-based methods.
• Overfitting: a flexible model may learn noise rather than general patterns.
• Computational cost: storage, training and prediction require more resources.
• Interpretability: analysts cannot easily inspect many variables simultaneously.
• Redundancy: correlated variables may repeat the same information.

Dimensionality reduction is not automatically beneficial. A discarded feature may contain a small but important signal, and a transformed component may be difficult to explain to a manager or regulator. The objective is therefore not simply to minimise the number of variables; it is to achieve an acceptable balance among predictive performance, computational efficiency, interpretability and governance.

2. Feature Selection

Feature selection retains original variables and removes irrelevant, redundant or unstable ones. Unlike PCA, it preserves the original meaning of selected features, which is valuable in regulated or operational settings.

2.1 Filter methods

Filter methods evaluate features independently of a particular predictive model. Common examples include:

• variance thresholding, which removes almost constant variables;
• correlation screening, which removes one of two highly correlated predictors;
• mutual information, which measures statistical dependence between a feature and a target;
• chi-square tests for suitable categorical data;
• information gain or related measures for classification.

Filter methods are fast and suitable for an initial screening stage. However, a feature that appears weak alone may be valuable in combination with another feature. Correlation also measures only linear association unless an appropriate alternative is used.

For two random variables X and Y, Pearson correlation is commonly written as:

Here, cov(X,Y) is covariance and σ_X and σ_Y are standard deviations. The formula assumes numerical variables and measures linear association. A value close to zero does not prove that no relationship exists.

2.2 Wrapper methods

Wrapper methods search for a feature subset by repeatedly training and evaluating a predictive model. Recursive feature elimination, for example, begins with many features, ranks their importance, removes the least useful features and repeats the process.

Wrappers can find subsets tailored to a model, but they are computationally expensive. Feature selection must occur within each training fold during cross-validation; otherwise, information from the validation data can leak into the model-development process.

2.3 Embedded methods

Embedded methods perform selection during model training. Examples include decision-tree importance and regularisation. In linear regression with L1 regularisation, coefficients may be driven exactly to zero. A simplified objective is:

Here, y_i is the target, x_i is the feature vector, β_0 is the intercept, β is the coefficient vector, n is the number of observations and λ ≥ 0 controls the penalty. Larger λ generally produces more shrinkage and may set coefficients to zero. Features should normally be scaled before applying this penalty when their units differ. The method assumes that a linear representation is useful and that the chosen penalty is appropriate for the task.

2.4 A practical selection workflow

A professional workflow is as follows:

1. Define the modelling or operational objective.
2. Remove identifiers, duplicated fields and variables unavailable at prediction time.
3. Correct data types and address missing values.
4. Separate training, validation and test data before target-informed selection.
5. Apply domain rules and unsupervised filters.
6. Use a supervised filter, wrapper or embedded method where a target exists.
7. Validate performance, stability and business interpretation.
8. Document every removal decision and monitor selected features after deployment.

3. Principal Component Analysis

PCA is an unsupervised feature-extraction method. It transforms p possibly correlated variables into p mutually orthogonal principal components. The first component captures the greatest possible variance among all unit-length linear combinations; each subsequent component captures the greatest remaining variance subject to being orthogonal to earlier components.

3.1 Preprocessing

PCA is sensitive to measurement scale. If income is measured in thousands and age in years, the larger numerical scale may dominate. Standardisation is often appropriate:

x_{ij} is the value of feature j for observation i, μ_j is the training-set mean and σ_j is its training-set standard deviation. The resulting value has no physical unit. Standardisation is not always desirable: if the original units and their variance are substantively meaningful, covariance-based PCA may be more suitable. Means and standard deviations must be computed on training data only.

Missing values should be handled before PCA, and categorical variables require an appropriate representation. One-hot encoded variables can complicate interpretation because a component may combine many indicator variables.

3.2 Covariance and eigenvectors

After centring the data matrix X, the sample covariance matrix is:

S is a p × p matrix. Its diagonal contains feature variances and its off-diagonal entries contain covariances. PCA solves:

where v_k is the kth eigenvector, λ_k is its eigenvalue and v_k is conventionally normalised so that v_k^T v_k = 1. The principal-score vector is:

The eigenvalues are non-negative for a covariance matrix and represent the variance of the corresponding component. Components are ordered so that λ_1 ≥ λ_2 ≥ … ≥ λ_p.

The proportion of variance explained by the first k components is:

This ratio lies between 0 and 1. It is a descriptive measure, not a guarantee of predictive accuracy. A low-variance direction can contain highly important information, particularly for classification or anomaly detection.

3.3 Small worked example

Suppose a centred two-feature dataset has covariance matrix:

The eigenvalues are 3 and 1, with corresponding unit eigenvectors proportional to (1, 1) and (1, −1). Thus, the first principal component follows the direction in which both features increase together. Its explained variance is:

Therefore, one component preserves 75% of the total variance. The second component describes the contrast between the two features and preserves the remaining 25%. This example illustrates that PCA can compress correlated variables, but it does not say that the second direction is unimportant for every business question.

3.4 Choosing the number of components

Common aids include a scree plot, a cumulative explained-variance target, cross-validated task performance and interpretability. A fixed threshold such as 90% is a heuristic, not a universal rule. The selected number should depend on the intended use. For visualisation, two or three components may be adequate. For prediction, cross-validation should determine whether additional components improve generalisation.

PCA can be computed using eigendecomposition or singular value decomposition (SVD). SVD is often numerically stable and efficient for large or sparse matrices. The sign of an eigenvector is arbitrary: multiplying a component by −1 gives an equivalent solution, so component signs should not be interpreted independently of their loadings.

4. Visualisation and Interpretation

A two-dimensional projection can reveal clusters, gradients, overlap and possible outliers. A scatterplot of the first two principal scores is often useful for exploratory analysis. Colouring points by a known class can help assess separation, but it must not be confused with proof of a causal structure.

Visualisation should be accompanied by:

• the preprocessing description;
• explained variance for each displayed component;
• loading interpretation;
• awareness of projection loss;
• investigation of points that appear unusual in the original variables.

Methods such as t-SNE and UMAP can produce useful exploratory maps, especially for complex nonlinear structures, but distances and cluster sizes in the map may not have a straightforward global meaning. They should not automatically replace a validated analytical model or be used as sole evidence for high-impact decisions.

Figure: PCA identifies an axis of maximum variance and a perpendicular secondary axis.

5. Anomaly Detection

An anomaly is an observation that is sufficiently inconsistent with a reference pattern to warrant investigation. The reference may be a statistical distribution, a neighbourhood structure or a model of normal behaviour. Terminology varies: point anomalies are unusual individual observations, contextual anomalies are unusual in a particular context, and collective anomalies are unusual sequences or groups.

Anomaly detection differs from ordinary classification. Often only normal data are available, anomaly labels are scarce, and the cost of false negatives differs greatly from the cost of false positives. An anomaly score ranks observations; a threshold converts scores into an operational decision.

5.1 Statistical methods

For a univariate variable X that is approximately normal, a standardised score is:

where μ and σ are the reference mean and standard deviation. A rule such as |z_i| > 3 is a screening convention, not a universal law. It is unreliable for skewed, multimodal or contaminated data.

Robust alternatives use the median and median absolute deviation (MAD):

A robust score can be written as:

The constant 0.6745 makes the score approximately comparable to a standard z-score under a normal distribution. It should not be used when MAD is zero without an appropriate alternative. Robust methods are useful when a small number of extreme values would distort the mean and standard deviation.

For multivariate approximately elliptical data, the squared Mahalanobis distance is:

x_i is the feature vector, μ is the reference mean vector and Σ is the covariance matrix. The inverse requires a nonsingular, reliably estimated covariance matrix. With p variables and an appropriate multivariate normal model, D_i² is often compared with a chi-square reference distribution with p degrees of freedom, but the assumption and parameter-estimation uncertainty must be considered. Robust covariance estimates may be preferable when contamination is expected.

5.2 Distance-based methods

Distance methods identify observations with few close neighbours or large distances from a reference group. k-nearest-neighbour scoring, for example, may use the distance to the kth nearest observation. Local Outlier Factor (LOF) compares local density with the density around neighbouring observations. A point in a sparse region within a dense population may receive a high score.

These methods require a meaningful distance metric. Numeric variables should generally be scaled, categorical variables need suitable treatment and irrelevant dimensions can overwhelm distances. The choice of k affects sensitivity: small k detects very local deviations, while large k captures broader structure.

5.3 Isolation-based methods

Isolation Forest builds random partitioning trees. An unusual observation is typically separated from other observations after fewer random splits, producing a shorter average path length. It does not require a complete probability model and is often efficient for high-dimensional data.

Its results depend on subsampling, tree construction and the assumed meaning of unusualness. It can be less effective when anomalies form a large dense group, when normal behaviour has strong temporal structure, or when categorical and numerical variables are poorly encoded.

5.4 One-class modelling

One-class support vector machines and autoencoders learn a boundary or reconstruction pattern for normal data. An autoencoder maps an input to a compressed representation and reconstructs it; a large reconstruction error may indicate an anomaly. Such methods require careful validation and can learn to reconstruct anomalies if anomalous examples are present in training data.

6. Combining Reduction and Detection

A typical pipeline is:

1. Define the normal operating population and the decision context.
2. Split data chronologically or randomly as appropriate, keeping future information out of training.
3. Impute missing values and scale features using training statistics.
4. Remove leakage variables and apply domain-informed feature selection.
5. Fit PCA or another reduction method on training data only.
6. Train the anomaly detector on the intended reference population.
7. Score validation and test observations.
8. Set a threshold using costs, capacity, known labels or a target alert rate.
9. Investigate alerts and feed verified outcomes into monitoring and retraining.

Reduction may improve speed and reduce noise, but a detector should sometimes be fitted both to original and reduced features. Comparing the two can reveal whether PCA has removed a rare but meaningful direction. A hybrid system may use PCA for visualisation while retaining selected original variables for the final alert decision.

Evaluating anomaly detectors

When labels exist, useful measures include precision, recall, F1-score, area under the precision–recall curve and cost-weighted loss. For a set of predicted anomalies:

TP denotes true positives, FP false positives and FN false negatives. Precision answers “Of the alerts, how many were genuine?” Recall answers “Of the genuine anomalies, how many were detected?” In highly imbalanced settings, accuracy can be misleading.

When labels are unavailable, evaluation may use expert review, stability over time, simulated faults, alert workload, investigation yield and downstream outcomes. Threshold selection should reflect the relative costs of missed events, unnecessary investigations and delayed action.

7. Applied Case: Illustrative Payment-Fraud Screening

Background

A digital payment provider records transaction amount, merchant category, time, country, device history, account age and recent transaction frequency. Fraud is rare, labels arrive weeks after chargeback investigations, and the business wants to prioritise cases for review.

Problem

The original data contain correlated frequency variables, high-cardinality categorical fields and several identifiers. A direct distance-based method would be dominated by scale and redundant measurements. A fraudster may also imitate common transaction amounts, so a single-variable rule is inadequate.

Analysis and approach

The analytics team first removes customer IDs from the modelling features and creates time-aware training, validation and test periods. Categories are encoded consistently, numerical variables are robustly scaled, and missingness is represented explicitly where it carries operational meaning. Features calculated after the transaction, such as a later chargeback status, are excluded to prevent leakage.

A filter stage removes near-constant fields and one of several strongly redundant frequency measures. PCA is fitted only to the training period for exploratory visualisation and computational comparison. The team then compares three detectors: robust Mahalanobis distance, Isolation Forest and a supervised classifier trained on confirmed historical labels. Because fraud labels are delayed and incomplete, the unsupervised scores are used as additional signals rather than as automatic proof of fraud.

Thresholds are selected using a review-capacity constraint: investigators can examine 500 alerts per day. Validation data are used to choose a score cut-off that fits this capacity while maximising confirmed fraud yield. Alerts are ranked, not immediately declined, and analysts review representative high-, medium- and low-score samples.

Outcome

The combined system improves investigation efficiency because it detects both known fraud patterns and novel combinations of behaviours. However, a high anomaly score is treated as a reason for review, not a final decision. Model performance is monitored by time period, merchant category, geography and customer segment to identify drift and unequal error rates.

Learning points

1. Anomaly detection is often a prioritisation system rather than a binary truth machine.
2. Time-aware splitting is essential when behaviour and fraud strategies evolve.
3. Feature selection and leakage prevention can matter more than selecting a sophisticated algorithm.
4. PCA supports exploration and efficiency but may reduce interpretability.
5. Human review, feedback and governance are part of the technical design.

8. Limitations, Ethics and Governance

Anomaly scores are not inherently explanations. A point may be unusual because of a legitimate change in behaviour, a data pipeline error or an underrepresented population. Automated rejection based solely on an unexplained score can create financial, employment or safety harms.

Important controls include:

• documenting the reference population and definition of normality;
• recording preprocessing, model versions and threshold changes;
• restricting access to sensitive transaction or identity data;
• testing disparate false-positive and false-negative rates;
• providing review and appeal procedures for high-impact decisions;
• monitoring concept drift, data drift and sensor or logging failures;
• preserving an audit trail of alerts and investigator outcomes.

Privacy-preserving aggregation, access controls and retention limits should be considered. Fairness analysis is complicated when anomaly labels are incomplete, but lack of labels is not a reason to ignore possible unequal impacts. Regulators and organisational policies may require transparency, human oversight and documented risk controls.

9. Contemporary Practice and Best Practices

Current practice increasingly combines statistical rules, machine-learning scores, domain constraints and human feedback. Streaming systems update scores as events arrive, while drift detectors identify changes in feature distributions or alert rates. Representation learning can reduce complex data such as text, images and sensor signals, but learned representations may be difficult to audit.

Recommended practice is to:

• begin with a transparent baseline;
• scale and encode variables appropriately;
• fit transformations inside the training process;
• validate thresholds against operational capacity and costs;
• compare reduced and original feature spaces;
• use precision–recall measures for rare events;
• investigate alerts rather than equating unusualness with wrongdoing;
• monitor performance and fairness after deployment;
• document assumptions, limitations and intended use.

Chapter Summary

High-dimensional data create sparsity, computational burden, instability and interpretability problems. Feature selection retains original variables and may use filter, wrapper or embedded methods. PCA transforms centred and, often, standardised variables into orthogonal components ordered by explained variance. Eigenvectors define component directions, while eigenvalues quantify variance in those directions. PCA is useful for compression and visualisation but may discard task-relevant or interpretable information.

Anomaly detection identifies observations that are unusual relative to a defined reference population. Statistical methods use distributional assumptions or robust scores; distance-based methods rely on neighbourhood structure; isolation methods detect observations that are separated quickly by random partitions. Meaningful scaling, leakage prevention, threshold selection and evaluation are essential. In risk, fraud and quality monitoring, anomaly scores should normally support investigation and prioritisation rather than operate as unquestionable decisions. Ethical governance requires privacy protection, fairness monitoring, explanations, human oversight and continuous drift assessment.

Key Terms

Dimensionality reduction: Representing data with fewer variables while preserving useful information.

Curse of dimensionality: Difficulties caused by increasing numbers of features.

Feature selection: Retaining a subset of original variables.

Feature extraction: Creating new variables from existing variables.

Filter method: Feature selection based on general statistical criteria.

Wrapper method: Selection based on repeated evaluation of model performance.

Embedded method: Selection performed during model training.

Regularisation: Penalising model complexity to improve generalisation.

PCA: An unsupervised transformation into orthogonal maximum-variance components.

Principal component: A weighted linear combination of input features.

Eigenvalue: In PCA, the variance associated with a component.

Eigenvector: A direction defining a principal component.

Loading: The weight of an original feature in a component.

Explained variance: The variance represented by a component or set of components.

Anomaly: An observation inconsistent with a reference pattern.

Point anomaly: An individually unusual observation.

Mahalanobis distance: A covariance-adjusted multivariate distance.

Isolation Forest: A tree-based method that isolates unusual observations quickly.

Concept drift: Change over time in the relationship between inputs and outcomes.

Data leakage: Unintended use of information unavailable at prediction time.

Review Questions

Short Answer Questions

1. Why can high dimensionality make distance-based analysis unreliable?
2. Distinguish feature selection from feature extraction.
3. What does an eigenvalue represent in PCA?
4. Why should scaling be considered before PCA or distance-based detection?
5. Define a contextual anomaly and give one example.

Long Answer Questions

1. Explain filter, wrapper and embedded feature-selection methods, including their strengths and limitations.
2. Describe the complete PCA workflow from preprocessing to selecting the number of components.
3. Compare statistical, distance-based and isolation-based anomaly-detection methods.
4. Explain why precision and recall are more informative than accuracy for many fraud-detection problems.
5. Discuss the governance requirements for deploying anomaly detection in a high-impact business process.

Analytical / Application Questions

1. A quality-control dataset contains temperature, pressure, vibration and production speed measured in different units. Design a dimensionality-reduction and anomaly-detection pipeline, specifying preprocessing, validation and threshold decisions.
2. A PCA model explains 95% of total variance but fails to detect a known rare defect. Explain how this can occur and propose two corrective actions.
3. An alerting system produces many false positives for one geographic region. Analyse possible data, modelling and fairness causes, and recommend an investigation plan.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes feature selection from feature extraction?

A. Feature selection creates components, whereas feature extraction retains original variables.

B. Feature selection retains selected original variables, whereas feature extraction creates new variables.

C. Both methods always retain every original variable.

D. Feature extraction can only be used with labelled data.

Correct Answer: B. Feature selection retains selected original variables, whereas feature extraction creates new variables.

Question 2

In PCA, what does the first principal component represent?

A. The feature with the largest mean

B. The direction with the greatest variance subject to unit length

C. The direction with the smallest variance

D. The target variable with the highest correlation

Correct Answer: B. The direction with the greatest variance subject to unit length

Question 3

A covariance matrix has eigenvalues 3 and 1. What proportion of variance is explained by the first component?

A. 0.25

B. 0.50

C. 0.75

D. 3.00

Correct Answer: C. 0.75

Question 4

Why are numerical features commonly standardised before distance-based anomaly detection?

A. To guarantee that all observations are normal

B. To prevent variables with larger numerical scales from dominating distances

C. To remove all correlations between variables

D. To increase the number of dimensions

Correct Answer: B. To prevent variables with larger numerical scales from dominating distances

Question 5

Which method identifies observations that are separated by relatively few random tree partitions?

A. Isolation Forest

B. Ordinary least squares

C. k-means centroid averaging

D. One-hot encoding

Correct Answer: A. Isolation Forest

Question 6

For an anomaly detector, precision is the proportion of:

A. Actual anomalies that were detected

B. Predicted alerts that are genuine anomalies

C. Normal observations that were correctly ignored

D. All observations classified correctly

Correct Answer: B. Predicted alerts that are genuine anomalies

Question 7

What is a major risk of fitting feature selection or PCA before creating validation folds?

A. Underfitting caused by too few observations

B. Data leakage from validation information into model development

C. Automatic removal of the target variable

D. Conversion of numerical features into categories

Correct Answer: B. Data leakage from validation information into model development

Question 8

If the mean is 50 and the standard deviation is 5, what is the z-score for an observation of 65?

A. −3

B. −1

C. 3

D. 13

Correct Answer: C. 3

Question 9

Which condition is especially important when using Mahalanobis distance in its standard form?

A. The covariance matrix must be suitably estimated and invertible

B. All features must be binary

C. There must be no numerical variables

D. The data must contain exactly two observations

Correct Answer: A. The covariance matrix must be suitably estimated and invertible

Question 10

Why should a high anomaly score in payment screening generally not be treated as automatic proof of fraud?

A. Anomaly scores are always random

B. Unusual behaviour may be legitimate, and model errors or bias are possible

C. Fraud never creates unusual behaviour

D. Thresholds cannot be changed after deployment

Correct Answer: B. Unusual behaviour may be legitimate, and model errors or bias are possible