Dimensionality Reduction, Association, and Anomaly Detection

Introduction

Business data increasingly contains hundreds or thousands of variables: transaction attributes, customer interactions, sensor readings, text features, product combinations, and time-stamped operational measures. High dimensionality can improve descriptive richness, but it also creates computational cost, visualisation difficulty, redundant information, unstable estimates, and increased risk of overfitting. Three families of methods are especially useful in business analytics:

• Dimensionality reduction, which represents many variables using fewer informative dimensions.
• Association analysis, which discovers items or events that occur together.
• Anomaly detection, which identifies observations that deviate substantially from expected behaviour.

These methods are complementary. A retailer may reduce customer variables for segmentation, discover frequently co-purchased products, and detect unusual transactions. Their value depends not only on algorithmic performance but also on appropriate preprocessing, validation, interpretation, governance, and business action.

Learning objectives

By the end of this chapter, the reader should be able to:

1. Explain the curse of dimensionality and the purpose of dimensionality reduction.
2. Derive and interpret the principal components produced by PCA.
3. Compare linear PCA with nonlinear manifold methods.
4. Construct and evaluate association rules using support, confidence, and lift.
5. Distinguish statistical, distance-based, density-based, and isolation-based anomaly detection.
6. Design an end-to-end workflow for fraud, risk, and high-dimensional analytics.
7. Evaluate limitations, ethical risks, and deployment requirements.

1. High-dimensional data and the curse of dimensionality

A dataset with n observations and p predictors is high dimensional when p is large relative to n, or when the meaningful structure lies in a small number of latent dimensions despite a large observed p. High dimensionality causes several problems:

• Sparsity: observations become far apart, leaving few nearby examples.
• Distance concentration: nearest and farthest distances can become relatively similar.
• Redundancy: correlated variables add storage and estimation burden without proportional information.
• Visualisation limits: humans cannot directly inspect more than a few dimensions.
• Overfitting: flexible models can learn noise, particularly when p is large and labels are limited.

Scaling is essential before methods based on variance or distance. For variable j, the standardised value is

where xᵢⱼ is observation i on variable j, μⱼ is the sample mean, and σⱼ is the sample standard deviation. This transformation is appropriate when variables have different units and when standard deviation is a meaningful scale. Robust alternatives use the median and interquartile range when extreme values are substantial. Scaling parameters must be fitted on training data only and then applied unchanged to validation and production data.

Dimensionality reduction may be feature selection—retaining a subset of original variables—or feature extraction—creating new variables from the originals. Feature selection preserves interpretability; feature extraction can capture distributed structure more efficiently but may be harder to explain.

2. Principal component analysis

2.1 Concept and derivation

Principal component analysis (PCA) is an unsupervised linear feature-extraction method. It finds orthogonal directions that successively maximise variance. Let X be a centred n × p data matrix. The sample covariance matrix is

where n is the number of observations, p is the number of variables, and Xᵀ is the transpose of X. The units of S are variable-unit products; if X is standardised, S is the correlation matrix.

The first principal direction w₁ solves

Since Var(Xw) = wᵀSw, the constrained optimisation leads to

Thus w₁ is the eigenvector of S associated with the largest eigenvalue λ₁. Subsequent eigenvectors are orthogonal to earlier ones and correspond to decreasing eigenvalues. The component scores are

where W contains selected eigenvectors as columns. The variance of component k is λₖ, and the explained-variance ratio is

The ratios sum to 1 when all p components are retained. PCA assumes that a linear representation is useful and that variance is a reasonable proxy for information. It does not use a target variable; a low-variance direction may nevertheless be highly predictive of a business outcome.

2.2 Interpretation and selection

A loading is an element of an eigenvector and indicates the contribution of an original variable to a component. Large positive loadings indicate variables increasing together along that component; opposite signs indicate contrast. Sign is arbitrary: multiplying an eigenvector by −1 produces the same subspace and fit.

The number of components can be selected using:

• a cumulative explained-variance threshold;
• a scree plot and its elbow;
• cross-validated predictive performance in a downstream model;
• reconstruction error and operational constraints;
• interpretability and stability across samples.

If X is centred and the first q components are retained, the reconstruction is

where Z_q contains q scores and W_q contains q eigenvectors. A reconstruction-error measure is the mean squared Frobenius error

where ||·||_F is the Frobenius norm. This measure evaluates representation fidelity, not business usefulness.

2.3 Worked example

Suppose standardised customer data has three variables: visit frequency, basket value, and discount usage. The covariance matrix has eigenvalues 1.80, 0.90, and 0.30. The total variance is 3.00, so the explained-variance ratios are 60%, 30%, and 10%. Retaining two components preserves 90% of standardised variance. This may be a reasonable visualisation choice, but it does not prove that the two components preserve 90% of predictive information about churn. That question requires validation against churn labels.

2.4 Strengths and limitations

PCA is fast, reproducible, and useful for noise reduction, visualisation, compression, and preprocessing. It can also reduce multicollinearity in downstream regression. However, components may be difficult to name, sensitive to scaling and outliers, and unsuitable when relationships are strongly nonlinear. PCA should be fitted within the training fold during cross-validation; fitting it on the full dataset before validation leaks information from the validation set.

Sparse PCA constrains loadings so that fewer variables contribute to each component, improving interpretability. Kernel PCA maps observations into an implicit nonlinear feature space but introduces kernel selection and computational complexity. For supervised tasks, partial least squares uses outcome information and should not be confused with unsupervised PCA.

3. Manifold learning and visual analytics

A manifold is a lower-dimensional structure embedded in a higher-dimensional space. Manifold methods seek coordinates that preserve selected aspects of local or global geometry.

3.1 Multidimensional scaling

Multidimensional scaling (MDS) seeks low-dimensional points whose pairwise distances resemble observed dissimilarities. Classical MDS is closely related to eigen-decomposition of a transformed distance matrix. It is useful when a meaningful dissimilarity measure exists, including non-Euclidean business distances. Its output depends strongly on the distance definition and dimension.

3.2 Locally linear embedding and Isomap

Locally linear embedding (LLE) represents each point as a weighted combination of its neighbours and seeks low-dimensional coordinates preserving those weights. Isomap constructs a neighbourhood graph and approximates geodesic, or along-manifold, distances before applying MDS. Both methods can reveal curved structure that PCA cannot, but they are sensitive to the number of neighbours, disconnected graphs, noise, and out-of-sample extension.

3.3 t-SNE and UMAP

t-SNE is primarily a visualisation method. It converts local neighbourhoods into probability distributions in the original and low-dimensional spaces and minimises a divergence between them. It often reveals local clusters, but apparent cluster size, spacing, and global distances should not be interpreted naively. Different random seeds and perplexity values can produce different maps.

UMAP constructs a weighted neighbourhood graph and optimises a low-dimensional representation intended to preserve local structure while retaining more global structure than t-SNE in many settings. Nevertheless, UMAP plots are not automatically evidence of statistically distinct populations. Both methods should be used for exploration, not as unvalidated proof of segmentation or causality.

3.4 Practical manifold-learning protocol

1. Remove identifiers and leakage variables.
2. Impute missing values using a training-only procedure.
3. Scale variables or use a justified distance metric.
4. Reduce extreme noise and duplicate records.
5. Compare several parameter settings and random seeds.
6. Colour plots only by variables available at the intended decision time.
7. Validate discovered groups with stability tests and business outcomes.

A visualisation can support hypothesis generation; it cannot replace statistical testing, domain review, or prospective validation.

4. Association rule mining

Association analysis discovers co-occurrence patterns in transactional or event data. Let I be a set of items and T a transaction containing a subset of I. A rule has the form A → B, where A and B are disjoint itemsets.

4.1 Core measures

Support measures prevalence:

where N is the number of transactions. Confidence measures conditional frequency:

Lift compares the observed co-occurrence with independence:

Lift greater than 1 indicates positive association, lift equal to 1 is consistent with independence, and lift below 1 indicates negative association. These are descriptive measures, not evidence that A causes B.

Additional measures include leverage,

and conviction,

where conviction can become undefined when confidence equals 1. Rare-item rules can have high lift but little operational value, so support, effect size, statistical significance, stability, and profitability should be considered together.

4.2 Worked example

In 1,000 supermarket baskets, coffee appears in 200, biscuits in 300, and both in 120. For coffee → biscuits:

• Support = 120/1,000 = 0.12, or 12%.
• Confidence = 120/200 = 0.60, or 60%.
• Lift = 0.60/0.30 = 2.00.

Thus, 12% of all baskets contain both products, and biscuits occur twice as often among coffee buyers as in the overall population. The result does not establish that buying coffee causes biscuit purchases. Promotions, seasonality, store location, or customer type may explain the pattern.

4.3 Apriori and FP-Growth

Apriori uses the downward-closure property: if an itemset is infrequent, every larger itemset containing it is also infrequent. It therefore generates candidate itemsets level by level and prunes those below minimum support. This reduces search but can still be expensive with many items.

FP-Growth compresses transactions into an FP-tree and mines frequent patterns without generating the same volume of explicit candidates. It is often more efficient for dense data. Both methods require carefully selected minimum-support and minimum-confidence thresholds. Thresholds should reflect business cost, inventory constraints, and action capacity rather than being chosen solely for computational convenience.

4.4 Statistical and operational cautions

Mining thousands of rules creates multiple-comparison risk: some apparently strong rules occur by chance. Use holdout periods, bootstrap stability, false-discovery controls where appropriate, and domain review. Avoid recommending a rule merely because it is statistically strong; assess margin, cannibalisation, stock availability, customer experience, and fairness.

5. Anomaly detection

An anomaly is an observation that is inconsistent with an expected pattern. Terminology distinguishes:

• Point anomaly: one observation is unusual by itself.
• Contextual anomaly: normal in one context but unusual in another, such as a transaction at an unusual hour.
• Collective anomaly: a sequence or group is unusual even when individual points appear normal.

Anomaly detection may be supervised, semi-supervised, or unsupervised. Fraud labels are often incomplete because undetected fraud is unknown and confirmed cases are delayed. Consequently, an unsupervised score is usually a prioritisation signal, not a final fraud verdict.

5.1 Statistical methods

For an approximately normal univariate variable X with mean μ and standard deviation σ, a standardised score is

A large absolute z-score may indicate unusualness, but normality, independence, stable parameters, and an appropriate reference population are required. In multivariate settings, the Mahalanobis squared distance is

where x is a p-dimensional observation, μ is the mean vector, and Σ is the covariance matrix. If the multivariate normal model is appropriate and parameters are known, D² approximately follows a χ² distribution with p degrees of freedom. In practice, estimates of μ and Σ can be distorted by outliers; robust covariance estimation may be preferable. High dimensionality can also make Σ singular or unstable, requiring regularisation or dimension reduction.

5.2 Distance and density methods

k-nearest-neighbour methods score an observation using its distance to nearby points. Local Outlier Factor compares local density around an observation with the density of its neighbours. These methods can detect local anomalies that global thresholds miss, but results depend on scaling, distance choice, k, and computational cost. They are problematic when normal behaviour contains several groups with very different densities.

5.3 Isolation Forest

Isolation Forest repeatedly selects a random feature and a random split value to partition observations. Anomalies tend to be isolated in fewer splits than ordinary points. If E(h(x)) is the mean path length for observation x across isolation trees, the commonly used score is

where ψ is the subsample size and c(ψ) is the expected path length of an unsuccessful search in a binary search tree, used for normalisation. Higher scores indicate easier isolation. The score is algorithmic rather than a calibrated probability of fraud. Thresholds must be selected using operational capacity, investigation costs, and labelled validation where available.

5.4 One-class classification and autoencoders

One-class SVM estimates a boundary around training examples assumed to represent normality. It can model nonlinear boundaries through kernels but is sensitive to scaling and hyperparameters. Autoencoders learn to reconstruct inputs; a high reconstruction error may indicate an anomaly. However, a powerful autoencoder may reconstruct anomalous examples well, and a reconstruction threshold is not automatically a probability or a causal explanation. Neural methods require adequate data, monitoring, and careful protection against training contamination.

6. Integrated workflow for high-dimensional analytics

A defensible project follows a decision-centred sequence:

1. Define the decision: investigation queue, visual exploration, recommendation, or risk control.
2. Specify the unit and time: customer, transaction, account, device, or event; define prediction and observation windows.
3. Audit data quality: duplicates, missingness, leakage, drift, inconsistent units, and access rights.
4. Create a baseline: simple rules or statistical summaries provide a performance and interpretability reference.
5. Preprocess correctly: fit imputation, scaling, PCA, and anomaly models on training data only.
6. Choose the method: PCA or manifold learning for representation; association mining for co-occurrence; anomaly detection for deviation.
7. Validate: use temporal or grouped splits, stability analysis, labelled metrics where possible, and expert review.
8. Translate scores into actions: alerts should specify owner, evidence, response time, and escalation path.
9. Monitor: track drift, alert volume, precision, investigation yield, false positives, and changing data quality.
10. Govern: document features, thresholds, model versions, retention, explanations, and override decisions.

For rare-event detection, accuracy is misleading. If only 0.1% of transactions are fraudulent, a model that labels every transaction legitimate is 99.9% accurate but useless. Precision is

and recall is

where TP, FP, and FN denote true positives, false positives, and false negatives. Precision–recall curves are often more informative than ROC curves under severe class imbalance. At a fixed investigation budget, precision among the top-k alerts and expected prevented loss may be the most relevant measures.

7. Illustrative case: monitoring payment fraud

Background

A digital payments firm processes millions of transactions monthly. It has confirmed fraud labels, but labels arrive weeks later and many fraudulent transactions remain undiscovered. Transaction value, merchant, device, location, time, account age, and velocity variables are available.

Problem

The existing rule system generates too many alerts during holiday periods and misses coordinated attacks distributed across accounts. Analysts need a ranked queue rather than an automatic accusation.

Analysis

The analytics team first defines a transaction-level observation window and prevents use of information created after the transaction. Highly skewed monetary variables are transformed and robustly scaled. PCA is used only for exploratory visualisation of correlated behavioural features; its components are not treated as inherently meaningful risk factors. An Isolation Forest produces an unsupervised anomaly score, while a supervised gradient-boosting model uses confirmed historical labels. Association mining identifies combinations such as device, merchant category, and unusual time windows that occur disproportionately in reviewed fraud cases.

Approach and solution

The production system combines:

• a transparent rules layer for known high-risk conditions;
• a supervised probability model where labels are reliable;
• an anomaly score for novel behaviour;
• association-derived features used only after temporal validation;
• a human review queue ranked by expected loss and investigation capacity.

The team evaluates precision at the daily review capacity, recall over a fixed fraud-confirmation horizon, prevented loss, customer-friction rate, and subgroup performance. It uses a time-based holdout rather than a random split, because random splitting can place nearly identical campaigns in both training and test sets.

Outcome

In this illustrative scenario, the integrated system reduces low-value alerts and improves the proportion of reviewed alerts later confirmed as fraud. The anomaly component is particularly useful for emerging attack patterns, while the supervised model performs better on recurring patterns. Analysts retain authority to overturn alerts, and overrides are logged for future model review.

Learning points

The case demonstrates that dimensionality reduction, association rules, and anomaly scores are not interchangeable. A visual cluster is not a fraud label, a high-lift association is not causation, and an anomaly score is not a probability. Effective deployment combines complementary evidence with temporal validation, human accountability, and continuous monitoring.

8. Limitations, ethics, and governance

Data and statistical limitations. Missing data may itself encode process behaviour. Outliers can be errors, rare legitimate events, or fraud. PCA and distance methods are sensitive to scaling and covariance estimates. Association rules can reflect confounding and selection bias. Unsupervised anomaly methods lack a definitive ground truth and may reproduce historical surveillance bias.

Fairness and proportionality. If location, device, language, or behavioural proxies correlate with protected characteristics, an anomaly system may impose disproportionate scrutiny. Organisations should test alert rates, false-positive rates, review outcomes, and customer impact across relevant groups, subject to lawful and privacy-preserving governance.

Privacy and security. Use data minimisation, purpose limitation, access controls, retention schedules, encryption, and documented lawful bases. Association mining can reveal sensitive relationships; anomaly systems can expose behavioural profiles. Security controls should protect both training data and model outputs.

Explainability and contestability. Investigators and affected customers need understandable reasons for an alert where required by policy or law. A component loading, rule metric, nearest neighbour, or isolation path can provide evidence, but explanations should not imply certainty. Provide review, appeal, and correction mechanisms.

Drift and feedback loops. Fraudsters adapt, products change, and customer behaviour shifts. If only alerted cases are investigated, future labels become selection-biased. Monitor population stability, feature distributions, alert rates, confirmed-event rates, and performance by time period. Recalibration and retraining should be controlled changes, not automatic reactions to every fluctuation.

9. Contemporary developments and best practices

Modern representation learning can produce embeddings for text, images, graphs, and event sequences, followed by clustering or anomaly detection. Graph-based methods are valuable when relationships among accounts, devices, merchants, or suppliers matter more than individual records. Contrastive learning and self-supervised learning can exploit abundant unlabelled data. However, these methods increase opacity, computational requirements, and governance burden.

Recommended practice is to:

• begin with a clear decision and a transparent baseline;
• separate exploration from confirmatory evaluation;
• use training-only preprocessing and temporal validation;
• report stability, not just a single metric;
• calibrate scores when probabilities are required;
• assess business cost, customer harm, and analyst capacity;
• retain human review for consequential decisions;
• document assumptions, versions, thresholds, and limitations;
• monitor drift and investigate feedback loops.

Chapter Summary

High-dimensional business data requires methods that simplify representation, discover co-occurrence, and identify unusual behaviour. PCA creates orthogonal linear components ordered by variance and is valuable for compression and visualisation, but its components depend on scaling and may not preserve predictive information. Manifold methods can reveal nonlinear neighbourhood structure, although their maps are sensitive to parameters and should not be overinterpreted.

Association rules quantify co-occurrence using support, confidence, lift, leverage, and related measures. They support recommendations and operational discovery but do not establish causation and may generate spurious rare-item patterns. Anomaly detection includes statistical scores, Mahalanobis distance, nearest-neighbour and density methods, Isolation Forest, one-class models, and autoencoders. Scores require appropriate preprocessing, threshold design, validation, and human interpretation.

A professional workflow defines the decision and time boundary, prevents leakage, validates using realistic splits, evaluates rare-event metrics, monitors drift, and governs privacy, fairness, explainability, and accountability. The strongest applications combine complementary methods rather than treating any single projection, rule, or score as truth.

Key Terms

Association rule: An implication A → B describing co-occurrence between itemsets.

Anomaly: An observation or pattern that deviates from an expected distribution or context.

Confidence: The conditional frequency P(B | A) of an association rule.

Curse of dimensionality: Difficulties caused by sparsity, distance concentration, computation, and overfitting in high-dimensional spaces.

Dimensionality reduction: Representing data using fewer variables or dimensions.

Explained-variance ratio: The proportion of total variance attributed to a principal component.

Feature extraction: Construction of new variables from existing variables.

Feature selection: Retention of a subset of original variables.

Isolation Forest: A tree-based method that identifies observations isolated by short random partition paths.

Lift: The ratio of rule confidence to consequent support.

Loading: The coefficient of an original variable in a principal component.

Manifold: A potentially lower-dimensional structure embedded in a higher-dimensional space.

Mahalanobis distance: A covariance-adjusted multivariate distance from a reference mean.

PCA: Principal component analysis, an unsupervised linear dimensionality-reduction method.

Precision: The proportion of predicted positives that are true positives.

Recall: The proportion of actual positives that are detected.

Support: The proportion of transactions containing an itemset.

t-SNE: A nonlinear visualisation method focused mainly on preserving local neighbourhoods.

UMAP: A graph-based manifold visualisation and embedding method.

Review Questions

Short Answer Questions

1. Why should variables commonly be scaled before PCA or distance-based anomaly detection?
2. Define support, confidence, and lift for an association rule.
3. What is the difference between a point anomaly and a contextual anomaly?
4. Why does PCA not necessarily preserve predictive information?
5. Why is accuracy unsuitable as the sole metric for rare-event fraud detection?

Long Answer Questions

1. Derive the PCA eigenvalue problem from the objective of maximising projected variance under a unit-length constraint. Explain how eigenvalues and loadings are interpreted.
2. Compare PCA, MDS, t-SNE, and UMAP in terms of geometry, interpretability, stability, and appropriate business use.
3. Explain the Apriori principle and compare Apriori with FP-Growth, including computational and operational considerations.
4. Compare Mahalanobis distance, Local Outlier Factor, Isolation Forest, and autoencoders for anomaly detection.
5. Design a governed end-to-end anomaly-detection workflow for a financial-services organisation.

Analytical / Application Questions

1. A rule has support 0.08, antecedent support 0.20, and consequent support 0.25. Calculate its confidence and lift, and interpret both values.
2. A fraud team can investigate only 500 alerts per day. Explain which evaluation measures and validation design you would use to select an anomaly-ranking model.
3. A UMAP plot shows three visually separated customer groups. Develop a rigorous plan to determine whether the groups are stable, commercially meaningful, and suitable for targeted action.

Multiple Choice Questions with Answers

Question 1

Which statement best describes the first principal component?

A. It is the variable with the largest original variance.

B. It is a direction that maximises projected variance subject to a unit-length constraint.

C. It is the direction that maximises the target variable's accuracy.

D. It is always the most interpretable business dimension.

Correct Answer: B. It is a direction that maximises projected variance subject to a unit-length constraint.

Question 2

A rule has support 0.12 and antecedent support 0.20. What is its confidence?

A. 0.06

B. 0.20

C. 0.60

D. 1.67

Correct Answer: C. 0.60

Question 3

If a rule has confidence 0.60 and consequent support 0.30, what is its lift?

A. 0.18

B. 0.30

C. 1.50

D. 2.00

Correct Answer: D. 2.00

Question 4

Which measure compares observed co-occurrence with the co-occurrence expected under independence?

A. Support

B. Confidence

C. Lift

D. Recall

Correct Answer: C. Lift

Question 5

What is a key limitation of t-SNE visualisations?

A. They can only process labelled data.

B. Global distances and cluster sizes may be misleadingly interpreted.

C. They require all variables to have identical units.

D. They cannot represent local neighbourhoods.

Correct Answer: B. Global distances and cluster sizes may be misleadingly interpreted.

Question 6

In Mahalanobis distance, why is the covariance matrix used?

A. To convert categorical variables into labels

B. To adjust distances for scale and correlation among variables

C. To maximise association-rule confidence

D. To remove the need for validation

Correct Answer: B. To adjust distances for scale and correlation among variables

Question 7

In an Isolation Forest, an observation with a shorter average path length is generally considered:

A. More anomalous

B. More typical

C. More highly labelled

D. More correlated

Correct Answer: A. More anomalous

Question 8

Which metric is especially important when evaluating a rare-event alert system under a fixed investigation capacity?

A. Precision among the top-ranked alerts

B. Training-set accuracy only

C. Number of principal components

D. Average itemset size

Correct Answer: A. Precision among the top-ranked alerts

Question 9

Which statement about association rules is correct?

A. High lift proves that the antecedent causes the consequent.

B. Confidence is always symmetric between A → B and B → A.

C. High lift can occur for a rare rule with limited operational value.

D. Support is calculated only among transactions containing the antecedent.

Correct Answer: C. High lift can occur for a rare rule with limited operational value.

Question 10

What is a serious form of leakage in a temporal fraud model?

A. Standardising a variable using training-set means

B. Using information created after the transaction being scored

C. Evaluating precision at the investigation budget

D. Using a time-based holdout set

Correct Answer: B. Using information created after the transaction being scored