8. Unsupervised Learning and Representation Methods

Introduction and Context

Most predictive models are trained with labelled examples: each observation is associated with a known outcome. In many important settings, however, labels are unavailable, incomplete, expensive, or unreliable. A retailer may possess millions of customer transactions without a trustworthy customer segment. A manufacturer may record sensor measurements without knowing which observations represent equipment failure. A financial institution may have large volumes of transactions but only a small number confirmed as fraudulent. Unsupervised learning addresses such situations by identifying structure in data without a predefined target variable.

Unsupervised learning is not simply a substitute for supervised learning. It serves a different analytical purpose. Rather than estimating an outcome directly, it helps discover groups, unusual observations, latent dimensions, associations, and useful representations. These discoveries can support exploratory analysis, customer segmentation, data quality assessment, feature engineering, visualization, compression, and subsequent predictive modelling.

Representation methods are particularly important because raw data are often high-dimensional, noisy, sparse, or difficult to interpret. A representation transforms observations into a new coordinate system that preserves information relevant to analysis while reducing redundancy or revealing hidden structure. Principal component analysis (PCA), manifold learning, autoencoders, and embeddings are examples of such methods.

The central challenge is interpretive. Since there is often no labelled answer, the analyst must determine whether a discovered pattern is meaningful, stable, and useful rather than merely an artefact of scale, noise, sampling, or algorithmic assumptions.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Explain the purpose, assumptions, and terminology of unsupervised learning.

2. Select and apply clustering methods for appropriate data structures and business questions.

3. Interpret PCA and related dimensionality-reduction techniques.

4. Design anomaly-detection workflows and distinguish anomalies from legitimate rare cases.

5. Compare unsupervised methods using statistical, visual, operational, and business criteria.

6. Use representation learning to improve visualization and predictive pipelines.

7. Identify common methodological, ethical, and governance risks in unsupervised analysis.

The Unsupervised Learning Problem

Let the data be represented by a matrix X containing n observations and p features. In supervised learning, a target vector y guides model fitting. In unsupervised learning, the analyst seeks a function, partition, density, or lower-dimensional representation based primarily on X. The result may be a set of clusters, anomaly scores, latent variables, or transformed features.

Several concepts recur across methods:

• Similarity and distance: A measure of how alike two observations are. Euclidean distance is common for continuous variables, while Manhattan, cosine, Jaccard, or Gower distance may be more suitable in other settings.
• Cluster: A set of observations that are more similar to one another than to observations outside the set, according to a chosen criterion.
• Centroid: A representative mean vector used by some partitioning algorithms.
• Density: The concentration of observations in a region of feature space.
• Latent variable: An unobserved factor inferred from observed variables.
• Dimensionality reduction: Transformation of data from a high-dimensional space into fewer dimensions.
• Anomaly or outlier: An observation whose behaviour is substantially inconsistent with a reference pattern.
• Embedding: A numerical representation designed to place semantically or structurally related objects near one another.

Preprocessing is often more consequential than the choice of algorithm. Variables measured in large units can dominate distance calculations; skewed variables can distort centroids; missing values can create artificial separation; and categorical variables require suitable encoding or distance functions. Standardization, robust scaling, log transformation, imputation, feature selection, and leakage control should therefore precede modelling when justified.

Clustering as Pattern Discovery

Clustering seeks a partition or hierarchical organization of observations. There is no universally correct clustering because the answer depends on the similarity measure, feature representation, scale, and purpose. A useful cluster solution is one that is reasonably stable, interpretable, operationally actionable, and aligned with the decision problem.

Partitioning Methods

K-means is one of the most widely used algorithms. Given a selected number of clusters k, it assigns each observation to the nearest centroid and repeatedly updates centroids to minimize within-cluster squared Euclidean distance:

J = Σᵢ ||xᵢ − μ_{c(i)}||².

Its strengths are speed, simplicity, and suitability for large numerical datasets. Its limitations are equally important: it assumes compact, roughly spherical clusters of comparable scale, is sensitive to outliers and initialization, and requires k to be specified. Multiple initializations are therefore advisable.

K-medoids resembles K-means but uses actual observations as cluster representatives. Because medoids are less affected by extreme values and can be based on arbitrary distance measures, this method is useful when robust representatives or mixed-type distances are required, although it is usually more computationally expensive.

Hierarchical Clustering

Hierarchical clustering creates a tree of nested groups, called a dendrogram. In agglomerative clustering, each observation begins as a separate group; the closest groups are successively merged. The result depends on the linkage rule. Single linkage uses the closest pair and can produce chaining. Complete linkage uses the most distant pair and tends to form compact groups. Average linkage uses average pairwise distance, while Ward’s method merges groups with the smallest increase in within-group variance.

Hierarchical methods are useful when the analyst wants to inspect solutions at several levels rather than commit immediately to one number of clusters. They can be computationally demanding for very large datasets and are sensitive to the chosen distance and linkage criteria.

Density-Based Clustering

DBSCAN defines dense regions separated by areas of low density. It uses a neighbourhood radius and a minimum number of observations. Points in sufficiently dense regions form clusters, while isolated observations may be labelled noise. DBSCAN can discover non-spherical shapes and identify noise without requiring the number of clusters in advance. However, it struggles when clusters have substantially different densities or when the feature space is very high-dimensional.

HDBSCAN extends the density-based idea by constructing a hierarchy of density-connected groups and selecting relatively stable clusters. It is often more flexible than DBSCAN, particularly when density varies, but its output still requires careful interpretation.

Probabilistic and Soft Clustering

Gaussian mixture models assume that observations arise from a mixture of probability distributions, commonly multivariate Gaussian components. Rather than assigning an observation rigidly to one cluster, the model estimates a membership probability for each component. This is useful when group boundaries overlap. Model selection can use criteria such as AIC or BIC, although statistical fit does not automatically imply business usefulness.

Soft membership is valuable in applications such as customer segmentation, where an individual may reasonably resemble more than one segment. Probabilities should not be interpreted as objective truths; they depend on the model and its assumptions.

Choosing and Evaluating Clusters

The number of clusters should not be selected mechanically. The elbow method examines the reduction in within-cluster variation as k increases, looking for a point beyond which improvement becomes marginal. The silhouette coefficient compares an observation’s cohesion within its own cluster with its separation from the nearest alternative cluster. Values closer to one indicate better separation, while values near zero suggest overlap. The Davies–Bouldin index provides another internal criterion, with lower values generally preferred.

Internal metrics are only part of evaluation. Stability testing involves refitting the algorithm on bootstrap samples, time periods, or perturbed data and assessing whether similar groups emerge. External validation may use known outcomes not used during clustering, such as later retention, response, or equipment failure. Such outcomes should be used carefully: a cluster is not automatically valuable merely because it correlates with a target.

A practical clustering workflow is:

1. Define the decision or exploratory purpose.

2. Audit variables, missingness, outliers, and measurement scales.

3. Select a meaningful representation and distance measure.

4. Standardize or transform features where appropriate.

5. Fit several plausible algorithms or parameter settings.

6. Examine internal quality, stability, and interpretability.

7. Profile clusters using variables not necessarily used in fitting.

8. Test whether the groups support a real intervention or decision.

9. Monitor whether clusters remain meaningful as data and behaviour change.

Principal Component Analysis and Linear Representation

PCA transforms correlated variables into a smaller set of uncorrelated linear combinations called principal components. The first component captures the greatest possible variance subject to a unit-length constraint; each subsequent component captures the greatest remaining variance while being orthogonal to earlier components.

If X is centred and represented by covariance matrix S, PCA obtains eigenvectors of S. Equivalently, singular value decomposition can be applied directly to the centred or standardized data. The component scores are the coordinates of observations in the new system, while the loadings describe the contribution of original variables to each component.

PCA is useful for visualization, noise reduction, compression, multicollinearity management, and feature construction. The proportion of variance explained by a component is its eigenvalue divided by the sum of all eigenvalues. A scree plot can help identify a useful dimensionality, but the retained number should also reflect interpretability and predictive performance.

Scaling changes PCA substantially. If variables have different units, covariance-based PCA may be dominated by high-variance variables. Standardized PCA gives each variable comparable initial influence, though this is not always scientifically appropriate. PCA is also sensitive to outliers and captures linear structure; it does not necessarily preserve rare but important patterns.

Components are mathematical directions, not automatically meaningful causes. Interpretation requires examination of loadings, variable relationships, domain knowledge, and sensitivity to preprocessing choices. A component with high explained variance is not necessarily the most useful feature for prediction.

Nonlinear Representation Methods

Many datasets lie near a lower-dimensional nonlinear manifold. Methods such as t-distributed stochastic neighbour embedding (t-SNE) and Uniform Manifold Approximation and Projection (UMAP) are designed primarily for visualization. They attempt to preserve local neighbourhood relationships, although the resulting axes generally have no direct substantive meaning. Apparent gaps or clusters in a two-dimensional plot should not be treated as proof of natural groups. Results can vary with initialization, perplexity, neighbourhood size, metric, and random seed.

Autoencoders are neural networks trained to reconstruct input data through a constrained bottleneck. An encoder maps x to a latent representation z, and a decoder reconstructs x from z. Reconstruction loss encourages the latent code to retain information needed for recovery. Variants include denoising autoencoders, variational autoencoders, and sparse autoencoders. They can represent complex nonlinear structure and support image, text, sensor, and transaction applications. Their flexibility increases the need for validation, regularization, interpretability, and careful separation of training and evaluation data.

Embeddings represent objects such as words, products, users, or documents as vectors. Similarity in the embedding space reflects relationships learned from co-occurrence, interaction, or other context. Embeddings can support clustering, recommendation, retrieval, and downstream prediction, but they may encode historical bias and should not be interpreted as neutral measures of meaning.

Anomaly Detection

Anomaly detection identifies observations that differ substantially from a reference distribution or behavioural pattern. The reference may be global, local, temporal, or conditional. This distinction matters: a transaction can be normal globally but anomalous for a particular account, location, or time of day.

Statistical methods include z-scores, robust deviations based on the median and median absolute deviation, and multivariate distance such as Mahalanobis distance. These methods are transparent but depend on distributional assumptions and covariance estimation.

Isolation Forest detects anomalies by measuring how quickly observations become isolated through random partitioning. Anomalies tend to require fewer partitions. One-class support vector machines learn a boundary around typical observations, while local outlier factor compares the density of an observation with the density of its neighbours. Autoencoder-based detection uses reconstruction error, treating unusually poorly reconstructed records as potential anomalies.

Anomaly scores require a decision threshold. A threshold should reflect investigation capacity, the cost of false positives, the cost of missed anomalies, and the expected prevalence of events. In many operational systems, anomaly detection is a prioritization mechanism rather than an automatic accusation. Human review, feedback loops, and threshold monitoring are essential.

Applications Across Organizations

Unsupervised methods have broad uses. Marketing teams can segment customers according to behaviour, value, and engagement. Banks can identify unusual transactions, discover borrower profiles, and detect shifts in portfolio behaviour. Manufacturers can group machines by operating signatures and flag deviations from normal sensor patterns. Healthcare researchers can explore patient phenotypes, compress clinical variables, or identify unusual physiological patterns, subject to strong privacy and clinical validation requirements. Cybersecurity teams can identify unusual network flows and previously unseen attack patterns. Operations managers can reduce thousands of measures to a few interpretable dimensions for monitoring and reporting.

In predictive pipelines, unsupervised features may be used as inputs to supervised models. Cluster labels, distances to centroids, anomaly scores, PCA scores, and learned embeddings can enrich a classifier or regression model. To prevent leakage, representation learning must be fitted only on the training portion of each validation split. If PCA or clustering is fitted using the entire dataset before evaluation, information from the test set may influence the learned representation and produce optimistic results.

Illustrative Case Study: Industrial Sensor Monitoring

Background

Consider an illustrative manufacturing company operating several hundred industrial pumps. Each pump produces temperature, vibration, pressure, flow, and power measurements at regular intervals. Confirmed failure labels are rare, while normal-operation records are abundant.

Situation and Problem

The engineering team wants to identify operating regimes, detect early deviations, and improve a later failure-prediction model. A single global threshold is unsuitable because pumps operate under different loads and environmental conditions.

Analysis

The team first removes impossible sensor readings, investigates missingness, aligns measurements by time, and creates rolling statistical features. Features are robustly scaled because vibration and power contain extreme values. Correlation analysis reveals substantial redundancy among several sensors.

PCA is fitted on the training period to create a compact representation. The first components are examined through their loadings: one reflects thermal behaviour, another reflects mechanical vibration, and another reflects load-related pressure and flow. These interpretations are checked with engineers rather than assumed from variance alone.

K-means and HDBSCAN are then compared. K-means provides stable operating regimes when the data are standardized, while HDBSCAN identifies a small set of sparse operating conditions and noise points. Cluster stability is assessed across weeks and pump types. An isolation forest is fitted within each major operating regime, rather than across all pumps, to avoid labelling normal high-load operation as anomalous.

Approach and Solution

The monitoring system produces three outputs: the current operating-regime label, the distance from the regime prototype, and a regime-specific anomaly score. These features are supplied to a supervised failure model trained using time-based validation. Alerts are ranked by severity and persistence; a single unusual reading does not automatically trigger a maintenance order.

Outcome

In this illustrative scenario, the combined representation makes sensor relationships easier to visualize, reduces redundant variables, and allows maintenance staff to compare a pump with appropriate peers. The main benefit is not the cluster label itself but the contextualized anomaly score and improved diagnostic workflow.

Learning Points

The case demonstrates that unsupervised learning is most effective when connected to an operational decision. It also illustrates the importance of temporal validation, domain interpretation, regime-specific modelling, and human review. A mathematically elegant segmentation would have little value if engineers could not act on it.

Challenges, Risks, and Common Mistakes

A frequent mistake is assuming that every algorithmically generated group is a real natural category. Clusters may reflect measurement scale, missing-data patterns, collection procedures, or a single dominant variable. Another error is using Euclidean distance for mixed categorical and numerical data without justification.

High dimensionality creates distance concentration: observations may appear similarly distant from one another, weakening neighbourhood methods. Dimensionality reduction can help, but reducing dimensions before understanding the data may remove information relevant to rare events. Outliers may be either valuable discoveries or data errors; deleting them automatically can conceal the phenomenon being studied.

Analysts should avoid choosing the number of clusters solely because it produces a visually attractive plot. They should also avoid interpreting t-SNE or UMAP axes as quantitative scales, treating PCA components as causal factors, or converting anomaly scores directly into claims of fraud or misconduct.

Drift is another major concern. Customer behaviour, sensor conditions, market conditions, and attack strategies change over time. A representation learned from historical data may become obsolete. Monitoring should therefore include cluster proportions, distance distributions, anomaly rates, reconstruction error, and operational outcomes.

Ethical, Governance, and Professional Considerations

Unsupervised systems can affect access to services even when they do not use an explicit target. A customer segment may become a basis for pricing, eligibility, or differential treatment. Proxies for protected characteristics can enter through location, purchasing behaviour, language, or device information. Consequently, segmentation and anomaly detection require fairness review, explainability appropriate to the use case, documentation, and human oversight.

Privacy protection is especially important when embeddings or clusters are created from individual behaviour. Data minimization, access controls, retention limits, secure processing, and applicable data-protection requirements should be addressed. Governance documentation should record data sources, preprocessing, distance measures, algorithm settings, validation procedures, known limitations, and intended uses.

A professional system distinguishes an investigative signal from a confirmed fact. Alerts should support review rather than impose unjustified conclusions. Where automated decisions have significant consequences, organizations should provide escalation, contestability, and audit mechanisms.

Emerging Directions and Best Practices

Current practice increasingly combines self-supervised representation learning with supervised fine-tuning. Contrastive learning attempts to bring related examples closer and separate dissimilar examples without requiring conventional labels. Foundation models produce reusable representations for text, images, audio, and multimodal data, but their embeddings may be difficult to interpret and may inherit biases from training corpora.

Scalable vector databases now support similarity search over large embedding collections. Streaming and online algorithms are becoming important for continuously changing data, while federated and privacy-preserving approaches seek to reduce centralization of sensitive information. These developments do not remove the need for sound statistical reasoning.

Best practice is to begin with the decision, not the algorithm. Use domain-informed features and an appropriate distance measure; document transformations; fit preprocessing within validation folds; compare methods; test stability; inspect representative and borderline observations; validate against operational outcomes; monitor drift; and maintain a clear distinction between discovery, prediction, and decision-making.

Chapter Summary

Unsupervised learning discovers structure without a predefined target. Clustering methods include partitioning, hierarchical, density-based, probabilistic, and soft approaches, each relying on different assumptions about similarity and group shape. Cluster quality should be assessed through internal metrics, stability, domain interpretation, and practical usefulness.

PCA provides a linear, variance-oriented representation that can reduce redundancy, support visualization, and create predictive features. Nonlinear methods, autoencoders, and embeddings represent more complex structures but require careful validation and interpretation. Anomaly detection identifies observations that depart from a reference pattern; its success depends on context, thresholds, false-alarm costs, and human review.

Across applications, preprocessing, leakage prevention, monitoring, ethical governance, and actionability are as important as algorithm selection. An unsupervised result becomes professionally valuable when it is stable, interpretable, responsibly governed, and connected to a legitimate decision or research question.

Key Terms

Unsupervised learning: Learning patterns from data without a specified labelled target.

Clustering: Grouping observations according to a similarity or distance criterion.

Distance measure: A mathematical function quantifying dissimilarity between observations.

Centroid: The mean vector representing a cluster in centroid-based clustering.

Medoid: An actual observation selected as the representative of a cluster.

K-means: A partitioning algorithm minimizing within-cluster squared Euclidean distance.

Hierarchical clustering: Clustering that creates nested groups represented by a dendrogram.

Density-based clustering: Clustering based on concentrations of observations separated by sparse regions.

Gaussian mixture model: A probabilistic model representing data as a mixture of component distributions.

Silhouette coefficient: A measure comparing within-cluster cohesion with nearest-cluster separation.

Principal component: An orthogonal linear direction capturing variance in the data.

Loading: The contribution or weight of an original variable to a principal component.

Dimensionality reduction: Transforming data into fewer dimensions while retaining selected information.

Manifold learning: Learning a lower-dimensional nonlinear structure embedded in a higher-dimensional space.

Embedding: A vector representation designed to preserve meaningful relationships among objects.

Anomaly detection: Identifying observations that depart from an expected reference pattern.

Isolation Forest: An anomaly method based on the ease with which observations are isolated by random partitions.

Concept drift: Change over time in the data-generating process or relationships.

Data leakage: Use of information during training that would not be available at prediction time.

Review Questions

Short Answer Questions

1. Why is preprocessing particularly important before distance-based clustering?

2. Distinguish between a centroid and a medoid.

3. What information do PCA loadings provide?

4. Why should anomaly detection be contextual rather than purely global in many applications?

5. What is the purpose of testing clustering stability?

Long Answer Questions

1. Compare K-means, hierarchical clustering, DBSCAN, and Gaussian mixture models with respect to assumptions, outputs, and appropriate uses.

2. Explain PCA mathematically and discuss the effects of centring, scaling, and outliers.

3. Describe a complete workflow for developing an anomaly-detection system in an industrial setting.

4. Discuss how unsupervised representations can improve a supervised predictive pipeline and explain how leakage can occur.

5. Evaluate the ethical and governance issues associated with customer segmentation and behavioural anomaly detection.

Analytical / Application Questions

1. A dataset contains income, age, transaction count, region, and product category. Which distance measure, preprocessing steps, and clustering methods would you consider, and why?

2. A two-dimensional UMAP plot shows three visually distinct groups, but the groups are unstable across random seeds. How should the analyst interpret and investigate this result?

3. An anomaly detector produces many alerts after a change in sensor calibration. Design a diagnostic and remediation plan that distinguishes data-quality problems from genuine operational anomalies.

22. Multiple Choice Questions with Answers

Question 1

Which situation best reflects an appropriate use of unsupervised learning?

A. Estimating house prices from labelled historical sales

B. Discovering customer groups when no reliable segment labels exist

C. Testing a medical treatment using randomized outcomes

D. Predicting loan default from confirmed default records

Correct Answer: B. Discovering customer groups when no reliable segment labels exist

Question 2

A major limitation of standard K-means clustering is that it

A. cannot calculate distances between observations

B. requires labelled outcomes for every observation

C. assumes compact clusters and is sensitive to scale and outliers

D. always produces hierarchical clusters

Correct Answer: C. assumes compact clusters and is sensitive to scale and outliers

Question 3

What does the first principal component represent?

A. The variable with the largest original measurement unit

B. The direction capturing the maximum variance subject to the PCA constraint

C. The smallest cluster identified in the data

D. The dimension containing all anomalous observations

Correct Answer: B. The direction capturing the maximum variance subject to the PCA constraint

Question 4

Why might DBSCAN be preferred over K-means for some datasets?

A. It requires the analyst to specify the exact number of clusters

B. It can identify non-spherical dense regions and label sparse points as noise

C. It is designed only for normally distributed variables

D. It always performs better in very high-dimensional spaces

Correct Answer: B. It can identify non-spherical dense regions and label sparse points as noise

Question 5

In a Gaussian mixture model, soft membership means that

A. each observation must belong equally to every cluster

B. cluster assignments are based only on medoids

C. an observation can receive probabilities of belonging to different components

D. the model does not estimate probability distributions

Correct Answer: C. an observation can receive probabilities of belonging to different components

Question 6

Which practice would create leakage when evaluating a predictive model with PCA features?

A. Fitting PCA only on each training fold

B. Applying a training-fitted PCA transformation to validation data

C. Fitting PCA on the entire dataset before cross-validation

D. Examining component loadings after fitting on training data

Correct Answer: C. Fitting PCA on the entire dataset before cross-validation

Question 7

An observation has a normal global value but is highly unusual for its account and time of day. This illustrates the importance of

A. contextual or local anomaly detection

B. removing all rare observations

C. using only Euclidean distance

D. increasing the number of PCA components

Correct Answer: A. contextual or local anomaly detection

Question 8

Which statement about t-SNE and UMAP is most accurate?

A. Their plotted axes generally have direct causal interpretations

B. A visible group in a plot automatically proves a natural cluster

C. They are primarily representation or visualization tools whose results require cautious interpretation

D. They eliminate the need for domain validation

Correct Answer: C. They are primarily representation or visualization tools whose results require cautious interpretation

Question 9

In the industrial sensor case, why were anomaly scores calculated within operating regimes?

A. To make all pumps have identical measurements

B. To avoid treating normal behaviour under different conditions as anomalous

C. To remove the need for maintenance staff

D. To guarantee that every alert represents a failure

Correct Answer: B. To avoid treating normal behaviour under different conditions as anomalous

Question 10

Which is the most responsible interpretation of an anomaly-detection alert in a high-impact application?

A. It is proof that the person or event is fraudulent

B. It is an investigative signal requiring appropriate review and validation

C. It should automatically trigger a severe penalty

D. It is always caused by a data-entry error

Correct Answer: B. It is an investigative signal requiring appropriate review and validation