Unsupervised Learning and Representation Discovery

Introduction

Supervised learning learns from labelled examples. Unsupervised learning addresses a different problem: discovering structure when the desired output is unknown or labels are unavailable. Given observations such as customer transactions, sensor readings, documents, images, or network events, an unsupervised method may identify groups, compress variables, discover co-occurring items, detect unusual cases, or construct a lower-dimensional representation.

The absence of labels does not mean the absence of assumptions. Every unsupervised method defines what counts as similarity, regularity, independence, density, or reconstruction quality. Consequently, the central professional question is not simply “Which algorithm should be used?” but “What structure is meaningful for this data and decision?”

This chapter develops five major families of methods: clustering, dimensionality reduction, association analysis, anomaly detection, and latent representation learning. It also discusses validation, interpretation, implementation, governance, and limitations.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the purpose and assumptions of unsupervised learning.
2. Select and apply appropriate clustering methods.
3. Use principal component analysis and related techniques for dimensionality reduction.
4. Interpret association rules using support, confidence, and lift.
5. Compare statistical, distance-based, and density-based anomaly detection.
6. Explain latent representations, autoencoders, and embedding spaces.
7. Evaluate unsupervised results without relying on labels alone.
8. Identify risks involving bias, privacy, leakage, instability, and misuse.

1. The Unsupervised Learning Problem

Let a dataset contain n observations and p measured features. It can be represented by a matrix X ∈ ℝⁿˣᵖ, where xᵢⱼ is the value of feature j for observation i. In unsupervised learning there is no target vector y supplied during training. The algorithm seeks a useful structure, such as:

• a partition of observations into groups;
• a projection from p variables into k dimensions, where k < p;
• frequent combinations of events or products;
• a score indicating unusualness;
• a latent vector that summarizes an observation.

A useful objective must be distinguished from a verified truth. A clustering algorithm can always produce clusters, even in random data. Therefore, interpretation requires domain knowledge, stability tests, sensitivity analysis, and—where possible—later business or scientific validation.

Similarity and distance

Many methods depend on a distance function. The Euclidean distance between two p-dimensional observations x and z is:

Here xⱼ and zⱼ are corresponding feature values. Euclidean distance is appropriate when numeric variables are comparably scaled and straight-line separation is meaningful. It is sensitive to scale and can be distorted by outliers.

The Manhattan distance is:

It may be useful when accumulated coordinate-wise differences are more meaningful. For mixed data, specialised measures such as Gower distance can combine numeric, categorical, and binary attributes. The choice of distance is a modelling decision, not a minor implementation detail.

Data preparation

Before fitting an unsupervised model, practitioners should inspect missingness, duplicate records, measurement units, skewness, categorical encoding, and extreme values. Standardisation is often used for numeric variables:

where μⱼ is the sample mean of feature j and σⱼ is its sample standard deviation. The resulting variable is dimensionless, has mean approximately zero, and has standard deviation approximately one when σⱼ > 0. Standardisation is appropriate when variables measured in different units should contribute comparably; it is not automatically appropriate when magnitude itself carries meaning.

2. Clustering: Discovering Groups

Clustering assigns observations to groups so that members of the same group are more similar under a selected criterion than members of different groups. The result is descriptive rather than inherently predictive.

2.1 K-means clustering

K-means seeks k centroids μ₁, …, μₖ and assignments cᵢ ∈ {1, …, k} that minimise within-cluster sum of squared Euclidean distances:

The algorithm alternates between two operations:

1. Assign each observation to its nearest centroid.
2. Replace each centroid with the arithmetic mean of its assigned observations.

It repeats these steps until assignments stabilise or improvement becomes negligible. Because the objective is non-convex with respect to assignments, different initial centroids can produce different solutions. K-means therefore commonly uses multiple initialisations and retains the solution with the lowest objective.

K-means works best for compact, approximately spherical clusters with reasonably similar scale and density. It requires k in advance, is sensitive to outliers, and is unsuitable without modification for purely categorical data.

#### Choosing k

The elbow method plots J for several values of k and looks for a point after which additional clusters produce diminishing improvement. The silhouette coefficient measures whether an observation is closer to its own cluster than to neighbouring clusters:

Here a(i) is the mean distance from observation i to points in its own cluster, and b(i) is the smallest mean distance from i to points in another cluster. Values range from −1 to 1. Higher values generally indicate better separation, but no threshold proves that the clustering is substantively meaningful.

2.2 Hierarchical clustering

Agglomerative hierarchical clustering begins with each observation as a separate cluster and repeatedly merges the closest pair. A dendrogram displays the sequence of merges. The number of clusters can be selected by cutting the dendrogram at a chosen height.

The linkage rule determines the distance between clusters. Single linkage uses the closest pair of observations and can create chaining. Complete linkage uses the farthest pair and tends to create compact groups. Average linkage uses the average pairwise distance. Ward’s method merges clusters to minimise the increase in within-cluster squared error and is closely related to variance minimisation.

Hierarchical clustering is useful when a nested structure is plausible and when an exploratory dendrogram is valuable. Its computational cost can become substantial for very large datasets.

2.3 Density-based clustering

DBSCAN identifies dense regions separated by lower-density regions. It uses a neighbourhood radius ε and a minimum number of points, MinPts. A point with sufficiently many neighbours within ε is a core point; points reachable from core points may form a cluster, while isolated points may be labelled noise.

DBSCAN can discover non-spherical shapes and explicitly identify noise. However, it struggles when clusters have very different densities or when the feature space is high-dimensional. HDBSCAN extends density-based reasoning by constructing a hierarchy and often handles variable density more effectively, although parameter and interpretation issues remain.

2.4 Mixture models

A Gaussian mixture model assumes that observations arise from a weighted mixture of Gaussian components:

The mixing weights satisfy πₖ ≥ 0 and ∑ₖ πₖ = 1. Each component has mean μₖ and covariance matrix Σₖ. Unlike hard clustering, the model provides a posterior probability that an observation belongs to each component. Parameters are commonly estimated using expectation–maximisation: the expectation step estimates memberships, and the maximisation step updates component parameters.

Mixture models are valuable when soft membership is meaningful, but their Gaussian assumptions may be inappropriate. Covariance matrices can also become unstable when the sample is small relative to the number of features.

3. Dimensionality Reduction

High-dimensional data may contain redundant variables, noise, and correlations. Dimensionality reduction seeks a lower-dimensional representation that preserves information relevant to analysis, visualisation, storage, or modelling.

3.1 Principal component analysis

Principal component analysis (PCA) constructs orthogonal directions of maximum variance. After centring the data, the sample covariance matrix is:

where X is the centred n × p data matrix. PCA finds eigenvectors vⱼ and eigenvalues λⱼ satisfying:

The eigenvectors define principal directions, and the eigenvalues indicate the variance captured in those directions. The first component has the greatest variance subject to unit length; each subsequent component is orthogonal to earlier components and captures the greatest remaining variance.

The proportion of variance explained by the first k components is:

This ratio is meaningful for centred numeric data and, when variables are standardised, describes variance in standardised feature space. PCA is a linear method. It may fail to preserve nonlinear manifolds and can be strongly affected by outliers.

Worked example: explained variance

Suppose the eigenvalues of a covariance matrix are 5, 3, 1, and 1. The total variance is 10. The first two components explain:

Thus, they explain 80% of the variance. This does not mean that 80% of every feature, or 80% of predictive information, has been retained; it means 80% of total sample variance under the PCA representation.

3.2 Nonlinear methods

t-distributed stochastic neighbour embedding (t-SNE) is designed primarily for visualising local neighbourhoods. It can reveal local groups but may distort global distances, cluster sizes, and inter-cluster relationships. UMAP also seeks a low-dimensional neighbourhood-preserving representation and is often faster, but its output depends on parameters and random initialisation.

These methods should not be treated as objective evidence that visible gaps represent real clusters. A two-dimensional plot is a hypothesis-generating tool, not a substitute for validation.

3.3 Feature selection versus feature extraction

Feature selection retains a subset of original variables, improving interpretability. Feature extraction creates new variables, such as principal components or embeddings. Selection is often preferable when regulatory explanation is essential; extraction may be more effective when variables are highly correlated or numerous.

4. Association Analysis

Association analysis discovers recurring combinations in transactional or event data. Let an itemset X be a set of items and T be the set of transactions.

Support measures how often an itemset occurs:

For an implication X → Y, confidence is:

Confidence estimates the proportion of transactions containing X that also contain Y. It is not the probability that X causes Y.

Lift compares confidence with the baseline prevalence of Y:

Lift greater than 1 indicates positive association relative to independence; lift equal to 1 is consistent with independence; lift less than 1 indicates negative association. Rules should also be assessed using occurrence counts, statistical significance, business relevance, and possible confounding.

The Apriori algorithm exploits the downward-closure property: if an itemset is infrequent, every superset must also be infrequent. This reduces the search space but can still be expensive when there are many items. FP-Growth compresses transactions into an FP-tree and avoids generating as many candidate itemsets.

5. Anomaly Detection

An anomaly is an observation that differs substantially from expected behaviour. Anomalies may be errors, fraud, equipment failures, attacks, rare but valid events, or previously unseen operating conditions.

Main approaches

• Statistical methods: model a distribution and flag low-probability observations. They work best when distributional assumptions are reasonable.
• Distance-based methods: identify points far from neighbours or a reference population.
• Local density methods: Local Outlier Factor compares the density around a point with densities around its neighbours.
• Isolation Forest: recursively partitions features; anomalies tend to be isolated in fewer random splits.
• One-class support vector machines: learn a boundary around normal data, subject to kernel and parameter choices.
• Autoencoder methods: flag observations with unusually high reconstruction error, assuming the model learns normal patterns.

An anomaly score does not establish that an observation is harmful. It ranks unusualness under a model. Operational systems require thresholds, investigation workflows, feedback, and monitoring of false positives and false negatives.

6. Latent Representation Learning

A latent representation is a compact internal description that is not directly observed. In machine learning, an encoder maps input x to latent vector z, and a decoder may reconstruct x from z.

An autoencoder typically minimises reconstruction loss:

where f is the encoder, g is the decoder, and ℓ is a loss function such as squared error for continuous data or cross-entropy for suitable binary or probabilistic outputs. A bottleneck, noise injection, sparsity constraint, or regularisation term prevents the network from simply copying the input.

A variational autoencoder (VAE) learns a probability distribution over latent variables. Its objective combines reconstruction quality with a regularisation term that encourages the approximate posterior to remain near a prior distribution. This can produce smooth latent spaces useful for generation, although generated data may be blurred or fail to represent rare modes.

Embeddings represent words, products, users, images, or other objects as vectors so that useful relationships correspond to geometric relationships. Similarity in an embedding space is meaningful only relative to the training data, objective, preprocessing, and metric. Embeddings can encode social stereotypes, historical inequities, and sensitive attributes.

7. Evaluating Unsupervised Results

Evaluation should combine quantitative, structural, and domain-based evidence.

A common error is to tune parameters until a desired visual or business conclusion appears. Holdout periods, sensitivity analysis, preregistered criteria, and independent review reduce this risk. Labels created after clustering should not be treated as independent validation if they were influenced by the clustering itself.

8. Detailed Applied Case: Industrial Sensor Monitoring

The following is an illustrative case.

Background

A manufacturing plant collects temperature, vibration, pressure, motor current, and cycle-time measurements from 1,200 machines. Most records describe normal operation, while documented failure labels are incomplete and delayed.

Problem

The engineering team wants to discover operating regimes, identify early warnings, and reduce unnecessary maintenance alerts. Raw sensor scales differ substantially, and some sensors are strongly correlated.

Analysis

The team first removes impossible readings, records missingness indicators, aggregates measurements over comparable time windows, and standardises continuous variables. It examines sensor drift and uses robust methods to reduce the effect of extreme measurement errors.

PCA is used to inspect correlation structure. The first components reveal a shared “load” pattern involving motor current and temperature, while another reflects vibration. These components support visualisation but are not automatically interpreted as physical causes.

K-means and a Gaussian mixture model are compared. K-means produces three compact operating groups, whereas the mixture model supplies soft membership and identifies transitional observations. DBSCAN flags isolated points, but its results change considerably with the neighbourhood radius and are therefore used only as a supplementary screen.

An autoencoder is trained primarily on periods judged operationally normal. Its reconstruction error is monitored by machine and by time. Alerts are not issued solely when an error exceeds a global threshold; thresholds are calibrated against maintenance capacity and reviewed by engineers.

Approach and solution

The final monitoring system uses:

1. PCA for exploratory visualisation and sensor redundancy analysis.
2. Soft clustering to describe operating regimes.
3. A reconstruction-based anomaly score for candidate events.
4. A human review process linking alerts to maintenance logs.
5. Monthly drift checks for sensor distributions and alert rates.

Outcome

In the illustrative deployment, engineers report that regime-specific thresholds reduce nuisance alerts compared with a single global threshold. Several unusual patterns are found to be sensor calibration problems rather than mechanical failures. The system therefore labels anomalies as “investigate” rather than “failure,” preserving human responsibility for diagnosis.

Learning points

Unsupervised methods are most effective when combined rather than treated as autonomous truth. Data quality, temporal leakage, maintenance feedback, concept drift, and operational costs are as important as algorithmic accuracy. A rare event that is technically unusual may be valuable evidence rather than an error.

9. Implementation and Professional Practice

A practical workflow is:

1. Define the decision or discovery purpose.
2. Audit data provenance, units, missingness, and sampling process.
3. Select features and a similarity measure consistent with the domain.
4. Establish preprocessing and avoid leakage from future information.
5. Fit multiple plausible methods or parameter settings.
6. Evaluate stability, interpretability, and operational usefulness.
7. Profile groups or investigate high-scoring anomalies with domain experts.
8. Document assumptions, random seeds, versions, thresholds, and limitations.
9. Monitor drift and revalidate after deployment.

Ethical and governance issues

Clustering can create categories that influence prices, services, employment, or access to credit. Such categories may reproduce protected-group disparities even when sensitive attributes are excluded, because proxy variables remain. Dimensionality reduction and embeddings can conceal discriminatory structure rather than remove it.

Privacy is also important. Transactional data, location traces, and behavioural embeddings may enable re-identification. Access controls, minimisation, retention limits, secure computation, and impact assessments should be considered. Organisations should document intended use, prohibited use, human review requirements, and appeal mechanisms.

Contemporary developments

Current practice increasingly combines self-supervised learning, foundation-model embeddings, contrastive objectives, graph representation learning, and multimodal data. These methods can provide powerful representations but introduce substantial computational, environmental, and governance costs. Large pretrained representations should be evaluated for domain shift, memorisation, security vulnerabilities, and hidden bias before use in high-impact settings.

Challenges and Limitations

1. No universally correct structure: different distances or algorithms may produce different but internally coherent results.
2. Curse of dimensionality: distances become less discriminating as dimensions increase, especially with irrelevant variables.
3. Parameter sensitivity: k, ε, linkage, latent dimension, and anomaly thresholds influence outcomes.
4. Outliers and missingness: both can distort centroids, covariance estimates, and embeddings.
5. Interpretability: latent dimensions may not correspond to human concepts.
6. Temporal drift: patterns learned from historical data may cease to apply.
7. Multiple testing: searching many rules increases the chance of spurious associations.
8. Feedback loops: acting on discovered groups can change the future data distribution.
9. Computational cost: pairwise distances, repeated clustering, and deep representation learning may be expensive.

Chapter Summary

Unsupervised learning discovers structure without a predefined target. Clustering identifies groups using assumptions about distance, density, or probability distributions. K-means is efficient for compact numeric clusters; hierarchical methods reveal nested structure; density-based methods identify irregular groups and noise; mixture models provide probabilistic membership.

Dimensionality reduction compresses data for visualisation, noise reduction, and modelling. PCA uses eigenvectors of a covariance matrix and reports explained variance, while nonlinear methods are mainly exploratory. Association analysis identifies recurring item combinations using support, confidence, and lift, but associations do not prove causation. Anomaly detection ranks unusual observations through statistical, distance, density, isolation, or reconstruction-based methods. Latent representation learning creates compact vectors through encoders, autoencoders, and related models.

Reliable practice requires careful preprocessing, an appropriate similarity measure, stability analysis, domain interpretation, governance, and monitoring. Unsupervised outputs are hypotheses and decision aids—not automatically objective facts.

Key Terms

Unsupervised learning: Learning patterns from data without target labels.

Clustering: Grouping observations according to a similarity criterion.

Centroid: The mean vector representing a cluster in methods such as K-means.

K-means: A clustering algorithm that minimises within-cluster squared distance.

Hierarchical clustering: Clustering that builds a nested sequence of merges or splits.

DBSCAN: A density-based method that identifies dense regions and noise.

Gaussian mixture model: A probabilistic model representing data as a weighted mixture of Gaussian components.

Distance metric: A rule for quantifying dissimilarity between observations.

Principal component analysis: A linear transformation into orthogonal directions of decreasing variance.

Eigenvalue: In PCA, the variance associated with a principal direction.

Explained variance ratio: The proportion of total variance captured by selected components.

Association rule: An implication describing co-occurrence between itemsets.

Support: The proportion of transactions containing an itemset.

Confidence: The conditional frequency of the rule consequent given its antecedent.

Lift: Confidence divided by the baseline support of the consequent.

Anomaly: An observation judged unusual under a specified model.

Latent representation: A learned, unobserved vector summarising data.

Autoencoder: A neural network trained to encode and reconstruct inputs.

Embedding: A vector representation designed to preserve useful relationships.

Concept drift: Change over time in the data-generating process or its distribution.

Review Questions

Short Answer Questions

1. Why is feature scaling important for Euclidean-distance-based clustering?
2. State the meanings of support, confidence, and lift in association analysis.
3. What does the first principal component represent?
4. Why can an anomaly score not be interpreted automatically as evidence of fraud or failure?
5. Distinguish feature selection from feature extraction.

Long Answer Questions

1. Compare K-means, hierarchical clustering, DBSCAN, and Gaussian mixture models in terms of assumptions, outputs, strengths, and limitations.
2. Explain PCA using the covariance matrix, eigenvectors, eigenvalues, and explained variance ratio.
3. Describe a complete workflow for evaluating an unsupervised learning system in a business or industrial setting.
4. Explain how autoencoders learn representations and discuss their use in anomaly detection.
5. Discuss ethical, privacy, and governance risks associated with clustering and embedding systems.

Analytical / Application Questions

1. A retailer has transaction records containing thousands of products. Design an association-analysis workflow and explain how you would avoid promoting rules that are statistically strong but commercially meaningless.
2. A dataset contains income, age, distance, and transaction count. Explain how you would preprocess it before K-means, and discuss how your choices might change the resulting clusters.
3. An anomaly detector produces many alerts after a sensor replacement. Analyse possible causes and propose a validation and monitoring plan before changing the alert threshold.

Multiple Choice Questions with Answers

Question 1

Which objective does standard K-means minimise?

A. Within-cluster sum of squared Euclidean distances

B. Between-cluster average categorical entropy only

C. Number of principal components

D. Association-rule confidence

Correct Answer: A. Within-cluster sum of squared Euclidean distances

Question 2

What does a silhouette value near 1 generally suggest?

A. The observation is equally close to every cluster

B. The observation is well matched to its own cluster and separated from others

C. The dataset contains no outliers

D. The clustering must be causal

Correct Answer: B. The observation is well matched to its own cluster and separated from others

Question 3

If PCA eigenvalues are 5, 3, 1, and 1, what proportion of variance is explained by the first two components?

A. 20%

B. 50%

C. 80%

D. 90%

Correct Answer: C. 80%

Question 4

A lift of 1 for X → Y indicates that the rule is, under the measure,

A. consistent with independence between X and Y

B. proof that X causes Y

C. evidence that Y never occurs

D. always more useful than a rule with lift 2

Correct Answer: A. consistent with independence between X and Y

Question 5

Which method is specifically designed to identify dense regions and label isolated points as noise?

A. PCA

B. DBSCAN

C. Apriori

D. Ordinary least squares

Correct Answer: B. DBSCAN

Question 6

In a Gaussian mixture model, what do posterior membership probabilities provide?

A. The probability that an observation belongs to each component

B. The exact causal effect of each feature

C. The number of missing values in each feature

D. The percentage of variance explained by PCA

Correct Answer: A. The probability that an observation belongs to each component

Question 7

Why can t-SNE plots be misleading when interpreting global cluster relationships?

A. They cannot display any local neighbourhoods

B. They may distort global distances and cluster sizes

C. They require labelled target values

D. They always produce linear projections

Correct Answer: B. They may distort global distances and cluster sizes

Question 8

For the rule X → Y, confidence is calculated as

A. support(X) divided by support(Y)

B. support(X ∪ Y) divided by support(X)

C. support(Y) divided by support(X ∪ Y)

D. support(X ∪ Y) multiplied by the number of transactions

Correct Answer: B. support(X ∪ Y) divided by support(X)

Question 9

Why might an autoencoder be used for anomaly detection?

A. Anomalies always have the smallest reconstruction error

B. A model trained on normal patterns may reconstruct unusual observations poorly

C. It guarantees that all rare observations are fraudulent

D. It eliminates the need for threshold selection

Correct Answer: B. A model trained on normal patterns may reconstruct unusual observations poorly

Question 10

Which practice best addresses instability in an unsupervised model?

A. Report only one random initialisation

B. Change parameters until the preferred conclusion appears

C. Test results across seeds, samples, time periods, and reasonable parameter settings

D. Assume visual separation proves validity

Correct Answer: C. Test results across seeds, samples, time periods, and reasonable parameter settings