8. Unsupervised Learning and Representation Discovery

Introduction and Context

Much of the data used in artificial intelligence is not accompanied by reliable labels. Customer records may not indicate which users are likely to leave, medical images may not be annotated with every abnormality, and sensor streams may contain previously unknown operating states. In such settings, learning cannot begin with a predefined target variable in the same way as supervised learning. Instead, the system must discover structure, regularities, similarities, deviations, or compact representations directly from the data.

Unsupervised learning addresses this problem. Its purpose is not simply to place observations into groups. It also includes discovering latent variables, reducing dimensionality, identifying unusual cases, finding relationships among events, and learning representations that can support later predictive tasks. The term representation discovery refers to the transformation of raw observations into forms that preserve useful information while making patterns easier to analyse or model.

This chapter examines the principal methods used for these purposes: clustering, dimensionality reduction, association analysis, anomaly detection, and neural representation learning. The methods are complementary. Clustering can reveal segments; principal component analysis can expose dominant directions of variation; association analysis can identify co-occurring events; anomaly detection can highlight rare behaviour; and autoencoders can learn nonlinear compressed representations. Their value depends not only on mathematical performance but also on interpretability, data quality, operational context, and the decisions that follow from the discovered patterns.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Explain the purpose and assumptions of unsupervised learning.

2. Distinguish clustering, dimensionality reduction, association analysis, and anomaly detection.

3. Apply and interpret k-means, hierarchical clustering, DBSCAN, principal component analysis, and autoencoders.

4. Select suitable methods according to data type, geometry, scale, noise, and business objective.

5. Evaluate unsupervised results using both quantitative measures and domain judgement.

6. Identify risks involving scaling, validation, interpretability, fairness, privacy, and misuse.

7. Design an end-to-end unsupervised learning workflow for segmentation, exploration, or risk detection.

The Unsupervised Learning Problem

Let a dataset be represented by a matrix X containing n observations and p features. In supervised learning, each observation is paired with a target y. In unsupervised learning, the target is absent, incomplete, or not trusted. The algorithm therefore seeks a structure that can be described by a model, such as a set of clusters, a lower-dimensional coordinate system, a collection of rules, or a probability distribution.

A central difficulty is that there may be many mathematically valid structures in the same data. A customer dataset might be partitioned by spending, frequency of purchase, geography, or product preference. None is automatically the correct segmentation. The appropriate interpretation depends on the intended decision. Consequently, unsupervised learning is exploratory rather than entirely automatic: algorithms generate hypotheses, while analysts determine whether those hypotheses are meaningful, stable, and actionable.

Several concepts recur across methods. Similarity measures how close two observations are; distance is one common formalisation. A feature representation determines what aspects of an observation are visible to the algorithm. Scaling is important because a high-variance feature can dominate a distance calculation. Latent structure refers to variables that are not directly observed but are inferred from relationships among observed variables. A hyperparameter is a user-selected setting, such as the number of clusters or the neighbourhood radius in DBSCAN.

Clustering as Structural Discovery

Clustering divides observations into groups such that members of the same group are more similar according to a chosen criterion than members of different groups. It is used in customer segmentation, document organisation, image analysis, biological discovery, network analysis, and exploratory quality control.

k-Means Clustering

The k-means algorithm seeks k cluster centroids and assigns each observation to the nearest centroid. Its usual objective is to minimise the within-cluster sum of squared Euclidean distances:

J = Σ(k=1 to K) Σ(xᵢ in Cₖ) ||xᵢ − μₖ||².

The standard procedure is iterative. Initial centroids are selected, observations are assigned to their nearest centroid, centroids are recomputed as the means of assigned observations, and the process repeats until assignments or the objective change very little. The method is computationally efficient and often effective when clusters are compact, approximately spherical, and of comparable scale.

The number of clusters is not supplied by nature. Analysts may examine the elbow plot, which shows how within-cluster error decreases as k increases, or use the silhouette coefficient, which compares within-cluster cohesion with separation from the nearest alternative cluster. These tools assist judgement but do not prove that a particular k is correct.

k-means has important weaknesses. It is sensitive to feature scaling and outliers, requires k in advance, and performs poorly for elongated, nested, or irregularly shaped groups. Different initialisations can produce different solutions. k-means++ initialisation and multiple random restarts reduce, but do not eliminate, this problem.

Hierarchical Clustering

Hierarchical clustering creates a nested structure of groups represented by a dendrogram. In agglomerative clustering, every observation begins as its own cluster. The two closest clusters are repeatedly merged until one cluster remains. The analyst can cut the dendrogram at a selected level to obtain a desired number of groups.

The result depends on the linkage rule. Single linkage uses the closest pair of observations and can produce chain-like clusters. Complete linkage uses the farthest pair and tends to form compact groups. Average linkage uses average pairwise distance, while Ward linkage merges clusters in a way that seeks a small increase in within-cluster variance. The distance metric and linkage rule must be considered together.

Hierarchical methods are useful when analysts need to inspect relationships at several levels of granularity. They can be computationally demanding for very large datasets, and early merging decisions cannot usually be undone. A dendrogram is also not a substitute for substantive interpretation; a visually appealing hierarchy may have little operational value.

Density-Based Clustering: DBSCAN

DBSCAN defines clusters as dense regions separated by areas of low density. It uses two main parameters: epsilon, the neighbourhood radius, and minPts, the minimum number of points required to define a dense neighbourhood. A point with sufficient neighbours is a core point. Points reachable from core points may form a cluster, while isolated observations can be labelled noise.

Unlike k-means, DBSCAN can discover non-spherical shapes and does not require the number of clusters beforehand. Its explicit noise category is useful in spatial analysis and sensor data. However, it is sensitive to the choice of epsilon and struggles when clusters have substantially different densities or when the feature space is very high-dimensional. Distances become less informative as dimensionality increases, making preprocessing or dimensionality reduction important.

Interpreting Clusters

A cluster label has no inherent meaning. After fitting a model, analysts should profile groups using feature summaries, distributions, representative examples, and domain measures. A cluster described as “high-value loyal customers” should be supported by observable evidence, not inferred from an arbitrary numerical label. Stability testing is also valuable: repeat the analysis under resampling, alternative initialisations, or modest parameter changes. If the groups disappear easily, decisions based on them should be cautious.

Dimensionality Reduction and Principal Component Analysis

High-dimensional data often contain redundant variables, correlated measurements, and noise. Dimensionality reduction seeks a smaller representation that preserves selected properties of the original data. It can support visualisation, reduce computational cost, mitigate multicollinearity, and provide inputs for later modelling.

Principal Component Analysis

Principal component analysis (PCA) constructs new orthogonal variables called principal components. The first component is the direction along which the centred data have maximum variance. The second component is orthogonal to the first and captures the greatest remaining variance, and so forth. If X is centred, PCA can be obtained through the eigendecomposition of the covariance matrix or through singular value decomposition.

The transformed coordinates are called scores, while the weights linking original features to components are called loadings. Large loading magnitudes indicate that a feature contributes strongly to a component, although interpretation requires attention to signs, scaling, and correlations. The proportion of variance explained by a component is its eigenvalue divided by the sum of all eigenvalues.

PCA is especially useful for exploratory plots and compression. For example, a two-dimensional score plot may reveal separation between operational states. Nevertheless, maximum variance is not equivalent to maximum predictive relevance. A low-variance direction may contain an important rare event. PCA is also linear and sensitive to scaling and outliers. Standardisation is commonly appropriate when variables use different units, but it should not be applied mechanically when absolute scale is meaningful.

The number of components may be selected using a cumulative explained-variance threshold, a scree plot, reconstruction error, or performance in a downstream task. These are decision criteria rather than universal rules. PCA fitted on a training set should be applied unchanged to later data to avoid information leakage.

Nonlinear Representation Methods

Some data lie near a curved or folded manifold that linear PCA cannot represent efficiently. Nonlinear techniques such as manifold learning can produce useful visualisations, although many are sensitive to neighbourhood settings and may distort global distances. Their outputs should therefore be treated as exploratory maps rather than definitive measurements.

Autoencoders provide a flexible neural approach. An encoder maps an input x to a latent code z, and a decoder reconstructs an approximation x-hat from z. Training minimises a reconstruction loss, such as mean squared error for continuous variables or cross-entropy for suitable categorical representations. The bottleneck forces the network to encode information compactly.

A basic autoencoder is not automatically a meaningful representation learner. With an excessively large latent layer, it may simply learn an identity function. Regularisation, denoising objectives, sparsity constraints, or variational formulations can encourage more useful representations. Autoencoders can capture nonlinear patterns and are applied to images, industrial signals, text embeddings, and fraud-related behaviour profiles. Their latent dimensions are often less interpretable than PCA components, and reconstruction quality does not guarantee that the representation preserves the features relevant to a particular decision.

Association Analysis

Association analysis discovers relationships among items or events, especially in transactional data. A rule such as {bread, butter} → {jam} does not mean that the antecedent causes the consequent. It indicates a pattern of co-occurrence.

Support is the proportion of transactions containing all items in a rule. Confidence is the proportion of transactions containing the antecedent that also contain the consequent. Lift compares confidence with the consequent's overall frequency. A lift greater than one suggests positive association, although it does not establish causality. Algorithms such as Apriori and FP-Growth search for frequent itemsets and then derive rules.

A major challenge is multiplicity: searching thousands of possible combinations can produce apparently strong rules by chance. Minimum support and confidence thresholds should be chosen with the decision context in mind, and discovered rules should be validated on later data. Rare but strategically important combinations may be missed by high support thresholds, while very low thresholds generate an unmanageable number of rules. Privacy is also important when transactional patterns could reveal sensitive behaviour.

Anomaly Detection and Risk Identification

Anomaly detection identifies observations that differ substantially from an assumed pattern. An anomaly may indicate fraud, equipment failure, cyber intrusion, data corruption, or simply a legitimate but unusual event. The distinction is contextual: rarity is not synonymous with risk.

Methods include distance-based scores, density-based methods, isolation forests, one-class support vector machines, and reconstruction-based models. Isolation Forest detects anomalies by observing that unusual points tend to be isolated quickly in randomly partitioned trees. An autoencoder may assign a high anomaly score to an observation with large reconstruction error. Local Outlier Factor compares the local density of a point with the density around its neighbours.

Threshold selection is a decision problem. A fixed percentile, an expected alert volume, a cost-based threshold, or expert review may be appropriate. If labels become available, precision, recall, false-alert burden, and detection delay should be assessed. In highly imbalanced settings, accuracy is usually misleading. Analysts should also distinguish point anomalies from contextual anomalies, in which a value is unusual only at a particular time, location, or operating condition.

An End-to-End Unsupervised Learning Workflow

A robust project begins with a decision question rather than an algorithm. “Can we find segments that support differentiated service?” is more useful than “Can we run k-means?” The analyst then defines the observation unit, selects relevant features, removes leakage, handles missing values, and examines distributions and measurement quality.

Next, features are transformed appropriately. Numeric variables may be standardised; highly skewed variables may be log-transformed; categorical variables may require one-hot or embedding representations. The distance function should reflect the data. Mixed-type data may require specialised distances rather than Euclidean distance.

The analyst fits one or more candidate methods, varies important hyperparameters, and evaluates internal structure using measures such as silhouette score, within-cluster dispersion, reconstruction error, explained variance, or rule quality. External validation is preferable when available: do clusters predict meaningful service needs, do anomalies correspond to investigated incidents, or do associations improve a controlled intervention?

Finally, results are communicated with uncertainty and limitations. A useful deliverable includes profiles, representative cases, stability evidence, recommended actions, monitoring procedures, and safeguards against misuse. Deployment requires tracking drift, changes in data collection, alert rates, and whether users act consistently with the intended interpretation.

Illustrative Case Study: Detecting Risky Machine Behaviour

Background

A manufacturing company operates several hundred industrial pumps. Each pump produces hourly measurements of vibration, temperature, pressure, flow rate, and energy consumption. Failure labels are scarce because maintenance teams usually replace components before catastrophic failure.

Situation and Problem

The company wants to identify operating regimes and prioritise inspections. A simple threshold for each sensor generates many alerts because normal behaviour differs across pump models and production conditions.

Analysis

The data science team first defines the observation as a pump-hour and removes records collected during known sensor outages. Features are standardised within relevant equipment families. PCA is used for exploration, revealing that vibration and energy consumption vary jointly while pressure and flow describe a second operational direction. The team retains several components for modelling but uses the score plots only as exploratory evidence.

A density-based clustering method is then applied to the component scores. It identifies common operating regimes and a small set of noise points. The clusters are profiled by pump type, production load, and maintenance history. An isolation forest is trained within each normal operating regime, rather than across all pumps, to reduce false alerts caused by legitimate differences between regimes.

Approach and Outcome

Maintenance engineers review a sample of high-scoring alerts. Some correspond to known sensor faults, some to unusual but safe production changes, and others to early mechanical degradation. The team adjusts the alert threshold and introduces a two-stage process: automated scoring followed by engineer review. The resulting system does not claim to diagnose failure; it prioritises inspections and records feedback for future supervised modelling.

Learning Points

The case illustrates that unsupervised learning is most effective when embedded in operational knowledge. Scaling, regime-aware modelling, human review, and feedback are as important as algorithm choice. PCA supported exploration, clustering represented operating context, and anomaly detection supported risk prioritisation. None of the methods alone established causation or guaranteed a failure prediction.

Challenges, Governance, and Professional Practice

Unsupervised models can reproduce measurement bias. If one customer group is more heavily monitored, it may appear more anomalous simply because more data are available. Segmentation can also become a proxy for protected characteristics, leading to unequal prices, service levels, or scrutiny. Analysts should examine group-level error and alert patterns and document features, transformations, and intended uses.

Privacy risks arise because latent representations may encode sensitive information even when the original variables are removed. Access controls, data minimisation, retention limits, and appropriate legal bases for processing are therefore relevant. In regulated settings, records should show how data were collected, how models were selected, how thresholds were set, and how human review operates.

Common mistakes include clustering unscaled variables, treating a high silhouette score as proof of business value, interpreting PCA signs as causal effects, assuming every anomaly is fraud, selecting thresholds after inspecting future outcomes, and presenting autoencoder embeddings as inherently understandable. Another mistake is operationalising unstable segments without monitoring drift. Best practice is to combine quantitative diagnostics, domain review, sensitivity analysis, out-of-sample checking, clear documentation, and periodic reassessment.

Current Developments

Recent practice increasingly combines self-supervised and unsupervised representation learning with foundation models, graph methods, contrastive learning, and multimodal data. Contrastive objectives learn representations by bringing related examples closer and separating unrelated examples. Graph-based methods represent relationships among users, devices, transactions, or documents, making them valuable for network anomaly detection and recommendation.

At the same time, interpretability and responsible deployment have become central concerns. Modern systems are evaluated not only by compactness or reconstruction error but also by stability, subgroup behaviour, human usefulness, computational cost, and resistance to manipulation. The direction of progress is therefore not simply toward larger models; it is toward representations that are more transferable, auditable, privacy-aware, and aligned with decisions.

Chapter Summary

Unsupervised learning discovers structure without a trusted target label. Clustering methods organise observations, with k-means suited to compact numerical groups, hierarchical clustering exposing nested relationships, and DBSCAN identifying density-based shapes and noise. PCA creates orthogonal linear components for compression and exploration, while autoencoders learn nonlinear latent representations through reconstruction. Association analysis identifies co-occurring items using support, confidence, and lift, but association does not imply causation. Anomaly detection assigns scores to unusual observations and requires context-sensitive thresholds.

Successful application depends on a complete workflow: define the decision, prepare and scale data, select an appropriate similarity or representation, compare methods, validate stability and usefulness, interpret results with domain experts, and monitor deployment. Ethical practice requires attention to privacy, bias, fairness, explainability, governance, and the consequences of turning exploratory patterns into automated decisions.

Key Terms

Unsupervised learning: Learning patterns from data without a predefined target label.

Clustering: Grouping observations according to similarity.

Centroid: The mean location representing a cluster in k-means.

k-means: An iterative algorithm minimising within-cluster squared distance.

Hierarchical clustering: Clustering that produces a nested tree of group relationships.

Dendrogram: A tree diagram showing hierarchical merges or splits.

DBSCAN: A density-based clustering algorithm that identifies clusters and noise.

Core point: In DBSCAN, a point with enough neighbours within the specified radius.

Dimensionality reduction: Representing data with fewer variables or coordinates.

PCA: A linear transformation into orthogonal directions of decreasing variance.

Loading: The weight of an original feature in a principal component.

Latent representation: An inferred, usually lower-dimensional representation of observed data.

Autoencoder: A neural network trained to reconstruct its input through a bottleneck.

Association rule: A statement describing the co-occurrence of itemsets or events.

Support: The proportion of records containing an itemset.

Confidence: The conditional frequency of a rule consequent given its antecedent.

Lift: A measure comparing rule confidence with the consequent's baseline frequency.

Anomaly: An observation that differs substantially from an expected pattern.

Data leakage: Use of information during training that would not be available at intended deployment.

Concept drift: Change over time in the data-generating process or its distribution.

Review Questions

Short Answer Questions

1. Why is unsupervised learning considered exploratory rather than fully automatic?

2. State two assumptions under which k-means is likely to perform well.

3. What information is conveyed by PCA loadings?

4. Distinguish support, confidence, and lift in association analysis.

5. Why is an unusual observation not necessarily a risky observation?

Long Answer Questions

1. Explain the procedure, assumptions, evaluation, and limitations of k-means clustering.

2. Compare hierarchical clustering and DBSCAN in terms of structure, parameters, cluster shape, and noise handling.

3. Explain PCA mathematically and discuss the interpretation of scores, loadings, and explained variance.

4. Describe how autoencoders learn latent representations and explain why reconstruction quality alone is insufficient.

5. Design an end-to-end unsupervised learning workflow for customer segmentation, including validation and governance.

Analytical / Application Questions

1. A dataset contains income in currency units, purchase frequency, and a binary membership variable. Which preprocessing and similarity decisions would you make before clustering, and why?

2. A fraud detection system produces many alerts for transactions from a newly launched region. Analyse possible causes and propose a method for validating whether the alerts represent genuine anomalies.

3. A PCA plot shows that the first two components explain only 45 percent of variance, but they separate two operational states clearly. How should this result be interpreted and communicated?

22. Multiple Choice Questions with Answers

Question 1

Which characteristic most directly makes DBSCAN different from k-means?

A. DBSCAN requires every observation to belong to a cluster

B. DBSCAN can identify irregularly shaped clusters and label noise

C. DBSCAN minimises squared distance to predefined centroids

D. DBSCAN always determines the same number of clusters as k-means

Correct Answer: B. DBSCAN can identify irregularly shaped clusters and label noise

Question 2

What does a large loading magnitude in PCA generally indicate?

A. The feature contributes strongly to the corresponding component

B. The feature is necessarily causally responsible for the component

C. The feature has the largest raw measurement scale

D. The feature is always an anomaly

Correct Answer: A. The feature contributes strongly to the corresponding component

Question 3

A k-means analysis gives different cluster assignments across runs. What is the most appropriate initial response?

A. Assume the run with the largest cluster is correct

B. Increase the number of labels in the dataset

C. Use multiple initialisations and assess solution stability

D. Replace all numeric variables with categorical variables

Correct Answer: C. Use multiple initialisations and assess solution stability

Question 4

In association analysis, a lift greater than one suggests that:

A. The antecedent causes the consequent

B. The consequent occurs more often with the antecedent than expected from its baseline frequency

C. The rule has perfect confidence

D. The itemset appears in every transaction

Correct Answer: B. The consequent occurs more often with the antecedent than expected from its baseline frequency

Question 5

Why might standardisation be important before distance-based clustering?

A. It guarantees that the chosen number of clusters is correct

B. It prevents high-scale features from dominating distance calculations

C. It removes the need to inspect outliers

D. It converts unsupervised learning into supervised learning

Correct Answer: B. It prevents high-scale features from dominating distance calculations

Question 6

An autoencoder assigns a high reconstruction error to a machine-hour record. The safest interpretation is that:

A. The machine has definitely failed

B. The record differs from patterns represented by the autoencoder

C. The record must have been incorrectly labelled

D. The latent code is necessarily interpretable

Correct Answer: B. The record differs from patterns represented by the autoencoder

Question 7

Which evaluation approach best tests whether discovered customer segments are useful in practice?

A. Selecting the segmentation with the most colourful visualisation

B. Using only the numerical cluster labels

C. Checking stability and whether segments support meaningful, validated decisions

D. Choosing the largest possible number of clusters

Correct Answer: C. Checking stability and whether segments support meaningful, validated decisions

Question 8

Why can a high explained-variance percentage in PCA still be insufficient for a business task?

A. Variance is never measurable

B. Low-variance directions may contain information important to the decision

C. PCA cannot transform numerical data

D. PCA always creates nonlinear components

Correct Answer: B. Low-variance directions may contain information important to the decision

Question 9

Which situation is a contextual anomaly rather than a simple point anomaly?

A. A defective sensor value that is impossible under any condition

B. A transaction amount that is unusual only for a particular time and location

C. A duplicated row in a database

D. A common observation near the centre of a cluster

Correct Answer: B. A transaction amount that is unusual only for a particular time and location

Question 10

What is a major governance risk when unsupervised segments are used to allocate services?

A. The algorithm may use too few mathematical operations

B. Segments may encode protected characteristics or measurement bias and produce unequal treatment

C. The model will always require labelled data

D. PCA components cannot be stored digitally

Correct Answer: B. Segments may encode protected characteristics or measurement bias and produce unequal treatment