5. Unsupervised, Semi-Supervised, and Representation Learning

Introduction and Context

Machine learning systems are often presented as if abundant, accurately labelled data were readily available. In many research and industrial settings, this assumption is false. Organisations may possess millions of observations but only a small fraction have been annotated by experts. Images, text, sensor streams, user interactions, and scientific measurements frequently contain structure that is not captured by predefined labels. Learning from such data requires methods that discover regularities, infer hidden factors, exploit incomplete supervision, or construct useful representations before a task is attempted.

Unsupervised learning addresses the discovery of structure without target labels. Semi-supervised learning combines a limited labelled sample with a much larger unlabelled sample. Representation learning seeks transformations of raw data that preserve task-relevant information while making prediction, retrieval, generation, or scientific interpretation easier. Self-supervised learning creates supervisory signals from the data itself, whereas weak supervision uses noisy, indirect, heuristic, or incomplete sources of labelling.

These areas are closely related but not interchangeable. A clustering algorithm may discover useful groups without producing categories that correspond to human concepts. A low-dimensional embedding may improve visualisation while distorting distances important for classification. A powerful pretrained representation may transfer successfully between tasks, yet encode demographic bias or spurious correlations. Doctoral-level work therefore requires more than selecting algorithms: it requires reasoning about assumptions, identifiability, evaluation, uncertainty, and the relationship between learned structure and the intended scientific or organisational purpose.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Distinguish unsupervised, semi-supervised, weakly supervised, and self-supervised learning.

2. Select and critically evaluate clustering, dimensionality-reduction, manifold-learning, and density-estimation methods.

3. Explain latent-variable models, autoencoders, contrastive learning, and generative representation learning.

4. Analyse assumptions, identifiability, uncertainty, and failure modes in representation learning.

5. Design evaluation procedures that assess both representation quality and downstream utility.

6. Apply transfer learning and semi-supervised methods responsibly in professional and research settings.

7. Identify ethical, governance, reproducibility, and deployment concerns associated with learned representations.

From Unlabelled Observations to Useful Structure

Let a dataset be represented by observations X = {x_i} without observed targets y_i. Unsupervised learning attempts to estimate properties of the data distribution p(x), its geometry, or a latent structure that may have generated the observations. The absence of labels does not mean the absence of assumptions. Every method imposes preferences: k-means prefers compact, approximately spherical clusters; principal component analysis prefers directions of high variance; a variational autoencoder imposes a probabilistic latent prior and a reconstruction-generating mechanism.

A useful distinction is between descriptive and task-oriented structure. Descriptive structure summarises regularities in the observed data. Task-oriented structure is judged by whether it supports a later objective, such as anomaly detection, diagnosis, ranking, or forecasting. These objectives can conflict. A representation that preserves lighting conditions in photographs may be excellent for reconstruction but unhelpful for recognising object identity.

Important terminology

A feature is a measured variable or input attribute. A representation is a transformed encoding of an observation. A latent variable is an unobserved quantity introduced to explain observed data. Embedding usually refers to a vector representation in a space designed to preserve selected relationships. Inductive bias is the set of assumptions that makes generalisation possible. Identifiability concerns whether the underlying parameters or latent factors can be uniquely recovered from the observed distribution. Collapse occurs when a learned representation carries little useful variation, as in a neural encoder mapping many inputs to nearly the same vector. Distribution shift occurs when training and deployment distributions differ.

Clustering: Discovering Groups Without Pretending They Are Truth

Clustering partitions or organises observations according to a similarity or probability model. The choice of similarity is substantive: Euclidean distance, cosine similarity, correlation, or a domain-specific metric can produce radically different results.

K-means minimises within-cluster squared Euclidean distance:

J = Σ_i ||x_i − μ_{z_i}||²,

where z_i is the assigned cluster and μ is its centroid. It is computationally attractive, but requires a chosen number of clusters, is sensitive to scaling and outliers, and favours convex, similarly sized groups. K-medoids is more robust because cluster representatives are actual observations. Hierarchical clustering constructs a nested tree using agglomerative or divisive procedures; its result depends on linkage choices such as single, complete, average, or Ward linkage.

Density-based methods, including DBSCAN and related approaches, identify regions of high density and can label isolated points as noise. They are valuable when clusters have irregular shapes, although density thresholds become difficult to select in high-dimensional or uneven-density data. Gaussian mixture models represent the distribution as a weighted sum of Gaussian components and provide soft membership probabilities. The expectation-maximisation algorithm alternates between estimating component responsibilities and updating parameters. Model selection may use likelihood-based criteria such as AIC or BIC, but these criteria do not guarantee meaningful domain categories.

Cluster validation should combine internal, external, and stability evidence. Silhouette width compares within-cluster cohesion with separation from neighbouring clusters. Adjusted Rand index or normalized mutual information can be used when reference labels exist, but agreement with labels is not automatically evidence that the clustering captures a scientifically meaningful mechanism. Stability analysis—repeating clustering under resampling, perturbation, or alternative initialisation—is especially important. A cluster that disappears after minor perturbations should not be presented as a robust discovery.

Dimensionality Reduction and Manifold Learning

High-dimensional data often contain redundant or noisy variables. Dimensionality reduction seeks a lower-dimensional representation that preserves selected properties. Principal component analysis (PCA) projects centred data onto orthogonal directions of maximum variance. If X has covariance matrix Σ, the first principal component is the eigenvector associated with the largest eigenvalue. PCA is linear, interpretable through loadings, and useful for compression, denoising, and preprocessing. However, variance is not equivalent to predictive relevance, and PCA can be dominated by measurement scale.

Independent component analysis seeks statistically independent rather than merely uncorrelated components, commonly under non-Gaussianity assumptions. Non-negative matrix factorisation constrains factors and coefficients to be non-negative, often producing additive, parts-based representations. These methods can support interpretability but may be non-unique and sensitive to preprocessing.

Manifold learning assumes observations lie near a lower-dimensional manifold embedded in a higher-dimensional space. Isomap preserves approximate geodesic distances; locally linear embedding preserves local reconstruction relationships; Laplacian eigenmaps use neighbourhood graphs. t-SNE and UMAP are widely used for visual exploration, but their two-dimensional maps should not be interpreted as faithful global representations. Apparent clusters may result from perplexity, neighbourhood parameters, random seeds, density distortions, or preprocessing. A visual separation is a hypothesis for investigation, not proof of natural categories.

A practical reduction workflow begins with domain-informed scaling and missing-data treatment, compares linear and nonlinear methods, tests sensitivity to hyperparameters, and evaluates whether the reduced representation preserves the relationships required by the application. Reconstruction error, trustworthiness, continuity, neighbourhood preservation, and downstream performance answer different questions and should not be conflated.

Density Estimation and Anomaly Detection

Density estimation models p(x), either parametrically or non-parametrically. Parametric models such as Gaussian mixtures are interpretable but can be misspecified. Kernel density estimation places a kernel around each observation; its bandwidth controls the bias–variance trade-off. Normalising flows learn invertible transformations from a simple base distribution to a complex density and permit exact likelihood calculation under architectural constraints.

Density estimates are useful for simulation, novelty detection, reliability analysis, and probabilistic decision-making. Yet low likelihood does not always mean semantic abnormality. In high-dimensional spaces, typicality and likelihood can diverge: a model may assign greater likelihood to an observation that is unlike the data in a task-relevant sense. Anomaly detection should therefore combine density information with reconstruction error, predictive uncertainty, domain rules, and human review where consequences are serious.

Latent-Variable Models and Representation Learning

Latent-variable models explain observations through hidden causes. In a mixture model, a discrete latent variable selects a component. In factor analysis, continuous factors generate correlated measurements through a linear model with noise. Probabilistic principal component analysis places PCA in a likelihood framework. These models support uncertainty estimation and missing-data inference, but latent dimensions are not automatically interpretable causal variables.

An autoencoder consists of an encoder f_θ(x) and decoder g_φ(z), trained to minimise reconstruction loss ||x − g_φ(f_θ(x))||² or an appropriate task-specific loss. Bottlenecks encourage compression, but unrestricted autoencoders may learn identity-like codes or preserve nuisance variation. Denoising and contractive objectives can encourage robustness.

A variational autoencoder (VAE) introduces a distribution q_θ(z|x) and a prior p(z), optimising an evidence lower bound:

E_q[log p_φ(x|z)] − KL(q_θ(z|x) || p(z)).

The reconstruction term preserves information, while the Kullback–Leibler term regularises the latent space. VAEs enable sampling and interpolation, but may generate overly smooth outputs and suffer from posterior collapse. β-VAE and related methods alter the balance between reconstruction and regularisation to encourage factorisation, although disentanglement cannot generally be guaranteed without suitable inductive biases or additional information.

Representation quality is multidimensional. A representation may be evaluated for predictive usefulness, linear separability, sample efficiency, robustness to perturbations, transferability, calibration, interpretability, fairness, and information retained about nuisance variables. No single metric establishes quality. A representation that performs well on a benchmark may exploit shortcuts and fail under distribution shift.

Self-Supervised and Semi-Supervised Learning

Self-supervised learning creates a pretext or proxy task from unlabelled data. In language modelling, tokens may be predicted from context. In images, an encoder may learn by contrasting augmented views of the same image against views of other images. In masked autoencoding, portions of an input are hidden and reconstructed. The proxy task should force the model to learn information useful beyond the pretext task; otherwise, excellent pretext performance may have little transfer value.

Contrastive objectives encourage positive pairs to have similar representations and negative pairs to be separated. The design of augmentations is critical. An augmentation assumed to preserve identity may in fact remove clinically, scientifically, or legally important information. Non-contrastive methods avoid explicit negatives but still require mechanisms preventing collapse, such as predictor asymmetry, stop-gradient operations, or architectural constraints.

Semi-supervised learning uses labelled data L and unlabelled data U. Common strategies include pseudo-labelling, consistency regularisation, entropy minimisation, graph-based propagation, and teacher–student methods. A model may assign a pseudo-label only when confidence exceeds a threshold, while consistency training requires predictions to remain stable under perturbations. These methods rely on assumptions such as cluster structure, smoothness, or a low-density decision boundary. If the unlabelled population contains new classes or differs systematically from the labelled data, pseudo-labelling can amplify errors.

Weak supervision differs from semi-supervision because labels may exist but be noisy, indirect, coarse, or generated by heuristics. Labeling functions can encode expert rules, knowledge-base matches, or pattern detectors. Their conflicts and accuracies must be estimated rather than treated as ground truth. A small, carefully sampled validation set is essential for measuring performance after label aggregation.

Transfer Learning and Adaptation

Transfer learning reuses knowledge learned from one task or domain for another. A common pipeline pretrains an encoder on a large corpus, freezes or partially freezes it, and fine-tunes task-specific layers. Transfer can reduce data requirements and training cost, but negative transfer occurs when source and target domains differ in relevant ways. Domain adaptation addresses distribution differences, while domain generalisation seeks representations robust to unseen domains.

Researchers should distinguish feature transfer from parameter reuse. A transferred model may contain useful visual or linguistic abstractions, but its calibration, fairness, and failure patterns do not automatically transfer. Fine-tuning can erase general capabilities, whereas parameter-efficient methods such as adapters or low-rank updates can reduce computational and operational costs. Evaluation should include a target-domain test set, subgroup analysis, robustness tests, and comparison with a target-trained baseline.

An Illustrative Case: Semi-Supervised Fault Diagnosis in Industrial Sensors

Background. A manufacturing organisation collects vibration, temperature, and acoustic signals from rotating equipment. Engineers have labelled a small number of failures, while most records are unlabelled and represent normal operation under varying loads.

Situation and problem. A supervised classifier trained only on confirmed failures detects severe faults but misses early degradation. Simple clustering produces groups corresponding partly to operating conditions rather than health states. The organisation needs an earlier warning system without treating every unusual operating condition as a fault.

Analysis. Engineers first standardise signals by machine and operating regime, then construct time-window representations using spectral features and a self-supervised temporal encoder. Augmentations are restricted to transformations known not to destroy fault signatures. A density model characterises normal regimes, while a semi-supervised classifier uses the labelled failures and high-confidence pseudo-labels. Cluster stability is assessed across machines and time periods. Domain experts inspect representative windows from each cluster.

Approach. The final system combines three signals: classification probability for known fault types, distance from the learned normal representation, and persistence of the signal over successive windows. A human review queue is triggered when the signals disagree or when the equipment operates outside the training distribution. Thresholds are selected according to the asymmetric costs of missed failures and unnecessary inspections.

Outcome. In this illustrative case, success is defined not by a visually attractive embedding but by earlier detection at an acceptable false-alarm rate, stable behaviour across machines, and traceable evidence for maintenance personnel. Monitoring includes drift detection, periodic relabelling, and recalibration.

Learning points. Unlabelled data can improve a system without being treated as unquestioned truth. Representation learning is most valuable when connected to the operational decision. Human expertise remains necessary for defining valid augmentations, interpreting clusters, and governing high-cost errors.

Challenges, Ethics, and Governance

Unsupervised discovery can reify statistical groupings into social categories. Clustering customer, employee, or patient data may reproduce historical discrimination or expose sensitive attributes. Learned embeddings can enable inference of information that individuals did not disclose. Data minimisation, purpose limitation, access controls, retention policies, and privacy-preserving methods are therefore relevant even when labels are absent.

Common technical mistakes include applying Euclidean distance to unscaled variables, selecting cluster count solely from an elbow plot, interpreting t-SNE maps as ground truth, leaking test data into representation pretraining, accepting pseudo-labels without calibration, and reporting only average downstream accuracy. Research reports should document preprocessing, random seeds, hyperparameters, data exclusions, augmentations, pretraining corpora, evaluation splits, and compute conditions. If a representation is used in a consequential decision, provenance, auditability, contestability, and human oversight should be designed from the beginning.

Privacy risks may arise through memorisation, membership inference, and reconstruction. Differential privacy can reduce some risks but may affect utility. Federated learning can limit centralised data collection, although gradients and model updates may still leak information. Governance must also address licensing and provenance of pretrained data, especially when models are transferred across organisations or jurisdictions.

Current Directions and Best Practice

Contemporary research increasingly combines generative modelling, multimodal learning, foundation models, retrieval, and structured representations. Representation systems are being evaluated not only by benchmark accuracy but also by robustness, compositionality, causal usefulness, calibration, energy use, and social impact. Mechanistic interpretability and causal representation learning investigate whether internal features correspond to stable mechanisms rather than convenient correlations, although strong claims remain difficult.

A defensible project should begin with the decision or scientific question, not with a fashionable algorithm. It should define what similarity means, identify plausible nuisance variables, establish leakage-resistant evaluation, compare simple baselines, and test stability. Unlabelled data should be stratified and audited. Human validation should examine both representative and difficult cases. Finally, deployment should include drift monitoring, uncertainty reporting, recalibration, incident procedures, and a mechanism for updating or withdrawing the system.

Chapter Summary

Unsupervised learning discovers structure without target labels through clustering, dimensionality reduction, manifold learning, density estimation, and latent-variable modelling. Each method embeds assumptions about geometry, probability, similarity, or generative structure. Semi-supervised learning exploits limited labels alongside unlabelled data, while weak supervision uses noisy or indirect labelling sources. Self-supervised learning constructs proxy tasks such as masked prediction, reconstruction, and contrastive comparison. Transfer learning reuses representations across tasks but can produce negative transfer and inherited bias.

Representation quality cannot be reduced to one score. It should be assessed through downstream utility, robustness, sample efficiency, stability, uncertainty, interpretability, fairness, and resistance to distribution shift. Effective practice combines algorithmic analysis with domain expertise, leakage-resistant evaluation, governance, privacy protection, and continuous monitoring.

Key Terms

Unsupervised learning: Learning patterns or structure without observed target labels.

Semi-supervised learning: Learning from a small labelled dataset and a larger unlabelled dataset.

Representation learning: Learning a transformation that encodes useful properties of raw observations.

Embedding: A vector representation designed to preserve selected relationships.

Latent variable: An unobserved variable used to explain observed data.

Inductive bias: Assumptions that guide a learner toward particular generalisations.

K-means: A clustering method minimising squared distance to cluster centroids.

Gaussian mixture model: A probabilistic model representing data as a weighted mixture of Gaussian components.

Density estimation: Estimation of the probability distribution underlying observations.

Principal component analysis: Linear projection onto directions of maximum variance.

Manifold learning: Learning a lower-dimensional geometric structure embedded in a higher-dimensional space.

Autoencoder: An encoder–decoder model trained to reconstruct its input or a related target.

Variational autoencoder: A probabilistic autoencoder that regularises a latent distribution through variational inference.

Self-supervised learning: Learning from supervisory signals generated from the input data itself.

Contrastive learning: Learning representations by comparing related positive examples with distinct negative or implicit alternatives.

Weak supervision: Supervision derived from noisy, indirect, incomplete, or heuristic sources.

Pseudo-labelling: Assigning model-generated labels to unlabelled examples for further training.

Transfer learning: Reusing knowledge or representations learned in one task or domain for another.

Identifiability: The extent to which model parameters or latent factors can be uniquely recovered from data.

Posterior collapse: A variational-model failure in which latent variables carry little information about inputs.

Review Questions

Short Answer Questions

1. Why does the choice of similarity measure influence clustering results?

2. State one important difference between PCA and manifold-learning methods.

3. What is the purpose of a bottleneck in an autoencoder?

4. Define pseudo-labelling and identify one risk associated with it.

5. Why is a two-dimensional t-SNE or UMAP plot not proof of natural clusters?

Long Answer Questions

1. Compare k-means, hierarchical clustering, density-based clustering, and Gaussian mixture models in terms of assumptions, outputs, and appropriate applications.

2. Explain the objective of a variational autoencoder and discuss reconstruction, regularisation, and posterior collapse.

3. Examine how self-supervised learning can produce useful representations and why proxy-task design matters.

4. Discuss how representation quality should be evaluated in a research or industrial system.

5. Analyse the opportunities and risks of transfer learning under domain shift.

Analytical / Application Questions

1. You receive a high-dimensional customer dataset with mixed scales, outliers, and no labels. Design an exploratory workflow and justify each modelling and validation decision.

2. A semi-supervised classifier shows rapidly increasing accuracy after pseudo-labelling, but performance falls on a new demographic subgroup. Diagnose likely causes and propose an evaluation and mitigation plan.

3. Design a representation-learning system for early anomaly detection in medical or industrial time series. Specify the training signal, validation strategy, uncertainty handling, and governance controls.

22. Multiple Choice Questions with Answers

Question 1

Which statement best describes a central limitation of k-means clustering?

A. It cannot operate on numerical data.

B. It assumes clusters are broadly compact and represented by centroids.

C. It always produces probabilistic membership estimates.

D. It identifies arbitrary-density regions without parameter selection.

Correct Answer: B. It assumes clusters are broadly compact and represented by centroids.

Question 2

Why should a t-SNE or UMAP visualisation not be treated as proof that natural clusters exist?

A. These methods cannot process high-dimensional data.

B. They always preserve global distances exactly.

C. Their apparent structure can depend strongly on parameters, preprocessing, and randomisation.

D. They require labelled data for training.

Correct Answer: C. Their apparent structure can depend strongly on parameters, preprocessing, and randomisation.

Question 3

What does the KL-divergence term in a standard variational autoencoder primarily do?

A. It forces every input to be reconstructed perfectly.

B. It regularises the learned latent distribution toward a prior.

C. It selects the number of observed features.

D. It converts a continuous latent space into class labels.

Correct Answer: B. It regularises the learned latent distribution toward a prior.

Question 4

Which situation most directly illustrates negative transfer?

A. A pretrained encoder reduces the amount of target-domain labelled data required.

B. A source representation improves performance on several target tasks.

C. A source-trained representation encodes patterns that conflict with the target domain and reduces target performance.

D. A model is evaluated on a held-out target test set.

Correct Answer: C. A source-trained representation encodes patterns that conflict with the target domain and reduces target performance.

Question 5

A pseudo-labelling system performs poorly because the unlabelled data contain classes absent from the labelled sample. Which assumption has most clearly failed?

A. The labelled and unlabelled populations are sufficiently compatible for the inferred labels.

B. The input variables are measured in identical units.

C. The encoder has no trainable parameters.

D. PCA preserves all nonlinear relationships.

Correct Answer: A. The labelled and unlabelled populations are sufficiently compatible for the inferred labels.

Question 6

Which evaluation approach gives the strongest evidence that an unsupervised cluster is robust?

A. Selecting the clustering with the most visually separated plot.

B. Choosing the result from one arbitrary random initialisation.

C. Testing cluster stability under resampling and reasonable perturbations.

D. Increasing the number of clusters until every observation is isolated.

Correct Answer: C. Testing cluster stability under resampling and reasonable perturbations.

Question 7

What distinguishes weak supervision from ordinary semi-supervised learning?

A. Weak supervision uses only completely unlabelled data.

B. Weak supervision may use noisy, indirect, heuristic, or incomplete labelling sources.

C. Semi-supervised learning never uses model predictions.

D. Weak supervision requires all labels to be manually verified.

Correct Answer: B. Weak supervision may use noisy, indirect, heuristic, or incomplete labelling sources.

Question 8

In the industrial sensor case, why were classification probability, representation distance, and persistence combined?

A. To eliminate the need for domain experts.

B. To ensure that every unusual observation was classified as a failure.

C. To connect known fault evidence, novelty, and temporal reliability while managing false alarms.

D. To replace validation with an attractive two-dimensional embedding.

Correct Answer: C. To connect known fault evidence, novelty, and temporal reliability while managing false alarms.

Question 9

Which statement about representation quality is most defensible?

A. A representation is high quality whenever reconstruction error is low.

B. Linear separability on one benchmark proves fairness and robustness.

C. Quality should be assessed using criteria appropriate to the intended task, including robustness and distribution shift.

D. Representations do not retain information about nuisance variables.

Correct Answer: C. Quality should be assessed using criteria appropriate to the intended task, including robustness and distribution shift.

Question 10

Why can density-based anomaly detection produce misleading conclusions in high-dimensional data?

A. High-dimensional likelihood or density need not correspond to semantic abnormality for the task.

B. Density models can only be trained with class labels.

C. All high-dimensional observations have exactly equal probability.

D. Density-based methods cannot identify normal observations.

Correct Answer: A. High-dimensional likelihood or density need not correspond to semantic abnormality for the task.