Unsupervised, Self-Supervised and Generative Learning

Introduction and Context

Much of modern machine learning is trained with labelled examples: an input is paired with a target, and a model learns to predict that target. In many important settings, however, labels are scarce, expensive, private, ambiguous, or unavailable. Medical images may require specialist annotation; industrial sensor streams may contain only a few documented failures; and internet-scale text, images, and audio are available in enormous quantities without carefully curated labels.

Unsupervised, self-supervised, and generative learning address this problem from complementary perspectives. Unsupervised learning seeks structure in data without externally supplied target labels. Self-supervised learning creates training targets from the data itself, often by hiding, transforming, or comparing observations. Generative learning models the data distribution sufficiently well to produce new samples, reconstruct observations, or learn useful latent representations.

These paradigms are not completely separate. An autoencoder is unsupervised because it reconstructs its input; contrastive learning is self-supervised because augmented views provide supervisory relationships; and a variational autoencoder or diffusion model is generative while also learning representations. This chapter develops the principles, practical workflows, limitations, and responsible-use considerations associated with these methods in TensorFlow and PyTorch environments.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Distinguish unsupervised, self-supervised, and generative learning.

2. Explain clustering, dimensionality reduction, autoencoders, and latent spaces.

3. Formulate and interpret reconstruction, contrastive, variational, adversarial, and diffusion objectives.

4. Select suitable methods for representation learning, anomaly detection, synthesis, and limited-label problems.

5. Implement the main workflow using TensorFlow or PyTorch concepts.

6. Evaluate generated samples and learned representations critically.

7. Identify security, ethical, privacy, and governance risks.

1. A Conceptual Framework

Let an observed dataset be \(\mathcal{D}=\{x_i\}_{i=1}^{n}\), where each \(x_i\) may be an image, document, sound segment, or vector of sensor measurements. In supervised learning, the dataset additionally contains labels \(y_i\), and a model learns a conditional relationship \(p(y\mid x)\). In unsupervised learning, the task is to discover properties of \(p(x)\), such as groups, density, dimensions, or regularities.

Self-supervised learning uses a transformation of an observation to construct a training problem. For example, a sentence can be partially masked and the model can predict the missing token. No human label is needed, but the prediction target is still explicitly defined. The resulting encoder is often transferred to a downstream supervised task.

Generative models attempt to represent the data-generating process. Some explicitly estimate a probability density; others learn a procedure for sampling realistic outputs. Generation does not necessarily imply understanding: a model may produce convincing samples while encoding social bias, memorizing private data, or failing on unusual inputs.

A useful design question is: what information should the representation preserve? Clustering seeks group structure; anomaly detection seeks deviations from normality; contrastive learning seeks invariances; and generative modeling seeks enough information to reproduce the distribution or sample from it.

2. Clustering and Classical Representation Discovery

2.1 K-means clustering

K-means partitions observations into \(K\) clusters by minimizing within-cluster squared Euclidean distance:

\[

J=\sum_{i=1}^{n}\left\|x_i-\mu_{c_i}\right\|_2^2,

\]

where \(c_i\in\{1,\ldots,K\}\) is the assignment of observation \(i\), and \(\mu_k\) is the mean of cluster \(k\). The standard algorithm alternates between assigning each point to its nearest centroid and recomputing each centroid as the mean of its assigned points. It converges to a local optimum, not necessarily the global optimum; therefore, multiple random initializations are advisable.

K-means assumes that Euclidean distance is meaningful and tends to work best for compact, similarly sized, approximately spherical clusters. Feature scaling is essential when variables have different units. The number of clusters can be selected using domain knowledge, the elbow heuristic, silhouette analysis, or stability across resamples. These are aids rather than proofs that a natural number of clusters exists.

2.2 Other clustering approaches

Hierarchical clustering builds a tree of nested groups and is useful when a dendrogram is informative. Density-based methods such as DBSCAN identify dense regions and can label isolated points as noise, but their results depend on scale and density parameters. Gaussian mixture models represent data as a weighted mixture of Gaussian components and provide soft membership probabilities. Unlike K-means, a mixture model can represent elliptical clusters, but its assumptions should be checked.

Clustering evaluation requires more than a numerical score. Internal measures such as silhouette score assess separation under a chosen distance. External measures such as adjusted Rand index require reference labels. In professional settings, cluster stability, interpretability, operational usefulness, and fairness are equally important.

3. Autoencoders and Latent Representations

An autoencoder consists of an encoder \(z=f_\theta(x)\) and decoder \(\hat{x}=g_\phi(z)\). It is trained to reconstruct the input. For continuous, standardized data, a common objective is mean squared error:

\[

\mathcal{L}_{AE}=\frac{1}{n}\sum_{i=1}^{n}\|x_i-\hat{x}_i\|_2^2.

\]

For binary-valued data or normalized pixels interpreted as Bernoulli parameters, binary cross-entropy may be more appropriate. The loss must match the observation model and output activation; mean squared error is not automatically correct for every image or signal.

A bottleneck with fewer dimensions than the input can force the model to learn a compressed representation. However, a sufficiently powerful autoencoder may learn an almost identity mapping, especially when the bottleneck is not restrictive or when skip connections bypass it. Regularization, noise injection, sparsity penalties, or architectural constraints can make the representation more useful.

A denoising autoencoder receives a corrupted input \(\tilde{x}\) but reconstructs the clean input \(x\). It therefore learns features stable under the chosen corruption process. The corruption distribution is a design decision: masking may suit tabular features, while noise or geometric transformations may suit images.

For anomaly detection, an autoencoder is commonly trained on predominantly normal data. A reconstruction error such as \(r(x)=\|x-\hat{x}\|\) is compared with a threshold. A threshold selected on a validation set should reflect the cost of false alarms and missed anomalies. Reconstruction error is not a universal anomaly score: an overgeneralizing decoder can reconstruct anomalies well, while a poorly trained model can produce high errors for legitimate rare cases.

Worked example: reconstruction-based thresholding

Suppose validation reconstruction errors for normal observations are 0.08, 0.10, 0.12, 0.11, and 0.09. Their mean is \(0.10\). The population standard deviation is approximately 0.014. If an illustrative threshold is defined as mean plus three standard deviations, the threshold is approximately \(0.142\). A new observation with error 0.18 would be flagged. This threshold is only a rule of thumb; it is not a calibrated probability and should be replaced by a validation procedure when labelled incidents are available.

4. Self-Supervised Representation Learning

Self-supervised learning constructs a pretext or proxy task from unlabelled data. The central principle is that the proxy task should encourage features useful for later tasks rather than merely exploit superficial shortcuts.

4.1 Masked prediction

In masked language modelling, some tokens are hidden and the model predicts them from context. In masked image modelling, patches are hidden and the model predicts pixels or representations. The loss is generally cross-entropy for discrete tokens or an appropriate regression/reconstruction loss for continuous targets. Masking rates, corruption patterns, and target representations affect what is learned.

4.2 Contrastive learning

Contrastive methods create two augmented views of the same observation. An encoder maps them to representations, often followed by a projection head. Positive pairs should be close, while unrelated examples serve as negatives. A commonly used normalized temperature-scaled cross-entropy objective for a positive pair \((i,j)\) is:

\[

\ell_{i,j}=-\log\frac{\exp(\operatorname{sim}(z_i,z_j)/\tau)}{\sum_{k\ne i}\exp(\operatorname{sim}(z_i,z_k)/\tau)},

\]

where \(\operatorname{sim}\) is typically cosine similarity, \(\tau>0\) is a temperature, and the denominator contains candidate representations in the batch. The complete loss averages over directed positive pairs. Larger batches supply more negatives, although memory banks, queues, or non-contrastive objectives can reduce this requirement.

The quality of augmentations is crucial. In an image task, a crop or colour change may preserve identity; in a medical signal, the same transformation may remove clinically meaningful information. False negatives—different examples that belong to the same semantic class—can also damage learning. Evaluation should therefore include linear probing, fine-tuning, nearest-neighbour analysis, and robustness tests rather than contrastive loss alone.

5. Variational Autoencoders

A variational autoencoder (VAE) defines a latent-variable model with prior \(p(z)\), decoder \(p_\phi(x\mid z)\), and approximate posterior \(q_\theta(z\mid x)\). It maximizes the evidence lower bound (ELBO):

\[

\mathcal{L}_{ELBO}(x)=\mathbb{E}_{q_\theta(z\mid x)}[\log p_\phi(x\mid z)]-D_{KL}(q_\theta(z\mid x)\|p(z)).

\]

The first term rewards faithful reconstruction under the chosen likelihood. The KL divergence regularizes the approximate posterior toward the prior, making it possible to sample by drawing \(z\sim p(z)\) and decoding it. Maximizing the ELBO is equivalent to minimizing negative ELBO.

For a diagonal Gaussian posterior, the encoder outputs \(\mu(x)\) and \(\log\sigma^2(x)\), and sampling uses the reparameterization trick:

\[

z=\mu+\sigma\odot\epsilon,\qquad \epsilon\sim\mathcal{N}(0,I).

\]

This separates random sampling from differentiable parameters. A VAE provides a smooth, structured latent space, but samples may be blurrier than those from adversarial or diffusion models. Excessive KL pressure can cause posterior collapse, where the decoder ignores the latent code. KL annealing, architectural changes, or carefully chosen decoder capacity can help, but each changes the model’s behaviour.

6. Generative Adversarial Networks

A GAN consists of a generator \(G\) and discriminator \(D\). The generator maps noise \(z\) to synthetic samples; the discriminator distinguishes real from generated samples. The original minimax formulation is:

\[

\min_G\max_D V(D,G)=\mathbb{E}_{x\sim p_{data}}[\log D(x)]+\mathbb{E}_{z\sim p(z)}[\log(1-D(G(z)))] .

\]

Here, \(D(x)\) is the estimated probability that \(x\) is real. In practice, non-saturating generator loss, Wasserstein objectives, spectral normalization, and gradient penalties are often used to improve optimization. GAN training is a game rather than ordinary single-objective minimization and may suffer from mode collapse, instability, discriminator overpowering, or sensitivity to architecture and learning rates.

GAN evaluation should combine distributional metrics, human or expert assessment, nearest-neighbour checks, and task-specific utility. A generated image that looks realistic may still be duplicated from training data. Conditioning a GAN on class or metadata can improve control but may also reproduce biased or inappropriate associations.

7. Diffusion Models

Diffusion models define a gradual noising process and learn its reversal. In a simple variance-preserving formulation, the forward process adds Gaussian noise through a schedule \(\beta_t\):

\[

q(x_t\mid x_{t-1})=\mathcal{N}(\sqrt{1-\beta_t}x_{t-1},\beta_t I).

\]

With \(\alpha_t=1-\beta_t\) and \(\bar{\alpha}_t=\prod_{s=1}^{t}\alpha_s\), the noisy sample can be written directly as:

\[

x_t=\sqrt{\bar{\alpha}_t}x_0+\sqrt{1-\bar{\alpha}_t}\epsilon,\qquad \epsilon\sim\mathcal{N}(0,I).

\]

A neural network is trained to predict noise, a clean sample, or a related parameterization from \(x_t\) and time \(t\). Sampling begins with noise and repeatedly applies a learned reverse transition. Diffusion models often provide high sample quality and stable training, but sampling can be computationally expensive. Latent diffusion reduces computation by applying diffusion in an autoencoder’s latent space. Conditioning can use text, class labels, images, or other signals.

Diffusion systems inherit risks from their data and conditioning mechanisms: copyrighted material, personal likenesses, stereotypes, unsafe imagery, and prompt-based misuse. Content provenance, access controls, dataset documentation, and evaluation for memorization are important deployment practices.

8. A Practical TensorFlow and PyTorch Workflow

A robust workflow begins with data auditing. Remove accidental duplicates, document provenance, split data by subject or time where leakage is possible, and normalize features using training data only. For images, transformations must preserve the intended semantics. For time series, windows from the same event should not be split across training and test sets.

In PyTorch, an autoencoder typically uses an `nn.Module`, a `DataLoader`, an optimizer such as Adam, `model.train()` during optimization, and `model.eval()` for evaluation. TensorFlow provides analogous functionality through `tf.keras.Model`, `tf.data`, and `GradientTape`. Regardless of framework, the conceptual sequence is the same: encode, decode or predict, compute the objective, backpropagate, update parameters, and monitor validation behaviour.

For contrastive learning, a batch often produces two augmented views per sample. The implementation must correctly construct positive indices, mask self-similarities, normalize embeddings when cosine similarity is intended, and avoid accidental leakage between training and evaluation. For VAEs, implement the KL term with the correct sign and reduction; report reconstruction and KL components separately. For GANs, track generator and discriminator losses but do not treat either loss alone as a reliable quality measure.

Reproducibility requires recording data versions, random seeds, preprocessing, architecture, optimizer, learning-rate schedule, batch size, stopping rule, hardware, and evaluation code. A saved model without this information is difficult to audit or reproduce.

9. Illustrative Case: Predictive Maintenance with Limited Failure Labels

Background

Consider a fictional manufacturing company operating pumps with vibration, temperature, pressure, and acoustic sensors. Only twelve confirmed failures have been labelled, while six months of normal-operation data are available.

Problem

The company wants early warning of abnormal behaviour and a representation useful for later failure classification. A purely supervised classifier would be unreliable because twelve failures do not cover the range of operating conditions.

Analysis

The engineering team first groups data by machine and time to prevent windows from the same incident appearing in both training and testing. Sensors are standardized using training-period statistics. Operating regime, load, and maintenance state are recorded because a change in these variables may resemble a fault.

Approach

A denoising autoencoder is trained primarily on verified normal windows. The corruption process randomly masks a small proportion of sensor values and adds physically plausible noise. Reconstruction error is evaluated by machine and operating regime. A threshold is selected using a validation set and adjusted to the acceptable false-alarm rate.

Next, a temporal encoder is pretrained with a contrastive objective: two views of the same time window are generated using mild masking and jitter, while windows from different times provide candidate negatives. The encoder is then fine-tuned with the twelve labelled failures, with class weighting and leave-one-machine-out validation. Clustering in the latent space is used diagnostically, not as proof of fault categories.

Outcome

The system identifies several anomalous windows before documented failures, but engineers discover that some alarms correspond to a new production recipe rather than mechanical faults. The final deployment therefore combines the anomaly score with operating-context checks and a human review queue. Performance is reported using event-level recall, false alarms per machine-day, detection lead time, and calibration—not only pointwise accuracy.

Learning Points

The case demonstrates that representation learning can use abundant normal data, but operational context and leakage control are essential. Anomaly detection is a decision-support system rather than an autonomous diagnosis. Human feedback and periodic threshold review are necessary because the data distribution changes.

10. Choosing Among Methods

Use clustering when the principal goal is exploratory grouping and a meaningful distance or density notion exists. Use an autoencoder when reconstruction, compression, denoising, or a practical anomaly score is required. Use self-supervised learning when abundant unlabelled data and a downstream task are available. Use a VAE when a probabilistic latent space, interpolation, or controllable sampling is valuable. Use GANs when fast generation and high perceptual sharpness are important, provided training can be stabilized. Use diffusion models when sample quality and flexible conditioning justify greater computation.

These choices are not exclusive. A self-supervised encoder can initialize a VAE; a latent autoencoder can support diffusion; and clustering can explore embeddings learned by contrastive training. The correct selection depends on data, compute, interpretability, latency, risk, and evaluation criteria.

11. Limitations, Ethics, and Governance

Unsupervised systems can amplify dataset bias without an explicit label to expose it. Clusters may reflect measurement practices rather than meaningful social or scientific categories. Learned representations can encode sensitive attributes, enabling unwanted inference. Generative models may produce misinformation, impersonation, discriminatory outputs, or content that reproduces private or copyrighted examples.

Responsible practice includes data minimization, consent and provenance checks, privacy-preserving policies, access control, red-team testing, model cards or system documentation, and clear human accountability. Synthetic data should not automatically be treated as private: rare records may be memorized, and synthetic samples can still reveal sensitive patterns. In regulated domains, validation, audit trails, explainability appropriate to the decision, and documented change management are essential.

12. Contemporary Developments and Best Practices

Current systems increasingly combine large-scale self-supervised pretraining with task-specific fine-tuning, multimodal representations, retrieval, parameter-efficient adaptation, and latent diffusion. Foundation models make transfer learning powerful but increase concerns about data provenance, compute concentration, and hidden failure modes.

Best practices are to define the downstream purpose before selecting a pretext task; prevent train-test leakage; compare against simple baselines; inspect representations visually and quantitatively; evaluate subgroup and distribution-shift performance; test for memorization; calibrate operational thresholds; and retain human review for high-impact decisions. A sophisticated generative model is not automatically better than a simpler, more auditable method.

Chapter Summary

Unsupervised learning discovers structure without human-provided labels, while self-supervised learning creates predictive relationships from unlabelled data. Generative learning models data distributions or sampling procedures and can support synthesis, reconstruction, and representation learning. K-means, density methods, and mixture models provide classical structure discovery. Autoencoders learn compressed representations and can support denoising and anomaly detection, although reconstruction error requires careful calibration. Contrastive and masked prediction methods learn transferable features, with augmentation design and leakage control being central. VAEs provide probabilistic latent spaces through the ELBO and reparameterization trick. GANs generate samples through adversarial training but may be unstable or suffer mode collapse. Diffusion models learn to reverse a gradual noising process and generally offer strong quality at higher sampling cost. Effective deployment requires appropriate evaluation, reproducible engineering, attention to distribution shift, and governance for privacy, bias, misuse, and accountability.

Key Terms

Unsupervised learning: Learning patterns from data without externally supplied target labels.

Self-supervised learning: Learning from targets constructed automatically from the input data.

Generative model: A model designed to represent or sample from a data distribution.

Clustering: Partitioning observations into groups according to similarity or density.

Latent representation: An internal lower-dimensional or transformed description of an observation.

K-means: A clustering algorithm minimizing within-cluster squared Euclidean distances.

Autoencoder: An encoder-decoder model trained, commonly, to reconstruct its input.

Denoising autoencoder: An autoencoder trained to recover clean data from corrupted input.

Anomaly detection: Identifying observations that differ substantially from expected behaviour.

Contrastive learning: Learning representations by bringing positive pairs together and separating candidates.

Positive pair: Two views treated as semantically related during contrastive training.

Embedding: A numerical vector representation produced by a model.

Variational autoencoder: A probabilistic autoencoder trained using an evidence lower bound.

ELBO: Evidence lower bound combining expected reconstruction likelihood and a KL regularizer.

GAN: A generative adversarial network containing a generator and discriminator.

Mode collapse: A GAN failure in which generated samples lack distributional diversity.

Diffusion model: A generative model that learns to reverse a gradual noise-injection process.

Data leakage: Unintended use of information from evaluation data during training.

Distribution shift: A change between training conditions and deployment conditions.

Calibration: Agreement between predicted scores or probabilities and observed outcomes.

Review Questions

Short Answer Questions

1. What distinguishes self-supervised learning from ordinary unsupervised learning?

2. Why should features usually be scaled before applying K-means?

3. What are the encoder and decoder in an autoencoder?

4. State the two conceptual terms in the VAE ELBO.

5. What is mode collapse in a GAN?

Long Answer Questions

1. Compare K-means, hierarchical clustering, DBSCAN, and Gaussian mixture models, including their assumptions and suitable applications.

2. Explain how an autoencoder can be used for anomaly detection and why reconstruction error must be validated carefully.

3. Describe contrastive learning, including positive pairs, augmentations, temperature, and false negatives.

4. Explain the VAE architecture, ELBO, reparameterization trick, and posterior collapse.

5. Compare GANs and diffusion models in terms of training, generation quality, computational cost, and risks.

Analytical / Application Questions

1. Design a self-supervised learning strategy for a hospital’s unlabelled medical images. Specify augmentations, leakage controls, downstream evaluation, and ethical safeguards.

2. A factory’s anomaly detector has excellent pointwise accuracy but generates too many alarms during shift changes. Analyse possible causes and propose an evaluation and redesign plan.

3. Choose between a VAE, GAN, and diffusion model for generating synthetic minority-class images in a safety-critical application. Justify the decision using quality, controllability, privacy, compute, and governance criteria.

Multiple Choice Questions with Answers

Question 1

Which objective is minimized by standard K-means clustering?

A. Within-cluster squared Euclidean distance

B. Cross-entropy between class labels

C. KL divergence between two posteriors

D. Adversarial discriminator accuracy

Correct Answer: A. Within-cluster squared Euclidean distance

Question 2

What is the principal purpose of the bottleneck in an undercomplete autoencoder?

A. To force a compact representation

B. To guarantee class labels

C. To remove the need for validation data

D. To make every input an anomaly

Correct Answer: A. To force a compact representation

Question 3

In contrastive learning, what is a positive pair?

A. Two views treated as semantically related

B. Two samples with the largest Euclidean distance

C. A real sample and a discriminator label

D. Two randomly selected test labels

Correct Answer: A. Two views treated as semantically related

Question 4

What does the KL term in a standard VAE objective encourage?

A. The approximate posterior to remain close to the prior

B. The discriminator to classify real samples correctly

C. Clusters to have equal sizes

D. The input to be converted into one-hot labels

Correct Answer: A. The approximate posterior to remain close to the prior

Question 5

What is the reparameterization trick used for in a VAE?

A. To allow gradient-based learning through latent sampling

B. To select the number of clusters automatically

C. To eliminate the decoder

D. To calculate silhouette score

Correct Answer: A. To allow gradient-based learning through latent sampling

Question 6

Which problem is especially associated with GAN training?

A. Mode collapse

B. Mandatory feature standardization

C. Guaranteed posterior collapse

D. Inability to generate samples

Correct Answer: A. Mode collapse

Question 7

In a basic diffusion forward process, what is progressively added to the data?

A. Gaussian noise

B. Class labels

C. Cluster centroids

D. Human annotations

Correct Answer: A. Gaussian noise

Question 8

Why can an anomaly threshold based only on training reconstruction errors be unsafe?

A. It may not reflect deployment conditions or the costs of errors

B. Reconstruction errors are always negative

C. Thresholds cannot be used with autoencoders

D. It automatically creates labelled failures

Correct Answer: A. It may not reflect deployment conditions or the costs of errors

Question 9

Which practice best prevents temporal leakage in predictive maintenance data?

A. Splitting related windows by machine or event rather than randomly

B. Using all windows from the same failure in both sets

C. Normalizing with test-set statistics

D. Selecting the test set after reviewing its outcomes

Correct Answer: A. Splitting related windows by machine or event rather than randomly

Question 10

Which statement best describes a diffusion model?

A. It learns to reverse a gradual noising process

B. It assigns every point to the nearest centroid

C. It uses only labelled examples

D. It reconstructs inputs without any sampling process

Correct Answer: A. It learns to reverse a gradual noising process