Unsupervised Learning and Anomaly Detection

Introduction and Context

Unsupervised learning addresses problems in which observations are available but reliable target labels are absent. Rather than learning a direct mapping from inputs to known outcomes, the model searches for structure: groups of similar observations, regions of high or low probability, recurring item combinations, or observations that do not conform to the dominant pattern.

This makes unsupervised learning particularly valuable in exploratory data analysis and operational settings where labels are expensive, delayed, incomplete, or biased. Examples include segmenting customers, identifying machine operating states, discovering product affinities, detecting unusual financial transactions, and screening network activity for possible attacks.

The absence of labels does not mean that unsupervised learning is assumption-free. Every method embodies assumptions about similarity, density, geometry, independence, noise, or the meaning of an unusual observation. A technically sound analysis therefore combines algorithmic output with domain knowledge, validation, visualisation, and human review.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish clustering, density estimation, association analysis, and anomaly detection;
• select a suitable unsupervised method for a business or engineering problem;
• explain and apply k-means, hierarchical clustering, DBSCAN, Gaussian mixture models, and association-rule mining;
• calculate and interpret common clustering and anomaly-detection measures;
• design a practical anomaly-detection pipeline;
• evaluate results without relying exclusively on labelled accuracy;
• recognise limitations, ethical risks, and governance requirements.

1. The Unsupervised Learning Problem

Let a dataset contain n observations and p measured features. It can be represented as a matrix X = [x₁, x₂, …, xₙ], where xᵢ ∈ ℝᵖ. In supervised learning, a target y is available; in unsupervised learning, the algorithm seeks a useful representation or structure using X alone.

A central issue is the definition of similarity. Euclidean distance is often used for numerical, appropriately scaled features:

Here xᵢᵣ is feature r of observation i. The formula assumes that features are numeric, comparable in scale, and geometrically meaningful. If income is measured in thousands while age is measured in years, the larger numerical scale may dominate. Standardisation is commonly performed using:

where μᵣ is the sample mean and σᵣ is the sample standard deviation for feature r. This produces dimensionless values, assuming σᵣ > 0. Standardisation is not always appropriate: for example, in fraud detection, an extreme value may be the signal of interest, and robust scaling may be preferable.

For mixed data, alternative measures may be required. Manhattan distance, cosine similarity, Jaccard similarity, Gower distance, or domain-specific distances can be more suitable. The distance function is therefore a modelling decision, not merely a software setting.

2. Clustering: Discovering Groups

Clustering partitions or organises observations so that members of the same group are more similar according to a selected criterion than members of different groups. A cluster is not necessarily a naturally existing entity; it may be a useful operational abstraction.

2.1 k-Means Clustering

The k-means algorithm assigns each observation to one of k clusters and represents each cluster by its centroid. It minimises the within-cluster sum of squared errors (SSE):

Cⱼ is cluster j, μⱼ is its centroid, and ||xᵢ − μⱼ||² is squared Euclidean distance. The algorithm alternates between two steps:

1. Assignment: assign each observation to the nearest centroid.
2. Update: replace each centroid with the arithmetic mean of observations assigned to it.

The process stops when assignments stabilise, centroid movement becomes small, or a maximum number of iterations is reached.

A centroid is calculated feature by feature:

The method is computationally efficient and often effective for compact, approximately spherical clusters. However, it requires k in advance, is sensitive to initialisation and outliers, and may perform poorly when clusters have different sizes, densities, or non-spherical shapes. k-means++ initialisation generally improves starting-centre selection.

Worked Example: One-Dimensional k-Means

Consider observations 2, 3, 4, 10, 11, and 12, and choose k = 2. Suppose initial centroids are 3 and 11.

The first assignment gives C₁ = {2, 3, 4} and C₂ = {10, 11, 12}. Updated centroids are:

The assignments do not change, so the algorithm has converged. The SSE is:

This low value reflects two compact groups. It does not prove that two clusters are substantively correct; that requires contextual interpretation and comparison with alternative values of k.

2.2 Choosing the Number of Clusters

The elbow method plots SSE against k. SSE cannot increase when k increases, so the objective is to identify a point after which additional clusters yield only modest improvement. The silhouette coefficient provides a complementary measure. For observation i:

a(i) is the average distance from i to observations in its own cluster, and b(i) is the smallest average distance from i to observations in another cluster. Values approach 1 when the observation is well separated, are near 0 when it lies between clusters, and may be negative when it is probably assigned incorrectly. The average silhouette should be interpreted alongside domain usefulness, stability, and business actionability.

2.3 Hierarchical Clustering

Hierarchical clustering produces a nested sequence of groups displayed as a dendrogram. In agglomerative clustering, each observation begins as a separate cluster. The two closest clusters are repeatedly merged until one cluster remains. A horizontal cut through the dendrogram determines the final number of clusters.

The linkage rule determines distance between clusters:

• Single linkage: minimum pairwise distance; can create chaining.
• Complete linkage: maximum pairwise distance; favours compact groups.
• Average linkage: average pairwise distance.
• Ward linkage: merges clusters causing the smallest increase in within-cluster squared error.

Hierarchical methods are useful when the analyst wants to inspect structure at several resolutions. They can be computationally expensive for very large datasets and are sensitive to distance and linkage choices.

2.4 DBSCAN

Density-Based Spatial Clustering of Applications with Noise (DBSCAN) defines clusters as dense regions separated by sparse regions. It uses two parameters: ε, the neighbourhood radius, and MinPts, the minimum number of points required in that neighbourhood.

A point is a core point if at least MinPts observations lie within its ε-neighbourhood. Points reachable from core points are assigned to a cluster; points that are not sufficiently connected are labelled noise. DBSCAN can discover irregularly shaped clusters and identify noise without specifying the number of clusters. Its weaknesses include sensitivity to ε and MinPts and difficulty handling clusters with strongly different densities or high-dimensional distances.

3. Probabilistic Clustering and Density Estimation

Clustering methods based on geometry assign observations to groups, whereas probabilistic methods model how observations may have been generated. A Gaussian mixture model (GMM) assumes that the data distribution is a weighted combination of K Gaussian components:

πₖ is the mixing weight, μₖ is the mean vector, Σₖ is the covariance matrix, πₖ ≥ 0, and ∑πₖ = 1. The Gaussian density assumption is a modelling approximation and may not fit skewed or multimodal data.

Unlike k-means, a GMM gives soft assignments. The posterior probability that observation xᵢ belongs to component k is:

The parameters are commonly estimated using expectation-maximisation (EM). The expectation step calculates responsibilities γᵢₖ; the maximisation step updates component weights, means, and covariance matrices. These steps repeat until the likelihood improvement is sufficiently small.

Density estimation seeks to estimate the probability density f(x), either parametrically, as in a GMM, or non-parametrically, as in kernel density estimation. A one-dimensional kernel density estimate is:

n is the sample size, h is the bandwidth, and K is a kernel function, often Gaussian. A small h produces a detailed but potentially noisy estimate; a large h produces a smoother but potentially oversimplified estimate. In high dimensions, density estimation suffers from the curse of dimensionality: the volume of the feature space grows rapidly, making observations sparse.

4. Association Analysis

Association analysis discovers items or events that occur together. It is widely used in retail basket analysis, web usage analysis, healthcare co-occurrence studies, and equipment fault diagnosis.

An association rule has the form A → B, where A and B are disjoint itemsets. Three fundamental measures are:

N is the number of transactions. Support measures prevalence, confidence measures the conditional frequency of B given A, and lift compares the rule with what would be expected if A and B were independent. Lift greater than 1 indicates positive association; lift equal to 1 indicates independence under the measure; lift below 1 indicates negative association.

Worked Example: Association Measures

Suppose a store has 1,000 transactions. Bread appears in 300, milk in 400, and both in 180. For the rule bread → milk:

• support = 180/1,000 = 0.18, or 18%;
• confidence = 180/300 = 0.60, or 60%;
• lift = 0.60/0.40 = 1.5.

Thus, 18% of all transactions contain both products, 60% of bread transactions contain milk, and milk is 1.5 times as common among bread transactions as in the full population. This is association, not proof that buying bread causes milk purchases.

The Apriori algorithm reduces search by using the downward-closure property: if an itemset is infrequent, every larger itemset containing it is also infrequent. Frequent-pattern-growth methods avoid generating as many candidate itemsets and can be more efficient for large transaction databases.

Association rules require careful interpretation. High confidence can be misleading when the consequent is common. Rules can also reflect promotions, seasonality, store layout, or customer demographics rather than a meaningful behavioural relationship.

5. Anomaly Detection

An anomaly is an observation that departs substantially from an expected pattern. Terminology varies:

• Point anomaly: one observation is unusual by itself.
• Contextual anomaly: an observation is unusual in a particular context, such as a temperature that is normal annually but abnormal for a specific season.
• Collective anomaly: a sequence or group is unusual even though individual points may appear normal.

Anomaly detection is not identical to classification. Classification learns labelled categories, while anomaly detection often operates with few or no confirmed anomaly labels. In practice, the normal class may be contaminated by undetected anomalies, and the meaning of “normal” may change over time.

5.1 Statistical and Distance-Based Methods

For a normally distributed univariate variable, a standardised score is:

Large absolute values may indicate unusual observations, but the method is unreliable for skewed distributions, heavy tails, or contaminated estimates of μ and σ. Robust alternatives use the median and median absolute deviation (MAD):

A robust score can be formed as:

The factor 0.6745 makes the score approximately comparable to a standard normal score when the data are normal. It should not be used when MAD equals zero without an appropriate alternative.

For multivariate data, the Mahalanobis distance is:

where μ is the mean vector and Σ is the covariance matrix. This distance accounts for correlation and scale. It assumes that Σ is invertible and that the covariance estimate is reliable. In high dimensions or with collinear features, regularisation or dimensionality reduction may be necessary.

5.2 Isolation Forest

Isolation Forest detects anomalies by randomly selecting a feature and a split value. Anomalies tend to be isolated in fewer random partitions because they are sparse and often lie away from dense regions. The algorithm averages path lengths across many isolation trees. Shorter average paths indicate greater abnormality.

It is efficient for large numerical datasets and does not require a distance matrix. However, results depend on feature representation, contamination settings, randomisation, and the assumption that anomalies are easier to isolate than normal observations.

5.3 Local Outlier Factor

Local Outlier Factor (LOF) compares an observation’s local density with the local densities of its neighbours. An observation can be anomalous even if it is not globally distant, provided that it lies in a substantially sparser region than nearby observations. LOF is useful when normal data contain clusters with different densities, but it is sensitive to the number of neighbours and can be unstable in high-dimensional spaces.

5.4 One-Class Support Vector Machine and Autoencoders

A one-class support vector machine learns a boundary around most training observations and identifies points outside that boundary. It can model nonlinear boundaries through kernels but may require careful parameter tuning and scaling.

An autoencoder is a neural network trained to reconstruct its input. If trained primarily on normal data, reconstruction error may serve as an anomaly score:

f is the encoder, g is the decoder, and L(x) is squared reconstruction error. This approach is useful for complex data such as images, sensor streams, or logs, but it can reconstruct anomalies well if the model is too expressive or the training set contains many anomalies. The score must therefore be validated rather than treated as an automatic diagnosis.

6. Evaluating Unsupervised Results

Evaluation is difficult because there may be no ground-truth labels. Useful strategies include:

• Internal validation: silhouette score, SSE, density separation, or likelihood.
• Stability validation: repeat the method across samples, random seeds, or time periods and assess whether structure persists.
• External validation: compare with known categories, later outcomes, or expert judgement without assuming those references are perfect.
• Operational validation: measure whether results improve investigation time, maintenance planning, customer service, or other defined outcomes.
• Human validation: have domain experts review representative members, boundary cases, and alleged anomalies.

For anomaly detection, precision and recall become available when a sufficiently reliable labelled audit set exists:

TP denotes true positives, FP false positives, and FN false negatives. Precision answers “of flagged cases, how many were truly anomalous?” Recall answers “of known anomalies, how many were detected?” In highly imbalanced settings, accuracy is often misleading. Threshold selection should reflect the relative costs of missed anomalies, unnecessary investigations, and delayed decisions.

7. A Professional Workflow

A robust unsupervised-learning project usually follows these stages:

1. Define the decision. Clarify what action will follow a cluster, rule, or alert.
2. Construct the data. Choose an observation window, unit of analysis, and relevant features.
3. Audit quality. Examine missingness, duplicates, leakage, measurement changes, and data drift.
4. Prepare features. Scale numeric variables, encode categorical variables, transform skewed variables, and remove irrelevant identifiers.
5. Select similarity and method. Match the algorithm to geometry, density, data type, and operational constraints.
6. Tune and compare. Use stability, validation measures, computational cost, and expert review.
7. Interpret. Profile clusters, investigate rules, and inspect the most and least typical observations.
8. Set thresholds and controls. Define alert levels, escalation procedures, and review capacity.
9. Deploy with monitoring. Track drift, alert volume, investigation outcomes, and model performance.
10. Reassess regularly. Retrain or redesign when behaviour, systems, or business processes change.

8. Illustrative Case: Detecting Abnormal Industrial Equipment Behaviour

Background

A manufacturing company operates 240 pumps. Each pump records vibration, motor temperature, pressure, flow rate, and energy consumption every five minutes. Preventive maintenance is based mainly on fixed calendar intervals, producing unnecessary maintenance for some pumps while allowing unexpected failures in others.

Problem

The company wants to identify emerging abnormal behaviour early, without a large labelled database of failures. It also wants to distinguish a genuinely unusual pump state from a normal operating state at a different production load.

Analysis

The data science team first segments observations by operating regime because temperature and pressure naturally vary with production demand. Features are robustly scaled, missing sensor intervals are documented, and maintenance events are kept separate for later validation. A clustering analysis identifies four recurring operating regimes. Within each regime, an Isolation Forest and a local-density method are compared.

An anomaly score is not sent directly to engineers. Instead, alerts are generated only when a pump remains in the high-score region for three consecutive readings or shows a rapid score increase. This reduces responses to isolated sensor glitches. Engineers inspect vibration spectra and maintenance records for high-priority cases.

Approach and Solution

The system uses a two-stage design:

1. classify the current operating regime using a clustering model;
2. evaluate abnormality relative to that regime using an anomaly detector.

Thresholds are selected using historical data and a review budget. A sample of low-, medium-, and high-score observations is audited each month to estimate false-alert rates and detect blind spots.

Outcome

In this illustrative scenario, the company finds that several high-score events correspond to loose couplings, while others result from faulty temperature sensors. The system therefore becomes both a maintenance aid and a data-quality monitor. It does not replace engineering diagnosis; it prioritises inspection.

Learning Points

• Contextual anomalies may be missed by a single global threshold.
• Sensor quality must be treated as part of anomaly detection.
• Persistence and rate-of-change rules can reduce alert fatigue.
• Operational success depends on the investigation process, not merely on an algorithmic score.

9. Limitations, Ethics, and Governance

Unsupervised models can create apparently objective groups that reflect historical inequality, incomplete measurement, or inappropriate feature choices. A customer segment may be used to deny service, an anomaly score may trigger intrusive investigation, or a rare but legitimate behaviour may be treated as suspicious.

Important controls include data minimisation, purpose limitation, access control, audit logging, explainable alert reasons, human review, and documented appeal procedures. Sensitive attributes should not be used casually, but simply removing them does not eliminate unfairness because proxy variables may remain. Analysts should test whether alert rates and error rates differ across relevant groups where lawful and appropriate.

Privacy risks are especially important when association analysis reveals sensitive co-occurrences. Data should be aggregated or anonymised where possible, and rules should not be published if they expose individuals or small groups. Governance documentation should specify the data source, feature definitions, model version, threshold, known limitations, review responsibilities, and retention period.

10. Contemporary Developments and Best Practice

Modern systems increasingly combine representation learning with classical unsupervised methods. Embeddings can transform text, images, logs, or graph entities into numerical vectors, after which clustering or nearest-neighbour methods can be applied. Contrastive learning and self-supervised learning use automatically constructed tasks to learn useful representations without manual labels. Streaming algorithms address changing data and limited memory, while conformal and distribution-free methods can provide calibrated uncertainty under stated assumptions.

These developments do not remove the need for careful problem definition. Best practice is to:

• establish a meaningful baseline before adopting a complex model;
• prevent information leakage across time or entities;
• use robust preprocessing and document every transformation;
• compare several algorithms rather than assuming one is universally best;
• inspect examples, not only summary scores;
• measure stability and drift after deployment;
• design alerts around human capacity and consequences;
• retain a clear distinction between correlation, association, and causation.

Chapter Summary

Unsupervised learning discovers structure when target labels are unavailable or incomplete. Clustering methods such as k-means, hierarchical clustering, DBSCAN, and Gaussian mixture models differ in their assumptions about geometry, density, cluster shape, and assignment certainty. Density estimation models where observations are likely to occur, while association analysis identifies recurring item combinations using support, confidence, and lift.

Anomaly detection identifies point, contextual, or collective deviations from expected behaviour. Statistical scores, Mahalanobis distance, Isolation Forest, LOF, one-class SVM, and autoencoders provide different notions of abnormality. No anomaly score is universally valid: data preparation, context, thresholding, validation, and human review are essential.

Professional applications require more than running an algorithm. Analysts must define the decision, select appropriate features and similarity measures, evaluate stability, quantify operational costs, monitor drift, and address privacy, fairness, explainability, and governance. The most useful unsupervised systems support informed exploratory and operational decisions rather than presenting uncertain patterns as unquestionable facts.

Key Terms

Unsupervised learning: Learning structure from data without a specified target label.

Clustering: Organising observations into groups according to similarity.

Centroid: The arithmetic mean representative of a cluster.

k-means: A clustering algorithm that minimises within-cluster squared error.

SSE: The sum of squared distances from observations to their assigned centroids.

Silhouette coefficient: A measure of within-cluster cohesion and between-cluster separation.

Dendrogram: A tree-like display of hierarchical clustering results.

DBSCAN: A density-based method that identifies clusters and noise points.

Gaussian mixture model: A probabilistic model representing data as weighted Gaussian components.

Density estimation: Estimating the probability density of observations.

Association rule: A directional pattern written as A → B between itemsets.

Support: The proportion of transactions containing an itemset.

Confidence: The conditional proportion of transactions containing the consequent given the antecedent.

Lift: The ratio of rule confidence to consequent support.

Anomaly: An observation or pattern that deviates from expected behaviour.

Point anomaly: An individually unusual observation.

Contextual anomaly: An observation that is unusual under particular contextual conditions.

Isolation Forest: A tree-based method that isolates sparse observations using random partitions.

Concept drift: Change over time in the data distribution or the relationship being modelled.

Alert fatigue: Reduced attention caused by excessive or low-value alerts.

Review Questions

Short Answer Questions

1. Why is feature scaling important for distance-based clustering?
2. State the difference between support and confidence in association analysis.
3. What is the purpose of the ε parameter in DBSCAN?
4. Distinguish a point anomaly from a contextual anomaly.
5. Why can accuracy be misleading when evaluating anomaly detection?

Long Answer Questions

1. Explain the k-means algorithm, including its objective function, assumptions, strengths, and limitations.
2. Compare hierarchical clustering, DBSCAN, and Gaussian mixture models in terms of cluster shape, assignment, and parameter requirements.
3. Explain support, confidence, and lift, and discuss why a strong association does not establish causation.
4. Describe a complete professional workflow for deploying an anomaly-detection system.
5. Discuss the ethical and governance risks of using unsupervised models in customer or security applications.

Analytical / Application Questions

1. A dataset contains customer income, age, and annual spending. Explain how you would prepare the variables, choose a clustering method, select the number of clusters, and assess whether the resulting segments are useful.
2. A rule has support 0.12, antecedent support 0.20, and consequent support 0.40. Calculate its confidence and lift, and interpret both values.
3. A factory’s global anomaly detector flags high-temperature readings whenever production volume is high, but engineers state that these readings are normal under that operating regime. Design a contextual anomaly-detection approach and identify two measures you would use to control false alerts.

Multiple Choice Questions with Answers

Question 1

Which objective does standard k-means clustering minimise?

A. The total number of features

B. Within-cluster sum of squared distances

C. The classification error on labelled data

D. The number of association rules

Correct Answer: B. Within-cluster sum of squared distances

Question 2

What does a silhouette value close to 1 generally indicate?

A. The observation is well matched to its cluster and separated from other clusters

B. The observation is certainly an anomaly

C. All clusters have identical sizes

D. The model has used a Gaussian distribution

Correct Answer: A. The observation is well matched to its cluster and separated from other clusters

Question 3

Which DBSCAN characteristic distinguishes it from k-means?

A. It always requires the number of clusters in advance

B. It can identify noise and irregularly shaped dense clusters

C. It uses only labelled observations

D. It assigns every point to a spherical cluster

Correct Answer: B. It can identify noise and irregularly shaped dense clusters

Question 4

A rule has confidence 0.60 and consequent support 0.40. What is its lift?

A. 0.24

B. 0.67

C. 1.50

D. 2.40

Correct Answer: C. 1.50

Question 5

What does lift greater than 1 indicate for an association rule?

A. The antecedent and consequent occur together less often than expected under independence

B. The consequent never occurs without the antecedent

C. The antecedent and consequent have a positive association under the measure

D. The rule proves that the antecedent causes the consequent

Correct Answer: C. The antecedent and consequent have a positive association under the measure

Question 6

Which anomaly type is a sequence unusual as a whole even though individual observations appear normal?

A. Point anomaly

B. Collective anomaly

C. Randomised anomaly

D. Centroid anomaly

Correct Answer: B. Collective anomaly

Question 7

Why can standard z-scores be unreliable for heavily skewed data?

A. They require categorical variables

B. The mean and standard deviation can be strongly affected by skew and outliers

C. They cannot be calculated for numerical variables

D. They always produce values between 0 and 1

Correct Answer: B. The mean and standard deviation can be strongly affected by skew and outliers

Question 8

What is a principal idea behind Isolation Forest?

A. Anomalies usually require fewer random partitions to isolate

B. Anomalies always have the largest class label

C. Every anomaly belongs to the largest cluster

D. Anomalies are detected only through association rules

Correct Answer: A. Anomalies usually require fewer random partitions to isolate

Question 9

If TP = 30 and FP = 10, what is precision?

A. 0.25

B. 0.30

C. 0.75

D. 0.90

Correct Answer: C. 0.75

Question 10

Why should an anomaly-detection system be monitored after deployment?

A. Unsupervised models cannot process numerical data

B. Data distributions and normal behaviour may change over time

C. Monitoring guarantees that all anomalies are causal

D. Clusters never remain stable after training

Correct Answer: B. Data distributions and normal behaviour may change over time