Unsupervised Learning and Pattern Discovery

Introduction

Supervised learning estimates a relationship between inputs and known target values. Unsupervised learning addresses a different problem: the data contain observations and variables, but no authoritative target label is provided. The purpose is to discover structure, regularity, compact representations, or unusual observations within the data itself.

Unsupervised methods are central to exploratory data analysis and are widely used in customer segmentation, document analysis, image processing, fraud detection, bioinformatics, recommendation systems, network security, and scientific research. Their results are not automatically “truth.” A cluster may reflect a meaningful population, a measurement artefact, or merely the scale and variables selected by the analyst. Consequently, successful unsupervised learning combines algorithmic technique with domain knowledge, validation, visualisation, and careful interpretation.

This chapter examines four major families of techniques: clustering, association analysis, anomaly detection, and dimensionality reduction. It also explains how to prepare data, select methods, evaluate results, and communicate findings responsibly.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the purpose and assumptions of unsupervised learning.
2. Prepare mixed, high-dimensional, and sparse data for pattern discovery.
3. Apply and interpret partitioning, hierarchical, density-based, and model-based clustering.
4. Calculate and interpret support, confidence, and lift in association analysis.
5. Distinguish point, contextual, and collective anomalies.
6. Explain principal component analysis and use dimensionality reduction appropriately.
7. Select validation measures that fit the analytical objective.
8. Evaluate ethical, operational, and governance risks in unsupervised applications.

1. The Nature of Unsupervised Learning

Let a dataset be represented by an observation matrix X with n observations and p measured features. In supervised learning, a target vector y is available. In unsupervised learning, the analyst seeks a structure Z, a lower-dimensional representation T, or a set of relationships R without observing y.

The absence of labels creates both freedom and difficulty. The algorithm can reveal unexpected groups, but there is no universally correct answer against which to compare the output. Results depend on:

• the variables included;
• missing-value treatment;
• scaling and transformation;
• the distance or similarity measure;
• algorithm parameters;
• the number of groups requested; and
• the interpretation imposed after modelling.

A useful analytical workflow is:

1. Define the business or research question.
2. Audit data quality and measurement meaning.
3. Transform variables appropriately.
4. Select a method consistent with the data and objective.
5. Fit several plausible specifications.
6. Evaluate stability, separation, usefulness, and plausibility.
7. Profile and validate the discovered patterns.
8. Document limitations and monitor deployment.

Important terminology

• Observation: one case, record, customer, transaction, or measurement.
• Feature: a variable used to describe an observation.
• Similarity: a measure of how alike two observations are.
• Distance: a measure of dissimilarity.
• Cluster: a set of observations considered more similar to each other than to observations outside the set.
• Latent structure: an unobserved pattern inferred from measured variables.
• Dimensionality: the number of variables or coordinates used to represent observations.
• Outlier or anomaly: an observation or pattern that departs substantially from expected behaviour.

2. Data Preparation for Pattern Discovery

Unsupervised models are particularly sensitive to representation. Data preparation is therefore part of the model, not a preliminary administrative task.

2.1 Variable selection

Include variables that represent the phenomenon of interest and exclude identifiers, duplicated fields, leakage variables, and irrelevant administrative codes. An account number, for example, usually has no meaningful geometric distance. Including it may produce arbitrary clusters.

2.2 Scaling numerical variables

Euclidean distance is sensitive to units. If annual income is measured in thousands and age in years, income may dominate a distance calculation. A common standardisation is the z-score:

Here, x_{ij} is the value of feature j for observation i, μ_j is the sample mean of feature j, and σ_j is its sample standard deviation. The transformation is dimensionless and is appropriate when relative deviations from the feature mean are meaningful. It can be distorted by extreme outliers.

Min–max scaling maps a variable to a selected interval, commonly [0, 1]:

It preserves ordering but is also sensitive to extreme minimum and maximum values. Robust scaling, based on the median and interquartile range, is often preferable for heavy-tailed data.

2.3 Categorical, binary, and mixed data

One-hot encoding may represent nominal categories, but it can create many sparse dimensions. For binary variables, the choice between matching and non-matching similarity matters. In purchase data, two customers not buying the same thousands of products are often not genuinely similar; therefore, shared presences may matter more than shared absences.

Mixed numerical and categorical data require a suitable measure, such as Gower dissimilarity, rather than blindly applying Euclidean distance. Missing values should be investigated: imputation can create artificial similarity, while deletion can change the population represented.

2.4 Distance and similarity

For vectors x and y with p numerical features, Euclidean distance is:

It is appropriate when variables are commensurable or properly scaled and straight-line geometric separation is meaningful. Manhattan distance is:

It can be less sensitive to a single large coordinate difference. Cosine similarity focuses on the angle between vectors:

It is frequently useful for text and other high-dimensional sparse data, where orientation matters more than document length. No distance is universally best; the measure should reflect the domain concept of similarity.

3. Clustering Methods

Clustering assigns observations to groups or represents their density structure. It is used for segmentation, exploratory profiling, image quantisation, document organisation, and biological classification.

3.1 Partitioning methods: k-means

The k-means algorithm partitions n observations into k clusters. It seeks centroids that minimise the within-cluster sum of squared Euclidean distances:

C_r is cluster r, x_i is observation i, and μ_r is the mean vector of cluster r. The algorithm alternates between two operations:

1. Assign each observation to the nearest centroid.
2. Recalculate each centroid as the mean of observations assigned to it.

Iterations continue until assignments or the objective change negligibly, or a maximum iteration limit is reached.

K-means is efficient and easy to interpret, but it requires k in advance, favours roughly spherical clusters, is sensitive to scale and outliers, and can converge to a local rather than global optimum. Multiple random initialisations improve the chance of finding a good solution. K-medoids replaces means with actual observations and is more robust to unusual values.

#### Worked example: one-dimensional k-means

Suppose scaled observations are 1, 2, 3, 10, 11, and 12, and k = 2. Initial centroids are 2 and 11. The first three points are closer to 2, while the last three are closer to 11. The updated centroids are:

The assignments do not change, so the algorithm has converged. The within-cluster sum of squared distances is:

This result is meaningful only relative to alternative clusterings and the scale of the data.

3.2 Choosing the number of clusters

The elbow method plots within-cluster sum of squares against k and looks for a point after which additional clusters produce diminishing improvement. It is heuristic rather than a proof.

The silhouette coefficient for observation i is:

where a(i) is the mean distance from i to observations in its own cluster, and b(i) is the smallest mean distance from i to observations in another cluster. Values approach 1 when an observation is well separated, are near 0 at a boundary, and can be negative when assignment may be inappropriate. The mean silhouette is useful for comparison, but a high value does not guarantee business relevance.

3.3 Hierarchical clustering

Hierarchical clustering constructs a nested tree of groups called a dendrogram. In agglomerative clustering, each observation begins as its own cluster; the closest pair is repeatedly merged. The distance between clusters is determined by linkage:

• Single linkage: minimum pairwise distance; can create chaining.
• Complete linkage: maximum pairwise distance; tends to create compact groups.
• Average linkage: average pairwise distance.
• Ward linkage: merges that produce the smallest increase in within-cluster squared error, generally for numerical data.

A dendrogram can be cut at different levels, allowing the analyst to examine several resolutions. Hierarchical methods are valuable when the nested relationships themselves matter, but early merges cannot usually be undone and computation may become expensive for large datasets.

3.4 Density-based clustering

Density-based methods define clusters as dense regions separated by sparse regions. DBSCAN uses two important parameters: ε, the neighbourhood radius, and minPts, the minimum number of points required for a dense neighbourhood. A point is a core point if its ε-neighbourhood contains at least minPts points. Points connected to core points form clusters; isolated points may be labelled noise.

DBSCAN can find non-spherical groups and identify noise without specifying the number of clusters. It is sensitive to ε, minPts, scaling, and varying density. HDBSCAN extends the density perspective to hierarchical variable-density structures and can be useful when a single global density threshold is unsuitable.

3.5 Model-based clustering

Model-based approaches assume data arise from a probabilistic mixture. A Gaussian mixture model represents the density as:

where π_r is the mixing weight for component r, π_r ≥ 0, ∑π_r = 1, μ_r is the component mean, Σ_r is its covariance matrix, and 𝒩 denotes a multivariate normal density. Unlike k-means, a Gaussian mixture gives soft membership probabilities. Its assumptions may be inappropriate for skewed, heavy-tailed, or non-elliptical data. Model selection criteria such as AIC or BIC can assist in selecting complexity, but domain validation remains necessary.

4. Association Analysis

Association analysis discovers items or events that occur together. It is especially common in transaction data, where each record is a basket containing a subset of items. An association rule has the form X → Y, where X and Y are disjoint itemsets.

Let N be the number of transactions. Support measures how often X and Y occur together:

Confidence measures the conditional frequency of Y among transactions containing X:

Lift compares the observed co-occurrence with what would be expected if X and Y were independent:

A lift above 1 indicates positive association, lift equal to 1 is consistent with independence, and lift below 1 indicates negative association. These measures are descriptive, not evidence that X causes Y.

Worked example

Assume 1,000 baskets contain bread in 200 baskets, butter in 250 baskets, and both in 100 baskets. Then:

Thus, half of bread purchasers also purchase butter, and the joint occurrence is twice the rate expected from butter’s overall frequency. The rule does not prove that buying bread causes buying butter.

Association mining can generate an enormous number of candidate rules. Apriori reduces search by exploiting the downward-closure principle: if an itemset is infrequent, every larger itemset containing it must also be infrequent. FP-Growth uses a compressed prefix structure and can be more efficient for large transaction collections. Analysts should control for rule multiplicity, avoid relying solely on confidence, and assess whether a rule is actionable and stable over time.

5. Anomaly Detection

Anomaly detection identifies observations that differ substantially from a reference pattern. Three useful categories are:

• Point anomaly: one observation is unusual relative to the rest.
• Contextual anomaly: an observation is unusual in a particular context, such as a transaction at an unusual time.
• Collective anomaly: a sequence or group is unusual even though individual members may appear normal.

Methods include statistical, distance-based, density-based, isolation-based, and reconstruction-based approaches.

A simple standardised residual can flag unusually large deviations:

This approach assumes a meaningful reference distribution and can fail for skewed or multimodal data.

The k-nearest-neighbour distance method scores an observation by its distance to nearby observations. Local Outlier Factor compares local density with the density of neighbouring points, helping detect anomalies in regions with different densities. Isolation Forest randomly partitions data; unusual observations tend to be isolated using fewer partitions. Autoencoders learn to reconstruct inputs, and high reconstruction error can indicate unusual patterns, although the model may reconstruct anomalies if they are common in training data.

Evaluation is difficult because labels are often scarce. Useful approaches include expert review, historical incidents, synthetic test cases, precision at a fixed alert budget, and time-based back-testing. In operational systems, false positives can overwhelm investigators, while false negatives can cause financial or safety harm. Thresholds should therefore reflect the cost of errors, response capacity, and the possibility of concept drift.

6. Dimensionality Reduction

High-dimensional data are difficult to visualise, store, model, and interpret. Dimensionality reduction maps p variables to q variables, where q < p, while attempting to preserve a chosen property such as variance, neighbourhood structure, or reconstruction fidelity.

6.1 Principal component analysis

Principal component analysis (PCA) finds orthogonal directions of maximum variance. After centring variables, the covariance matrix is:

where X is the centred n × p data matrix and S is the p × p sample covariance matrix. PCA obtains eigenvectors v_j and eigenvalues λ_j satisfying:

The jth principal-component score for observation i is the projection:

Eigenvectors are orthogonal and conventionally normalised to unit length. The eigenvalue λ_j is the variance along component j. The proportion of variance explained by the first q components is:

PCA is appropriate for numerical variables when linear combinations and variance preservation are meaningful. Standardisation is generally required when variables have different units or scales. PCA does not necessarily preserve clusters, and high explained variance does not guarantee that the retained components are useful for a particular decision.

6.2 Other reduction methods

• Truncated singular value decomposition is effective for sparse document-term matrices.
• t-SNE is useful for visualising local neighbourhoods but can distort global distances and is not normally suitable as a general-purpose feature transformation without careful parameter analysis.
• UMAP often preserves local and some global structure efficiently, but its output depends on parameters and should not be interpreted as a literal map of the underlying population.
• Non-negative matrix factorisation produces additive, non-negative components that can be interpretable for counts, images, and topics.
• Autoencoders learn nonlinear representations through neural networks but require tuning, sufficient data, and validation against reconstruction and downstream objectives.

A two-dimensional plot is an aid to reasoning, not proof of a natural two-dimensional structure. Apparent islands may result from parameter choices, preprocessing, or projection artefacts.

7. Evaluating and Interpreting Discovered Structure

Unsupervised evaluation should combine several forms of evidence.

Internal validation

Internal measures use the data and assignments themselves. Examples include within-cluster dispersion, silhouette score, the Davies–Bouldin index, and the Calinski–Harabasz index. These measures can disagree because they reward different geometries.

Stability validation

Refit the method on bootstrap samples, time periods, feature subsets, or perturbed initialisations. Stable patterns are more credible than patterns that disappear after small changes. Cluster agreement may be assessed with adjusted Rand index or variation of information when two partitions are compared.

External and operational validation

If partial labels, outcomes, or expert classifications exist, compare discovered groups with them without pretending they are perfect ground truth. More importantly, assess whether the structure improves a legitimate decision: targeted service, inventory planning, risk review, scientific hypothesis generation, or resource allocation.

Profiling clusters

After clustering, describe each group using variables not necessarily used for fitting. Report size, central tendencies, distributions, representative examples, uncertainty, and differences from the overall population. Avoid assigning stigmatizing labels such as “bad customers” to groups that merely have different behavioural profiles.

8. Applied Case: Illustrative Retail Customer Segmentation

Background

A fictional online retailer has 80,000 active customers and wants to design differentiated retention programmes. It records purchase frequency, average order value, time since last purchase, product-category proportions, return rate, and customer-service contacts.

Problem

The marketing team currently uses a single discount for all customers. This is expensive and may reward customers who would have purchased without a discount. The objective is exploratory segmentation, not automatic exclusion from service.

Analysis

The data science team removes customer identifiers, investigates missing returns data, caps or transforms strongly skewed monetary values, and standardises numerical variables. Because category proportions are compositional and highly correlated, the team reviews whether all should be included directly. It compares k-means, hierarchical clustering, and a Gaussian mixture model. Candidate solutions with three through seven groups are assessed using silhouette scores, bootstrap stability, group sizes, and interpretability.

Approach and solution

A five-group solution is selected provisionally because it is reasonably stable and operationally understandable:

1. frequent, high-value customers with low returns;
2. frequent, moderate-value customers who respond to new categories;
3. infrequent but recently acquired customers;
4. formerly active customers with increasing inactivity; and
5. high-return customers requiring service investigation.

The team does not treat the fifth group as fraudulent. It examines product fit, sizing information, delivery problems, and geographic logistics before proposing interventions. A holdout period is used to test whether segment-specific communication improves retention without increasing complaints or unfairly reducing offers.

Outcome

The segmentation supports differentiated experiments: recognition benefits for loyal customers, onboarding education for new customers, and service recovery for customers with high returns. The organisation measures incremental retention against a control group rather than assuming that a cluster description proves campaign effectiveness.

Learning points

The result depends on feature design and scaling, not only on the clustering algorithm. Stability and actionability are both necessary. Segments are descriptive and can change over time. Ethical use requires that customers retain access to essential service and that decisions are not based on protected characteristics or their proxies without appropriate justification and review.

9. Professional, Ethical, and Governance Considerations

Unsupervised outputs can influence prices, access, surveillance, employment, credit, healthcare, and public services. Several risks deserve explicit attention:

• Proxy discrimination: apparently neutral variables may encode race, disability, gender, location, or socioeconomic status.
• Re-identification: unusual clusters or dimensionality-reduction plots may expose individuals in supposedly de-identified data.
• Opacity: stakeholders may treat an algorithmic segment as an objective identity rather than a modelling choice.
• Feedback loops: allocating more attention to a detected group can generate future data that reinforce the original pattern.
• Data drift: customer behaviour, threats, and economic conditions change.
• Multiple testing: searching many rules increases the chance of impressive but spurious associations.
• Automation bias: investigators may accept anomaly alerts without considering context.

Good governance includes documented purpose, lawful data use, data minimisation, access control, reproducible pipelines, versioned parameters, human review for consequential decisions, fairness testing, appeal routes, and monitoring after deployment. Analysts should record the distance measure, transformations, random seeds, selected parameters, validation results, and known limitations.

10. Best-Practice Checklist

Before presenting an unsupervised result, ask:

1. What decision or scientific question does the analysis support?
2. Are the variables conceptually valid, sufficiently complete, and appropriately transformed?
3. Is the distance or likelihood model compatible with the data type?
4. Were scale, outliers, missingness, and high cardinality addressed?
5. Were multiple algorithms or parameter settings compared?
6. Are the patterns stable across samples and time?
7. Do domain experts recognise and challenge the interpretation?
8. Are association rules distinguished from causal claims?
9. Are anomalies evaluated using realistic operational costs?
10. Could the output create unfair treatment, privacy harm, or unjustified automation?

Chapter Summary

Unsupervised learning discovers structure without a supplied target label. Its principal applications include clustering, association analysis, anomaly detection, and dimensionality reduction. Clustering may use centroid-based, hierarchical, density-based, or probabilistic methods; each embodies different assumptions about geometry, density, and membership. Association rules are evaluated using support, confidence, and lift, but association does not establish causation. Anomaly detection must distinguish point, contextual, and collective unusualness and must balance false alerts against missed events. PCA reduces numerical dimensions through eigenvectors of the covariance matrix, while nonlinear methods are mainly visualisation and representation tools whose geometry requires caution.

The quality of an unsupervised result depends on data representation, scaling, similarity definition, parameter choices, validation, stability, domain interpretation, and governance. A useful pattern is not merely statistically visible; it should be robust, meaningful, ethically defensible, and connected to a legitimate analytical or operational purpose.

Key Terms

Unsupervised learning: Learning from data without a supplied target label.

Clustering: Grouping observations according to a chosen similarity or distance measure.

Centroid: The mean location of observations assigned to a cluster.

K-means: A partitioning algorithm that minimises within-cluster squared Euclidean distance.

Hierarchical clustering: Clustering that constructs nested groups represented by a dendrogram.

Linkage: The rule used to measure distance between clusters.

Density-based clustering: Clustering based on dense regions separated by sparse regions.

Association rule: An implication-like relationship between itemsets or events.

Support: The proportion of transactions containing an itemset.

Confidence: The conditional proportion of transactions containing the consequent among those containing the antecedent.

Lift: The ratio of observed conditional frequency to consequent frequency.

Anomaly: An observation or pattern that differs substantially from expected behaviour.

Point anomaly: An individually unusual observation.

Contextual anomaly: An observation that is unusual under particular contextual conditions.

Principal component analysis: A linear transformation that orders orthogonal directions by variance.

Eigenvalue: In PCA, the variance associated with a principal direction.

Dimensionality reduction: Mapping data to fewer variables while preserving a selected property.

Stability validation: Testing whether discovered structure persists under resampling or perturbation.

Concept drift: Change over time in the data-generating process or relationship being analysed.

Review Questions

Short Answer Questions

1. Why can the inclusion of an identifier variable damage a clustering analysis?
2. Distinguish support from confidence in association analysis.
3. What is the difference between a point anomaly and a contextual anomaly?
4. Why is standardisation often needed before applying k-means?
5. What does a PCA eigenvalue represent?

Long Answer Questions

1. Compare k-means, hierarchical clustering, DBSCAN, and Gaussian mixture models in terms of assumptions, advantages, and limitations.
2. Explain how an analyst should choose and validate the number of clusters without treating one metric as definitive.
3. Describe the PCA workflow from centring the data through interpreting explained variance, including circumstances in which PCA may be inappropriate.
4. Explain why high confidence does not necessarily make an association rule useful, and discuss the role of support and lift.
5. Discuss the ethical and governance risks of using unsupervised segments in customer, employment, or public-service decisions.

Analytical / Application Questions

1. A retailer has transaction data in which 2,000 baskets contain coffee, 1,500 contain tea, and 600 contain both, out of 10,000 baskets. Calculate support for coffee → tea, confidence for coffee → tea, and lift for the rule. Interpret the result.
2. A clustering project produces a high silhouette score, but the smallest cluster contains 0.2% of observations and cannot be described by domain experts. What additional analyses should be conducted before deployment?
3. A fraud-monitoring model generates many anomaly alerts immediately after a new payment channel is introduced. Propose a diagnostic and governance plan that distinguishes genuine fraud from concept drift and operational change.

Multiple Choice Questions with Answers

Question 1

Which measure compares the observed co-occurrence of two items with the co-occurrence expected under independence?

A. Support

B. Confidence

C. Lift

D. Within-cluster sum of squares

Correct Answer: C. Lift

Question 2

What objective does standard k-means minimise?

A. The total number of features

B. Within-cluster squared Euclidean distances

C. The number of association rules

D. The largest pairwise distance only

Correct Answer: B. Within-cluster squared Euclidean distances

Question 3

A silhouette value close to 1 generally indicates that an observation is:

A. Well separated from other clusters

B. Definitely an anomaly

C. Missing from the dataset

D. Assigned to every cluster equally

Correct Answer: A. Well separated from other clusters

Question 4

Which clustering method can identify noise points without specifying the number of clusters in advance?

A. DBSCAN

B. PCA

C. Apriori

D. Ordinary least squares

Correct Answer: A. DBSCAN

Question 5

In the illustrative association example, if 100 of 1,000 baskets contain both bread and butter, the support is:

A. 0.01

B. 0.05

C. 0.10

D. 2.00

Correct Answer: C. 0.10

Question 6

In PCA, the eigenvalue associated with a principal component represents:

A. The number of observations in a cluster

B. The variance along that component

C. The probability that an observation is fraudulent

D. The number of missing values

Correct Answer: B. The variance along that component

Question 7

Which situation is a contextual anomaly?

A. A transaction amount that is unusual only at a particular time or location

B. A duplicated column name

C. A normal observation at the centre of a dense group

D. A cluster centroid

Correct Answer: A. A transaction amount that is unusual only at a particular time or location

Question 8

Why can confidence be misleading when evaluating an association rule?

A. It ignores whether the consequent is already very common

B. It cannot be calculated from transaction data

C. It always measures causation

D. It is identical to Euclidean distance

Correct Answer: A. It ignores whether the consequent is already very common

Question 9

Which practice most directly tests whether a discovered cluster is robust?

A. Changing the font on a cluster plot

B. Refitting the method on resampled or perturbed data

C. Removing all small clusters automatically

D. Using an identifier as an additional feature

Correct Answer: B. Refitting the method on resampled or perturbed data

Question 10

What is a major risk of interpreting a two-dimensional t-SNE plot as a literal map of the data?

A. It always preserves every global distance exactly

B. It may distort global distances and apparent group separation

C. It cannot display numerical variables

D. It requires labelled target values

Correct Answer: B. It may distort global distances and apparent group separation