Unsupervised Learning and Pattern Discovery

Introduction

Unsupervised learning analyses data without a provided target variable or class label. Instead of learning a direct mapping from inputs to known outcomes, an unsupervised algorithm searches for structure: groups of similar observations, low-dimensional representations, unusual cases, or recurring associations. This makes unsupervised learning especially valuable during exploratory analysis, when labels are expensive, unavailable, unreliable, or not yet known.

Common applications include customer segmentation, document organisation, image compression, fraud screening, sensor monitoring, genomic analysis, and exploratory business intelligence. However, unsupervised learning does not automatically discover objectively correct categories. Results depend on the representation of the data, the distance measure, the algorithm, and the analyst’s interpretation.

This chapter develops four core methods: k-means clustering, hierarchical clustering, DBSCAN, and principal component analysis (PCA). It also explains cluster evaluation, anomaly detection, responsible interpretation, and a practical workflow for pattern discovery.

Learning Objectives

By the end of this chapter, you should be able to:

• explain the purpose and assumptions of unsupervised learning;
• prepare data for clustering and dimensionality reduction;
• apply and interpret k-means, hierarchical clustering, DBSCAN, and PCA;
• calculate and interpret basic clustering and PCA quantities;
• evaluate cluster quality using internal and practical criteria;
• use unsupervised methods for segmentation, anomaly detection, and exploratory analytics;
• recognise limitations, ethical risks, and good professional practice.

1. Foundations of Unsupervised Learning

Let a dataset contain n observations and p measured features. It can be represented as a matrix X with n rows and p columns. In supervised learning, a target y is supplied. In unsupervised learning, the algorithm receives X but not y. The task is therefore exploratory rather than predictive in the usual labelled sense.

Important terms include:

• Observation: one case, record, image, customer, or measurement.
• Feature: a measured attribute of an observation.
• Cluster: a group whose members are more similar to one another than to members of other groups, according to a chosen criterion.
• Centroid: the arithmetic mean position of observations assigned to a cluster.
• Distance or dissimilarity: a numerical measure of how different two observations are.
• Dimensionality reduction: representing data with fewer variables while retaining important information.
• Anomaly: an observation that differs substantially from the prevailing pattern.

Similarity is not absolute. For example, Euclidean distance treats a one-unit change in height and a one-unit change in income as equally important unless the data are scaled. Consequently, preprocessing is part of the modelling decision, not merely a technical preliminary.

Data preparation

Numerical features commonly require standardisation. For feature j, the z-score of observation i is:

Here, x_{ij} is the original value, μ_j is the training-sample mean, and σ_j is the training-sample standard deviation. The result has no physical unit and, when σ_j is non-zero, has mean approximately 0 and standard deviation approximately 1. Standardisation is appropriate when variables are measured on different scales and no feature should dominate solely because of its units. Robust scaling, logarithmic transformation, or domain-specific normalisation may be preferable for heavily skewed data or strong outliers.

For two vectors x and y with p numerical features, Euclidean distance is:

It is suitable when features are numeric, comparably scaled, and straight-line separation is meaningful. Manhattan distance, the sum of absolute coordinate differences, may be more robust to individual large coordinate differences. Categorical variables require suitable representations and often a different dissimilarity measure; blindly assigning numbers to categories can create false order.

2. k-Means Clustering

Principle and objective

k-means partitions n observations into a preselected number k of clusters. Each observation is assigned to the nearest centroid, and each centroid is recalculated as the mean of its assigned observations. The method seeks to minimise the within-cluster sum of squared errors (SSE):

Here, C_r is cluster r, μ_r is its centroid, and ||x_i − μ_r||² is squared Euclidean distance. The objective assumes that compact, roughly spherical clusters are meaningful and that the arithmetic mean is appropriate.

Algorithm

1. Select k initial centroids, often using k-means++ initialisation.
2. Assign each observation to its nearest centroid.
3. Recalculate each centroid as the mean of its assigned observations.
4. Repeat assignment and recalculation until assignments stabilise or improvement becomes negligible.

The algorithm is efficient and easy to interpret, but it can converge to a local rather than global minimum. Running it several times with different initialisations and selecting the lowest-SSE solution is standard practice.

Worked example

Suppose one-dimensional, standardised observations are 1, 2, 3, 10, 11, and 12, and k = 2. A natural partition is C₁ = {1, 2, 3} and C₂ = {10, 11, 12}. The centroids are:

The SSE is (1−2)² + (2−2)² + (3−2)² + (10−11)² + (11−11)² + (12−11)² = 4. A competing partition would generally have a much larger SSE because it would mix low and high observations. This calculation illustrates the objective; it does not prove that two groups are substantively meaningful.

Choosing k

The elbow method plots SSE against k. SSE cannot increase when more clusters are allowed, so the analyst looks for a point beyond which additional clusters produce only modest improvement. The silhouette coefficient provides another perspective. For observation i, let a(i) be its average distance to observations in its own cluster and b(i) the smallest average distance to another cluster. Then:

For a(i) and b(i) defined on the same non-negative distance scale, s(i) lies between −1 and 1. Values near 1 indicate good separation, values near 0 indicate overlap, and negative values suggest possible misassignment. The mean silhouette is useful, but it should be combined with domain interpretation and stability checks.

Strengths and limitations

k-means is fast, scalable, and effective for compact numerical groups. It is sensitive to scaling, outliers, the selected k, and non-spherical cluster shapes. It is also unsuitable without adaptation for mixed data types or clusters with very different densities. A cluster with a large centroid distance is not necessarily a meaningful social or business category.

3. Hierarchical Clustering

Hierarchical clustering creates a nested structure of groups represented by a dendrogram. In agglomerative clustering, each observation begins as its own cluster. The two closest clusters are repeatedly merged until one cluster remains. The analyst can cut the dendrogram at a selected height to obtain a chosen number of groups.

The definition of distance between clusters is called a linkage criterion:

Ward linkage is closely related to the k-means objective, but it builds a hierarchy rather than directly selecting one partition. Hierarchical methods do not require k in advance, which is useful for exploration. However, early merges cannot be undone, and the dendrogram can be distorted by scaling, linkage choice, and noisy observations. Computational cost may also be substantial for large datasets.

4. DBSCAN: Density-Based Clustering

DBSCAN identifies dense regions separated by areas of lower density. It uses two parameters: ε, the neighbourhood radius, and MinPts, the minimum number of observations required for a dense neighbourhood. A point is a core point when its ε-neighbourhood contains at least MinPts observations, including the point itself. Points reachable from core points may form a cluster; points that do not belong to any dense region are labelled noise.

DBSCAN is valuable because it can find non-spherical clusters and explicitly identify noise. It does not require the number of clusters in advance. Its main weaknesses are sensitivity to ε and MinPts, difficulty with clusters having substantially different densities, and reduced effectiveness in high-dimensional spaces where distances become less discriminating.

A practical approach is to scale features, inspect a k-distance plot, test several parameter settings, and assess whether the resulting groups are stable and meaningful. “Noise” means not assigned under the chosen density definition; it does not automatically mean fraudulent, defective, or erroneous.

5. Principal Component Analysis

PCA is a linear dimensionality-reduction method. It transforms correlated variables into orthogonal principal components ordered by the amount of variance they explain. If X is centred and, where appropriate, standardised, the sample covariance matrix is:

Here, X is the centred n × p data matrix and S is a p × p covariance matrix. PCA finds eigenvectors v_j and eigenvalues λ_j satisfying:

The eigenvectors give component directions, and the eigenvalues give the variance captured along those directions. The first principal component maximises projected variance subject to unit length; subsequent components are orthogonal to earlier components and maximise remaining variance.

The proportion of variance explained by the first q components is:

This ratio is dimensionless and lies between 0 and 1 when eigenvalues are non-negative. PCA is appropriate for linear structure and numeric variables. If features have different units, standardising before PCA prevents high-variance units from dominating. PCA does not use class labels, does not necessarily preserve nonlinear structure, and does not guarantee that components have direct causal meaning.

Interpreting PCA

A component score is the projection of an observation onto a component direction. Loadings describe how strongly original variables contribute to a component. Large positive or negative loadings indicate association with the component, but the sign of an entire component is arbitrary: reversing all signs produces the same statistical solution. Analysts should therefore interpret relative patterns, not treat a sign as inherently meaningful.

PCA can support visualisation, noise reduction, compression, and preprocessing before clustering. It should not be used merely to create a visually attractive two-dimensional plot while hiding important dimensions.

6. Evaluating Discovered Structure

Unsupervised results require several kinds of validation.

Internal evaluation uses the data and labels generated by the algorithm. Examples include SSE, silhouette score, the Davies–Bouldin index, and cluster separation measures. Each metric has assumptions and may favour particular shapes.

Stability evaluation tests whether similar clusters appear after resampling observations, changing initialisation, or making small preprocessing changes. Unstable clusters should not support high-stakes decisions.

External evaluation compares clusters with an independent variable or later outcome, when such information exists. This is not the same as training on a target. External agreement can support usefulness, but disagreement does not automatically invalidate an exploratory grouping.

Practical validation asks whether clusters are understandable, actionable, sufficiently large, and relevant to the problem. Profiles should report distributions and uncertainty, not only a cluster number and a label such as “premium users.”

For two partitions, the adjusted Rand index (ARI) can measure agreement after correcting for chance. It is useful when a reference partition exists, but it is not a universal measure of cluster quality.

7. Pattern Discovery Applications

Segmentation

In customer segmentation, features may include purchase frequency, average order value, product diversity, and recency. A sound workflow defines the decision purpose, selects behaviourally relevant features, handles missingness, scales variables, compares algorithms, profiles clusters, and validates whether segments lead to different appropriate actions. Segment labels should describe observed behaviour rather than make unsupported claims about personality or worth.

Anomaly detection

Anomalies can be found through distance to a centroid, low local density, isolation-based methods, or reconstruction error after dimensionality reduction. A simple standardised distance for one feature is:

Large absolute z values may warrant investigation, but no universal cutoff is valid for every distribution or application. In DBSCAN, noise points may be candidates for review. In k-means, large distance to the assigned centroid can indicate an unusual observation. Human review, contextual rules, and confirmed labels are necessary before consequential action.

Exploratory analytics

PCA plots, dendrograms, cluster profiles, and heat maps help analysts ask better questions. They can reveal redundant variables, subpopulations, measurement problems, or possible data leakage. Exploratory findings generate hypotheses; they do not by themselves establish causation.

8. Illustrative Case: Retail Customer Segmentation

Background. A fictional online retailer wants to improve service design. It has twelve months of transaction data for customers, but no trusted segment labels.

Problem. Management wants to know whether customers differ in purchasing behaviour. It must avoid using protected characteristics or creating segments that result in discriminatory pricing.

Analysis. Analysts construct recency, purchase frequency, monetary value, product-category diversity, and return rate. They remove duplicate accounts, investigate missing values, winsorise extreme monetary values only with documented justification, and standardise the numerical features. Because the expected groups are unknown, they compare k-means, hierarchical clustering, and DBSCAN.

Approach and solution. The elbow and silhouette results suggest that four k-means groups offer a useful compromise. Hierarchical clustering produces broadly similar broad groups, while DBSCAN identifies a small set of unusual purchasing patterns. Analysts examine cluster sizes, bootstrap stability, feature distributions, and business interpretability. They describe groups as “frequent, moderate-value buyers” and “infrequent, high-value buyers,” rather than assigning psychological labels.

Outcome. The retailer designs different communication frequencies and product recommendations, while maintaining the same core service and price rules. A six-month follow-up evaluates whether recommendations improve engagement without increasing complaints or disadvantaging particular demographic groups.

Learning points. The algorithm did not discover a single true customer taxonomy. It produced one useful representation under specified features and assumptions. Validation, governance, and monitoring were as important as the clustering calculation.

9. Professional, Ethical, and Governance Considerations

Unsupervised models can reproduce historical measurement bias. If certain groups are under-recorded, a cluster may reflect data quality rather than real behaviour. Sensitive attributes may be omitted from modelling yet remain indirectly encoded by location, device, language, or purchasing patterns. Therefore, fairness assessment may require examining protected-group outcomes even when protected attributes are not model inputs.

Organisations should document the purpose, data sources, transformations, algorithm, parameters, evaluation results, limitations, and intended use. Personal data should be minimised, protected, and processed under applicable privacy law. Individuals should not be denied essential services solely because an opaque clustering system labels them unusual. Human oversight, appeal mechanisms, access controls, retention limits, and drift monitoring are essential for consequential applications.

10. Best-Practice Workflow

1. Define the analytical question and acceptable use.
2. Audit data quality, missingness, outliers, leakage, and sensitive attributes.
3. Select features based on domain reasoning, not convenience alone.
4. Choose scaling, encoding, distance, and transformation methods deliberately.
5. Establish simple exploratory baselines.
6. Fit several plausible algorithms and test sensitivity to parameters.
7. Evaluate internal quality, stability, external usefulness, and operational meaning.
8. Profile clusters using distributions and representative examples.
9. Communicate uncertainty and avoid causal or psychological claims.
10. Monitor the model and its impacts after deployment.

Contemporary Developments

Modern systems increasingly combine dimensionality reduction with scalable clustering, representation learning, and embedding models. Text and image embeddings can expose semantic structure, but their dimensions may encode undocumented biases and their distances may not have a simple human interpretation. Approximate nearest-neighbour search supports large-scale analysis, while streaming and online methods address changing data. These advances increase capability but do not remove the need for careful feature governance, evaluation, and human interpretation.

Chapter Summary

Unsupervised learning discovers structure without labelled outcomes. Clustering groups observations according to similarity, while dimensionality reduction represents data with fewer variables. k-means is efficient for compact numerical clusters but requires k and is sensitive to scaling and outliers. Hierarchical clustering provides a dendrogram and supports exploration of nested structure. DBSCAN discovers density-based groups and labels sparse observations as noise, but parameter selection and varying density present challenges. PCA creates orthogonal linear components ordered by explained variance and is useful for visualisation, compression, and preprocessing.

Reliable pattern discovery requires more than a high metric score. Analysts must prepare data carefully, compare methods, test stability, interpret clusters with domain knowledge, and distinguish exploratory association from causation. Applications in segmentation, anomaly detection, and exploratory analytics should include privacy, fairness, transparency, human oversight, and post-deployment monitoring.

Key Terms

Unsupervised learning: Learning structure from data without a supplied target label.

Clustering: Grouping observations according to a similarity or distance criterion.

Centroid: The mean position of observations assigned to a cluster.

Euclidean distance: Straight-line distance between numerical vectors.

Standardisation: Rescaling a feature to mean 0 and standard deviation 1.

k-means: A centroid-based algorithm that minimises within-cluster squared error.

SSE: The sum of squared distances from observations to their assigned centroids.

Silhouette coefficient: A measure comparing within-cluster and nearest-other-cluster distances.

Hierarchical clustering: Clustering that creates a nested hierarchy of groups.

Dendrogram: A tree diagram showing hierarchical merges or splits.

Linkage: The rule used to define distance between clusters.

DBSCAN: A density-based clustering algorithm using ε and MinPts.

Core point: A DBSCAN point with sufficiently many neighbours in its ε-neighbourhood.

Noise point: An observation not assigned to a DBSCAN cluster under chosen parameters.

PCA: Principal component analysis, a linear dimensionality-reduction method.

Eigenvector: A direction preserved in orientation by a matrix transformation.

Eigenvalue: The scalar indicating the variance associated with a PCA direction.

Explained variance: The proportion of total variance represented by selected components.

Anomaly: An observation substantially unlike the prevailing pattern.

Stability: The consistency of discovered structure under reasonable perturbations.

Review Questions

Short Answer Questions

1. Why is feature scaling important before many clustering algorithms?
2. State the main optimisation objective of k-means.
3. What information does a dendrogram provide?
4. How does DBSCAN treat sparse observations?
5. What does explained variance represent in PCA?

Long Answer Questions

1. Compare k-means, hierarchical clustering, and DBSCAN in terms of assumptions, strengths, and limitations.
2. Explain how PCA is calculated conceptually and how loadings should be interpreted.
3. Describe a complete workflow for evaluating a customer segmentation model.
4. Explain why internal cluster metrics cannot alone establish that clusters are useful.
5. Discuss ethical and governance risks when unsupervised models are used in high-impact decisions.

Analytical / Application Questions

1. A dataset contains income in dollars, age in years, and purchase count. Explain the preprocessing decisions you would make before k-means and justify them.
2. DBSCAN produces one large cluster, several noise points, and a small second cluster. Propose investigations before deciding whether the result is useful.
3. A two-component PCA solution explains 78% of variance, but the components are difficult to interpret. Should the analyst use it for segmentation? Defend your answer using both statistical and practical considerations.

Multiple Choice Questions with Answers

Question 1

Which objective does standard k-means clustering minimise?

A. Within-cluster sum of squared distances

B. Total number of features

C. Between-class classification error

D. Number of missing values

Correct Answer: A. Within-cluster sum of squared distances

Question 2

Why are numerical features commonly standardised before distance-based clustering?

A. To make every observation identical

B. To prevent units with large scales from dominating distance

C. To guarantee spherical clusters

D. To remove all outliers

Correct Answer: B. To prevent units with large scales from dominating distance

Question 3

What does a DBSCAN noise point represent?

A. A point that is always incorrectly measured

B. A point outside dense clusters under the chosen parameters

C. The centroid of the largest cluster

D. A point with the highest PCA score

Correct Answer: B. A point outside dense clusters under the chosen parameters

Question 4

Which method creates a dendrogram?

A. PCA

B. k-means

C. Hierarchical clustering

D. Standardisation

Correct Answer: C. Hierarchical clustering

Question 5

A silhouette coefficient close to 1 generally indicates that an observation is:

A. Closer to its own cluster than to neighbouring clusters

B. Equally distant from every cluster

C. Certainly an outlier

D. Assigned to a cluster by PCA

Correct Answer: A. Closer to its own cluster than to neighbouring clusters

Question 6

What do PCA eigenvalues indicate?

A. The number of missing values in each feature

B. The variance associated with principal-component directions

C. The number of clusters selected

D. The probability that a label is correct

Correct Answer: B. The variance associated with principal-component directions

Question 7

If two features use very different units and PCA is applied to the covariance matrix without scaling, what may happen?

A. The feature with the larger variance in its units may dominate

B. All components become categorical

C. The data automatically become causal

D. The number of observations changes

Correct Answer: A. The feature with the larger variance in its units may dominate

Question 8

Which clustering method does not require the number of clusters to be specified in advance?

A. k-means

B. DBSCAN

C. Fixed-partition random sampling

D. One-hot encoding

Correct Answer: B. DBSCAN

Question 9

For observations 1, 2, and 3 assigned to one one-dimensional cluster, what is their centroid?

A. 1

B. 2

C. 3

D. 6

Correct Answer: B. 2

Question 10

Why should unsupervised clusters be checked for stability?

A. To determine whether the discovered structure persists under reasonable changes

B. To ensure the algorithm uses a target label

C. To guarantee that clusters represent causal groups

D. To eliminate the need for domain expertise

Correct Answer: A. To determine whether the discovered structure persists under reasonable changes