Unsupervised Learning and Pattern Discovery

Introduction

Unsupervised learning concerns the discovery of structure in data when no target variable or labelled outcome is supplied. Instead of learning a mapping from inputs to known answers, the analyst seeks regularities such as groups, low-dimensional representations, co-occurring events, unusual observations, or latent factors. This makes unsupervised learning especially valuable when labelling is expensive, phenomena are poorly understood, or the primary objective is exploration.

The absence of labels does not mean the analysis is assumption-free. Every unsupervised method encodes assumptions about similarity, density, geometry, independence, sparsity, or probability. A clustering algorithm may assume that nearby observations belong to the same group; principal component analysis (PCA) assumes that informative variation can be represented by orthogonal directions; association analysis assumes that recurring item combinations are meaningful; and anomaly detection assumes that unusual observations are potentially important rather than merely errors.

A rigorous workflow therefore has two components: computational discovery and substantive validation. A mathematically coherent pattern is not necessarily a useful, stable, or causal pattern. The analyst must examine preprocessing, parameter choices, stability, external evidence, ethical consequences, and the decisions that the pattern will support.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the objectives, assumptions, and differences among clustering, dimensionality reduction, association analysis, anomaly detection, and representation learning.
2. Select distance measures, algorithms, and validation criteria appropriate to the data and business or scientific question.
3. Apply and interpret k-means, hierarchical clustering, DBSCAN, PCA, association rules, and common anomaly-detection methods.
4. Distinguish descriptive patterns from predictive or causal conclusions.
5. Evaluate stability, usefulness, fairness, privacy, and operational risks in unsupervised projects.
6. Translate discovered structures into defensible domain actions.

1. The Unsupervised Learning Problem

Let a dataset be represented by a matrix X with n observations and p variables. In supervised learning, a response vector y is available. In unsupervised learning, only X is observed. The task is to infer a structure Z, such as cluster memberships, latent coordinates, item relationships, or anomaly scores, using an objective function and a set of modelling assumptions.

A general unsupervised workflow is:

1. Define the decision or discovery question.
2. Audit variables, missingness, measurement scales, leakage, and sampling.
3. Transform and scale variables appropriately.
4. Fit one or more candidate methods.
5. Validate internal quality, stability, external relevance, and operational usefulness.
6. Interpret patterns with domain experts.
7. Monitor whether the discovered structure persists after deployment.

Similarity is central. For two numerical vectors x and z, Euclidean distance is

where x_j and z_j are values on variable j. It is appropriate when variables are commensurate or have been suitably scaled and when straight-line geometric distance is meaningful. Manhattan distance is

and can be more robust to individual large coordinate differences. For binary or categorical data, measures such as Jaccard similarity or Gower distance are often preferable. A distance measure is not a neutral technical detail: it determines what “similar” means.

Standardisation commonly uses

where μ_j is the sample mean and σ_j is the sample standard deviation of variable j. This produces dimensionless values, assuming σ_j > 0. Standardisation is useful when variables have different units, but it may give extreme observations substantial influence. Robust alternatives use the median and interquartile range.

2. Clustering: Discovering Groups

Clustering partitions or organizes observations so that members of the same group are more similar under a chosen criterion than members of different groups. There is no universally correct number of clusters. The appropriate result depends on the geometry of the data, the distance measure, the algorithm, and the intended application.

2.1 k-Means Clustering

For a selected number K of clusters, k-means assigns each observation to one cluster and estimates a centroid for each cluster. Its objective is to minimize within-cluster sum of squares:

Here, C_k is cluster k, μ_k is its centroid, and ||·|| is Euclidean norm. The algorithm alternates between assigning each point to its nearest centroid and recomputing each centroid as the arithmetic mean of assigned points. The procedure converges to a local optimum, not necessarily the global optimum; multiple initializations are therefore recommended.

K-means works best for numeric data with compact, approximately spherical clusters of comparable scale. It is sensitive to outliers, variable scaling, initialization, and the choice of K. It is unsuitable when clusters are strongly non-spherical or when means are not meaningful.

A common internal criterion is the silhouette coefficient. For observation i, let a(i) be its average distance to observations in its own cluster and b(i) the smallest average distance to another cluster. Then

provided the observation is not in a singleton-cluster special case. Values near 1 indicate good separation, values near 0 indicate overlap, and negative values suggest possible misassignment. The mean silhouette is useful for comparison but should not be treated as proof that a particular K is substantively correct.

2.2 Hierarchical Clustering

Hierarchical methods create a nested sequence of partitions displayed as a dendrogram. Agglomerative clustering begins with each observation as a separate cluster and repeatedly merges two clusters. Linkage defines the distance between clusters:

• Single linkage uses the closest pair and can produce chaining.
• Complete linkage uses the farthest pair and tends to form compact groups.
• Average linkage uses the mean pairwise distance.
• Ward’s method merges clusters that cause the smallest increase in within-cluster squared error, usually for Euclidean data.

The dendrogram permits multiple resolutions, but cutting it at a particular height remains an interpretive decision. Hierarchical results can be computationally expensive for large datasets and may be affected by early irreversible merges.

2.3 Density-Based Clustering

DBSCAN defines dense regions using a radius ε and a minimum number of observations, MinPts. Points in sufficiently dense neighborhoods are core points; points reachable from core points may be border points; remaining observations are labelled noise. DBSCAN can discover non-convex shapes and explicitly identify noise. It does not require K in advance. However, it struggles when cluster densities vary substantially or when the scale of variables is inappropriate.

HDBSCAN extends density-based reasoning to a hierarchy of density levels and can be useful when clusters have different densities. Density-based methods require careful parameter diagnostics and should not automatically interpret every noise point as a fraud or failure.

2.4 Cluster Validation

Validation should combine:

• Internal criteria: cohesion, separation, silhouette, Calinski–Harabasz, or Davies–Bouldin measures.
• Stability: agreement across resamples, initializations, time periods, and reasonable preprocessing choices.
• External validity: association with outcomes not used to create clusters, such as retention or clinical response.
• Interpretability: whether clusters have understandable profiles.
• Actionability: whether different interventions are justified and feasible.

Cluster labels are nominal, not inherently ordinal. Cluster 3 is not “more” of a cluster than cluster 1. Profiles should report distributions, uncertainty, sample size, and variables that distinguish groups rather than relying only on centroid values.

3. Dimensionality Reduction

High-dimensional data may contain redundant variables, noise, or correlated measurements. Dimensionality reduction maps X into a lower-dimensional representation while attempting to preserve a chosen property, such as variance, pairwise distance, neighborhood structure, or reconstruction accuracy.

3.1 Principal Component Analysis

PCA is a linear method. After centring the variables, the sample covariance matrix is

where X is the centred n × p matrix. If v_k is a unit eigenvector of S with eigenvalue λ_k, then the kth principal component score is

The eigenvalues satisfy λ_1 ≥ λ_2 ≥ … ≥ λ_p ≥ 0 and represent the variance explained by their corresponding components. The proportion explained by the first r components is

This ratio is dimensionless. PCA assumes that directions of large variance are useful for the purpose at hand, an assumption that may fail when low-variance signals are important. PCA components are orthogonal mathematical directions, not necessarily causal factors. Loadings require careful interpretation, especially after scaling.

3.2 Nonlinear and Probabilistic Methods

t-SNE emphasizes preservation of local neighborhoods for visualization. It is useful for exploratory plots but can distort global distances, cluster sizes, and apparent gaps. Results depend on perplexity, initialization, and random seed. UMAP also emphasizes local structure while often preserving more global organization, but its geometry remains parameter-dependent.

Autoencoders learn an encoder that maps input x to a latent representation h and a decoder that reconstructs x. Training commonly minimizes reconstruction loss:

where f is the encoder, g is the decoder, and ℓ is an appropriate loss such as squared error for continuous data or cross-entropy for suitable binary/probability outputs. Bottleneck size, regularization, architecture, and training data determine what is represented. Latent dimensions are not automatically interpretable or disentangled.

Dimensionality reduction should distinguish visualization from modelling. A two-dimensional plot can reveal candidate structure, but decisions should not be based solely on visual separation.

4. Association Analysis and Pattern Discovery

Association analysis discovers recurring combinations of items or events. In market-basket data, a transaction T is a set of purchased items. An association rule has the form A → B, where A and B are disjoint itemsets.

Support measures prevalence:

Confidence measures conditional frequency:

Lift compares confidence with the baseline prevalence of B:

Lift greater than 1 indicates positive association, lift near 1 indicates independence under the measured data, and lift below 1 indicates negative association. These measures are descriptive, not causal. A high-confidence rule may simply reflect that B is purchased by almost everyone. Rare-item rules require particular caution because their estimates can be unstable.

The Apriori principle states that if an itemset is infrequent, every superset must also be infrequent. This permits pruning when searching for frequent itemsets. FP-Growth uses a compact prefix structure and avoids generating as many candidates. In large-scale applications, analysts should control the number of tested rules, use minimum support and relevance thresholds, and validate promising rules on future data.

Association rules can support recommendation, cross-selling, web navigation, inventory planning, and event correlation. They should not be used to infer that placing A causes B to occur.

5. Anomaly Detection

Anomaly detection identifies observations that differ substantially from a reference pattern. An anomaly may represent fraud, equipment failure, cyber intrusion, data-quality error, or a legitimate rare event. The meaning depends on context.

Methods include:

• Statistical approaches: model a distribution and flag low-probability observations.
• Distance and density methods: identify observations far from or less dense than neighbours.
• Isolation Forest: recursively partitions data; points isolated with fewer splits receive higher anomaly scores.
• Local Outlier Factor: compares an observation’s local density with the density of its neighbours.
• One-class support vector machines: learn a boundary around typical data.
• Autoencoder methods: flag observations with unusually high reconstruction error.

For a Gaussian variable x with estimated mean μ and standard deviation σ, a standardized score is

A large absolute z-score may indicate unusualness, but the familiar ±2 or ±3 rules depend on distributional assumptions and multiple-testing context. In multivariate settings, the squared Mahalanobis distance is

where μ is the mean vector and Σ is a positive-definite covariance matrix. This measure accounts for correlation and is sensitive to estimation quality; robust covariance estimation may be necessary.

Anomaly systems should separate detection from response. A score threshold should reflect investigation capacity, false-positive costs, missed-anomaly costs, and feedback. Concept drift requires monitoring because normal behaviour changes over time.

6. Representation Learning and Latent Structure

Representation learning seeks features that make relevant structure easier to model. Unlike manually engineered variables, learned representations may capture nonlinear interactions, temporal context, or semantic relationships. Examples include word embeddings, graph embeddings, convolutional feature maps, variational autoencoders, and contrastive representations.

A representation is useful when it improves a clearly defined downstream task, supports transfer across related tasks, or yields meaningful compression without unacceptable information loss. Unsupervised pretraining can exploit large unlabelled datasets, but the learned representation may reproduce sampling bias, encode sensitive attributes, or discard minority-group information. Evaluation should therefore include subgroup performance, robustness, leakage checks, and interpretability appropriate to the application.

7. Worked Example: Customer Behaviour Segmentation

Consider an illustrative online retailer with 20,000 customers. The project team wants to design retention interventions, but no reliable customer segments exist. It constructs features from a six-month observation window: order frequency, monetary value, average basket size, discount proportion, days since last purchase, and product-category diversity.

First, the team removes direct identifiers, caps demonstrably erroneous transaction values, imputes missing category counts with a documented rule, and standardizes the numeric features. It tests k-means for K from 2 to 8, using 50 initializations per K. Mean silhouette is highest at K = 3, but K = 4 provides a more stable and operationally distinct solution across bootstrap samples.

The four segments are summarized as:

The team does not label Segment C “certain churners.” It evaluates the segmentation on a later three-month period and finds that Segment C has a substantially higher inactivity rate, but not universally. It pilots different interventions with randomized holdout groups. Only the intervention for Segment C produces a statistically and commercially meaningful improvement in reactivation.

The analysis demonstrates an important principle: clustering discovers a descriptive structure; experimentation is needed to establish whether an action changes outcomes. The segmentation is also monitored for drift because seasonal events may change purchasing behaviour.

8. From Pattern to Decision

A pattern becomes actionable only when five questions have credible answers:

1. What exactly was discovered, and under which representation and parameters?
2. Is it stable under sampling, time, and reasonable modelling choices?
3. Does it correspond to a domain-relevant phenomenon rather than an artefact?
4. What decision will change because of the discovery?
5. What are the costs, risks, and feedback mechanisms after implementation?

Analysts should maintain a pattern-discovery record containing data versions, transformations, random seeds, parameters, validation results, rejected alternatives, and interpretation notes. This improves reproducibility and prevents selective reporting of attractive plots or rules.

9. Limitations, Ethics, and Governance

Unsupervised models can amplify existing inequities. A cluster may proxy for protected characteristics because location, purchasing behaviour, language, or device type correlates with them. An anomaly detector may disproportionately flag a group whose behaviour is less represented in historical data. Automated segmentation can also lead to price discrimination, exclusion, or intrusive personalization.

Governance practices include data minimization, purpose limitation, access controls, documentation, human review for consequential actions, subgroup audits, and appeal mechanisms. Personal data used for pattern discovery must comply with applicable privacy and data-protection requirements. Re-identification risk may remain even after direct identifiers are removed, especially when high-dimensional behavioural data are combined with external sources.

Unsupervised outputs should be treated as hypotheses unless validated. Analysts should avoid causal language, disclose uncertainty, test sensitivity to preprocessing, and assess whether the benefits justify the surveillance or intervention burden. Explainability should focus not only on how a score was calculated but also on why the organization uses the score and what recourse affected individuals have.

10. Contemporary Developments and Best Practices

Modern systems increasingly combine representation learning with clustering, retrieval, graph analysis, and human feedback. Large multimodal models can produce embeddings for text, images, transactions, or events, enabling semantic search and cross-modal grouping. However, embedding spaces inherit training-data bias and may change across model versions. Versioning, benchmark sets, drift monitoring, and access governance are essential.

Best practices are to:

• begin with a domain question rather than an algorithm;
• select similarity measures that match measurement scales;
• scale and transform variables deliberately;
• compare multiple methods and parameter settings;
• evaluate stability and future-period performance;
• use visualizations as evidence, not proof;
• validate actions through experiments where possible;
• document uncertainty and subgroup effects;
• monitor drift, false positives, and unintended consequences.

Chapter Summary

Unsupervised learning discovers structure without labelled outcomes. Clustering identifies groups using a similarity criterion; k-means minimizes within-cluster squared error, hierarchical methods create nested partitions, and density-based methods identify regions of concentration and noise. Dimensionality reduction compresses data, with PCA preserving linear variance and nonlinear methods emphasizing other geometric properties. Association analysis measures recurring combinations using support, confidence, and lift, but rules are not causal explanations. Anomaly detection identifies unusual observations through distributional, distance, density, isolation, boundary, or reconstruction approaches. Representation learning creates latent features for visualization, transfer, retrieval, or downstream modelling.

Reliable pattern discovery requires suitable preprocessing, explicit assumptions, internal and external validation, stability analysis, domain interpretation, and operational monitoring. A discovered pattern should be treated as a hypothesis until its relevance and consequences are established. Ethical governance is essential because unsupervised systems can encode bias, expose private behaviour, and trigger consequential decisions without transparent justification.

Key Terms

Unsupervised learning: Learning structure from data without labelled target outcomes.

Similarity measure: A rule for quantifying resemblance or distance between observations.

Clustering: Organizing observations into groups based on similarity.

Centroid: The mean vector representing a cluster in k-means.

k-means: A partitioning algorithm minimizing within-cluster squared Euclidean distance.

Hierarchical clustering: Clustering that produces a nested sequence of partitions.

DBSCAN: A density-based algorithm identifying dense regions and noise.

Silhouette coefficient: An internal measure comparing within-cluster cohesion with between-cluster separation.

Dimensionality reduction: Mapping data into fewer dimensions while preserving a selected property.

PCA: A linear dimensionality-reduction method based on covariance eigenvectors.

Loading: The contribution of an original variable to a principal component.

Association rule: A directional descriptive relationship of the form A → B.

Support: The proportion of transactions containing an itemset.

Confidence: The conditional proportion containing B among transactions containing A.

Lift: Confidence divided by the baseline prevalence of the consequent.

Anomaly: An observation substantially inconsistent with a reference pattern.

Mahalanobis distance: A covariance-adjusted multivariate distance.

Representation learning: Learning features or embeddings from raw data.

Concept drift: Change over time in the data-generating process or normal behaviour.

Actionability: The extent to which a discovered pattern supports a feasible decision.

Review Questions

Short Answer Questions

1. Why is scaling important before applying Euclidean-distance-based clustering?
2. State the difference between support, confidence, and lift in association analysis.
3. What does a PCA eigenvalue represent?
4. Why does k-means often require multiple initializations?
5. Why should an anomaly score not automatically be interpreted as evidence of wrongdoing?

Long Answer Questions

1. Compare k-means, hierarchical clustering, and DBSCAN in terms of assumptions, strengths, and limitations.
2. Explain PCA mathematically and discuss when maximizing variance may fail to preserve useful information.
3. Describe a rigorous validation framework for a customer segmentation project.
4. Explain why association rules are descriptive rather than necessarily causal.
5. Discuss ethical and governance risks in unsupervised learning and propose safeguards.

Analytical / Application Questions

1. A dataset contains income in currency units, age in years, and transaction counts. Explain how you would prepare it for clustering and justify your distance measure.
2. An anomaly detector flags many observations from a small demographic group. Design an investigation to distinguish genuine anomalies from sampling or model bias.
3. A retailer finds the rule “mobile phone → screen protector” with high confidence but lift close to 1. Interpret the result and recommend whether it should drive a campaign.

Multiple Choice Questions with Answers

Question 1

Which objective does standard k-means minimize?

A. Total between-cluster variance

B. Within-cluster sum of squared distances to centroids

C. Number of principal components

D. Average association-rule lift

Correct Answer: B. Within-cluster sum of squared distances to centroids

Question 2

A lift value of approximately 1 for a rule A → B most directly suggests that:

A. A causes B

B. B never occurs without A

C. A and B occur independently under the measured data

D. The rule has maximum predictive accuracy

Correct Answer: C. A and B occur independently under the measured data

Question 3

Which method is specifically designed to identify dense regions and label some observations as noise?

A. PCA

B. DBSCAN

C. Apriori

D. Ordinary least squares

Correct Answer: B. DBSCAN

Question 4

In PCA, the eigenvalue associated with a component represents the:

A. Number of observations in a cluster

B. Variance explained by that component

C. Probability that an observation is anomalous

D. Minimum support of an itemset

Correct Answer: B. Variance explained by that component

Question 5

If an observation has silhouette coefficient near zero, it generally indicates that the observation:

A. Is equally close to all possible observations

B. Is near the boundary between clusters

C. Must be an extreme outlier

D. Explains all variance in the dataset

Correct Answer: B. Is near the boundary between clusters

Question 6

Which statement about t-SNE is most accurate?

A. It guarantees preservation of global distances

B. It is primarily useful for exploring local-neighbourhood structure

C. It always identifies the true number of clusters

D. It requires labelled response values

Correct Answer: B. It is primarily useful for exploring local-neighbourhood structure

Question 7

For a rule A → B, confidence is calculated as:

A. P(A ∩ B) ÷ P(A)

B. P(A) ÷ P(B)

C. P(A ∩ B) ÷ P(B)

D. P(A) + P(B)

Correct Answer: A. P(A ∩ B) ÷ P(A)

Question 8

Why can a high-confidence association rule be misleading?

A. Confidence cannot be calculated from transaction data

B. The consequent may be common even without the antecedent

C. High confidence proves causality

D. Confidence is always negative for rare items

Correct Answer: B. The consequent may be common even without the antecedent

Question 9

The Mahalanobis distance differs from ordinary Euclidean distance because it:

A. Uses labels to estimate error

B. Accounts for covariance among variables

C. Can only be used for categorical data

D. Measures association-rule support

Correct Answer: B. Accounts for covariance among variables

Question 10

Which practice best supports trustworthy unsupervised learning in a consequential application?

A. Deploying the most visually attractive two-dimensional plot

B. Ignoring subgroup effects because no labels exist

C. Testing stability, future-period performance, and unintended consequences

D. Treating every cluster as a causal category

Correct Answer: C. Testing stability, future-period performance, and unintended consequences