Unsupervised Learning and Pattern Discovery

Introduction

Supervised learning predicts a known target variable. Unsupervised learning addresses a different problem: discovering meaningful structure when the observations have no supplied labels. Given a dataset containing variables such as customer purchases, sensor readings, web activity, or images, an unsupervised method may reveal groups, low-dimensional structure, recurring combinations, or unusual observations.

The objective is not simply to produce visually attractive clusters or compressed plots. A useful discovery must be statistically credible, computationally stable, interpretable, and relevant to a decision. Unsupervised learning is therefore both a modelling activity and an investigative process. The analyst must define the unit of analysis, select appropriate representations, assess data quality, choose a method, validate the result, and connect patterns to domain knowledge.

Typical tasks include:

• Clustering: dividing observations into groups that are internally similar and externally different.
• Dimensionality reduction: representing many variables with fewer dimensions while preserving important information or relationships.
• Association analysis: identifying items or events that occur together.
• Anomaly detection: finding observations that differ substantially from expected behaviour.

These techniques are widely used in customer segmentation, recommendation systems, fraud screening, network security, document analysis, image processing, medical research, and industrial monitoring.

Learning Objectives

After completing this chapter, students should be able to:

1. Explain the purpose, assumptions, and differences among clustering, dimensionality reduction, association analysis, and anomaly detection.
2. Prepare unlabeled data using appropriate transformations, scaling, and similarity measures.
3. Apply and interpret k-means, hierarchical clustering, density-based clustering, principal component analysis, association rules, and common anomaly-detection methods.
4. Evaluate discovered structure using quantitative metrics, stability analysis, visualization, and domain usefulness.
5. Recognize limitations, ethical risks, and governance requirements in unsupervised analysis.
6. Design a practical pattern-discovery workflow and communicate its findings responsibly.

1. The Nature of Unlabeled Data

An unlabeled dataset can be represented as a matrix X with n observations and p variables. The entry xᵢⱼ is the value of variable j for observation i. Unlike supervised learning, there is no response variable y supplied during model fitting.

This absence of labels does not mean that the data are free from assumptions. Every unsupervised result depends on choices about:

• what constitutes an observation;
• which variables are included;
• how missing values and outliers are handled;
• how similarity or distance is measured;
• whether variables are scaled;
• how many groups or dimensions are retained; and
• what counts as a useful pattern.

For example, clustering customers using annual spending measured in currency may produce groups dominated by income or account size. If the aim is to understand purchasing preferences, variables may need to be standardized or transformed so that scale does not determine the answer.

Similarity and distance

For two observations represented by vectors x and z, the Euclidean distance is:

Here, p is the number of variables. Euclidean distance is appropriate mainly for numeric variables whose scales are comparable and for situations where straight-line separation is meaningful. Manhattan distance is:

It can be less sensitive than Euclidean distance to a single large coordinate difference. For binary or set-valued data, Jaccard similarity is often more suitable:

A and B are sets of present attributes or items. Jaccard similarity ignores attributes absent from both observations, which is useful when shared absence is not informative.

Distance selection is a modelling decision. A poor metric can create artificial structure even when the algorithm is correctly implemented.

2. A Reliable Pattern-Discovery Workflow

A professional workflow is iterative rather than automatic.

1. Frame the question. State what decision or investigation the analysis should support.
2. Audit the data. Examine types, missingness, duplicates, distributions, extreme values, and leakage from future information.
3. Construct representations. Aggregate events into meaningful observations and derive variables carefully.
4. Transform and scale. Use logarithms for strongly skewed positive variables, standardization for comparable numeric features, and suitable encodings for categorical variables.
5. Select a method. Match the algorithm to geometry, data type, noise level, and computational constraints.
6. Explore several specifications. Vary the number of clusters, distance measure, random seed, or algorithm where appropriate.
7. Validate. Combine internal metrics, resampling stability, visual inspection, and domain interpretation.
8. Profile and communicate. Describe groups or patterns using variables not used to create them when possible.
9. Operationalize cautiously. Monitor drift, fairness, false positives, and downstream effects.

A pattern is stronger when it persists under reasonable choices. A cluster that disappears after modest changes in scaling or sampling should not be treated as a stable fact.

3. Clustering

Clustering seeks groups of observations such that members of the same group are relatively similar. There is no universally correct clustering because different notions of “group” are possible: compact spherical groups, connected regions, overlapping communities, or groups separated by density gaps.

3.1 k-means clustering

The k-means algorithm partitions n observations into k clusters. It assigns each observation to the nearest centroid and updates each centroid to the mean of its assigned observations. Its objective is to minimize the within-cluster sum of squared distances:

Cₖ is cluster k, μₖ is its centroid, and ||·|| is Euclidean norm. The method assumes that means are meaningful, variables are numeric, and compact approximately spherical clusters are a reasonable model. It is sensitive to scale, initialization, extreme values, and the selected value of K.

The usual procedure is:

1. Choose K initial centroids, often using k-means++ initialization.
2. Assign each observation to its nearest centroid.
3. Recalculate each centroid as the mean of its assigned observations.
4. Repeat assignment and update until assignments or the objective change very little.

Because the objective is non-convex, different initializations can produce different local solutions. Multiple runs should therefore be used.

Worked example. Suppose one-dimensional observations are 1, 2, 3, 10, 11, and 12, and K = 2. Consider clusters C₁ = {1, 2, 3} and C₂ = {10, 11, 12}. Their centroids are μ₁ = 2 and μ₂ = 11. The objective is:

J = (1−2)² + (2−2)² + (3−2)² + (10−11)² + (11−11)² + (12−11)² = 4.

This partition is intuitively strong because the two compact groups are far apart. The example is one-dimensional, so the units of J are squared units of the original measurement.

Choosing K

The elbow method plots the within-cluster sum of squares against K. A point where additional clusters yield only small improvements may be considered, but elbows can be ambiguous. The silhouette coefficient compares cohesion and separation for each observation:

a(i) is the mean distance from observation i to observations in its own cluster; b(i) is the smallest mean distance from i to observations in another cluster. Values range from −1 to 1. Higher values generally indicate better separation, although the metric must be interpreted with the selected distance and data geometry.

Internal metrics do not prove that a segmentation is useful. A business may prefer four actionable customer groups over three mathematically compact groups.

3.2 Hierarchical clustering

Hierarchical clustering creates a nested structure rather than requiring one final K at the beginning. In agglomerative clustering, each observation starts as its own cluster. The algorithm repeatedly merges the closest pair until one cluster remains. A dendrogram displays the sequence and height of merges; cutting it at a selected height produces a partition.

The distance between clusters depends on the linkage rule:

• Single linkage: minimum pairwise distance; can produce chaining.
• Complete linkage: maximum pairwise distance; favours compact groups.
• Average linkage: average pairwise distance.
• Ward linkage: merges that produce the smallest increase in within-cluster squared error; generally associated with numeric data and Euclidean geometry.

Hierarchical clustering is useful when nested relationships matter or when a dendrogram supports interpretation. It can be computationally expensive for large datasets and early merges cannot normally be undone.

3.3 Density-based clustering

DBSCAN identifies dense regions separated by areas of lower density. It uses two major parameters: ε, the neighbourhood radius, and MinPts, the minimum number of observations required for a dense neighbourhood. Core points have at least MinPts observations within ε; border points lie near a core point but do not themselves meet the density condition; remaining observations are labelled noise.

DBSCAN can discover non-spherical clusters and explicitly identify noise. It is useful for spatial data and anomaly screening. However, one global density threshold may fail when clusters have very different densities. Results are also sensitive to ε and feature scaling.

Other methods, such as Gaussian mixture models, represent observations as probabilistic mixtures. A mixture model can assign soft membership probabilities rather than forcing every observation into one group. This is useful when boundaries are uncertain, but distributional assumptions must be checked.

Cluster profiling and validation

After clustering, calculate descriptive statistics for each group: counts, medians, proportions, and representative examples. Use variables not involved in the clustering to test whether the segments differ in meaningful outcomes. Avoid naming groups with unsupported stereotypes. “High-frequency, low-value users” is more defensible than “loyal customers” unless loyalty has been measured.

Validation should include:

• internal cohesion and separation metrics;
• sensitivity to scaling, initialization, and sample changes;
• comparison of alternative distance measures;
• external validation using later outcomes, if available;
• expert review and actionability assessment.

4. Dimensionality Reduction

High-dimensional data are difficult to visualize, interpret, and model. Dimensionality reduction maps p variables into q dimensions, where q < p. The reduced representation may preserve variance, pairwise distances, neighbourhoods, or another objective. Reduction can support visualization, noise filtering, compression, and preprocessing, but it can also hide important information.

4.1 Principal component analysis

Principal component analysis (PCA) constructs orthogonal directions that successively maximize variance. Before PCA, numeric variables are often centered; standardization is appropriate when variables have different units or variances that should contribute comparably.

Let X be a centered n × p data matrix. The sample covariance matrix is:

n is the number of observations and p is the number of variables. PCA finds eigenvectors vⱼ and eigenvalues λⱼ satisfying:

The principal component score for observation i on component j is the projection of its centered vector xᵢ onto vⱼ:

The eigenvalues measure the variance explained by each component. The proportion explained by component j is:

These equations assume numeric variables and finite covariance. PCA captures linear relationships and is sensitive to outliers because covariance and squared deviations are used.

Interpretation. A loading is an element of an eigenvector and indicates how strongly an original variable contributes to a component. A component is not automatically a real causal factor. It is a direction in the chosen representation. Loadings should be interpreted together with variable scales, signs, and domain knowledge. The overall sign of a component is arbitrary: multiplying vⱼ by −1 changes scores and loadings signs but not the underlying solution.

A practical selection rule may retain enough components to explain a chosen proportion of variance, inspect a scree plot, or select dimensions that support a downstream task. No universal threshold guarantees adequacy.

4.2 Nonlinear methods

t-distributed stochastic neighbour embedding (t-SNE) is designed mainly for visualization. It attempts to preserve local neighbourhood relationships, but distances between visually separated groups and the apparent size of groups should not be interpreted literally. Results depend on perplexity, initialization, and random seed.

Uniform Manifold Approximation and Projection (UMAP) also emphasizes local structure and can preserve some broader structure depending on parameters. Neither method should be treated as proof that visible islands represent objectively distinct populations. For reporting, the parameter settings, random seed, preprocessing, and limitations should be documented.

4.3 PCA worked example

Suppose standardized variables have a covariance matrix with eigenvalues 2.4, 1.1, and 0.5. Total variance is 4.0. The first component explains 2.4 ÷ 4.0 = 0.60, or 60%; the first two explain (2.4 + 1.1) ÷ 4.0 = 0.875, or 87.5%. Retaining two components may provide a compact representation, but the decision also depends on the information required by the application.

5. Association Analysis

Association analysis discovers co-occurrence relationships, commonly in transaction data. A transaction database contains a set of items for each transaction. An itemset is a collection of items, such as {bread, milk}. Association rules have the form A → B, where A and B are disjoint itemsets.

Three central measures are:

N is the total number of transactions. Support measures prevalence; confidence measures the conditional frequency of B among transactions containing A; lift compares the rule with the baseline prevalence of B. A lift greater than 1 indicates positive association relative to independence, although it does not establish causality.

Worked example. In 1,000 transactions, 200 contain coffee, 250 contain biscuits, and 100 contain both. For coffee → biscuits:

• Support = 100/1,000 = 0.10, or 10%.
• Confidence = 100/200 = 0.50, or 50%.
• Lift = 0.50/0.25 = 2.0.

Thus, half of coffee transactions contain biscuits, and biscuits are twice as prevalent among coffee transactions as in the full database. This does not show that coffee purchases cause biscuit purchases. It may reflect a shared occasion, promotion, season, or customer type.

The Apriori algorithm uses the downward-closure principle: if an itemset is infrequent, every larger itemset containing it must also be infrequent. This permits pruning of candidate itemsets. FP-Growth uses a compressed frequent-pattern tree and can be more efficient for some databases.

Association mining creates a multiple-comparisons problem. A large database can generate many apparently impressive rules by chance. Minimum support and confidence thresholds, statistical testing, holdout evaluation, redundancy reduction, and business validation are important. Rules should also be assessed for margin, inventory constraints, timing, and whether an intervention is feasible.

6. Anomaly Detection

An anomaly is an observation that is unusual relative to a reference population or expected process. Types include:

• Point anomalies: one observation is unusual by itself.
• Contextual anomalies: an observation is unusual in a particular context, such as high electricity use at night.
• Collective anomalies: a sequence is unusual even though individual values may appear normal.

Anomaly detection is not identical to error detection. An unusual observation may be a genuine rare event, a measurement error, fraud, or a previously unseen legitimate behaviour.

Common methods

Statistical methods model expected distributions. A z-score is:

where μ is the reference mean and σ is the reference standard deviation. A large absolute z-score may flag an observation, but the method assumes that mean and standard deviation are meaningful and can be distorted by outliers. Robust alternatives use the median and median absolute deviation.

Distance-based methods flag observations far from their neighbours or a centre. They require an appropriate metric and are affected by high dimensionality.

Isolation Forest randomly partitions data. Points that are isolated in fewer splits tend to receive higher anomaly scores. It is effective for many tabular settings but requires careful threshold selection.

Local Outlier Factor (LOF) compares the local density around an observation with the densities around its neighbours. It can detect local anomalies but is sensitive to neighbourhood size.

One-class support vector machines learn a boundary around normal data. They can be useful when only examples of normal behaviour are available, but parameter tuning and scaling are important.

Threshold selection should reflect the cost of false positives and false negatives. If human investigation is scarce, a high alert volume may make the system unusable. If missed fraud is extremely costly, a lower threshold may be justified, with additional review.

7. Integrated Case: Illustrative Retail Pattern Discovery

Background

A fictional online retailer, Northstar Market, has 18 months of transaction and browsing data. Management wants to improve campaign design without relying on pre-existing customer labels.

Problem

The retailer asks three questions: Are there useful customer segments? Which products are purchased together? Which transactions should be reviewed for possible account compromise?

Analysis and approach

The analytics team defines the customer-month as the unit for segmentation. Variables include order frequency, monetary value, average basket size, product-category proportions, return rate, and days since last purchase. Monetary variables are log-transformed where strongly skewed, and numeric features are standardized. Variables containing future outcomes are excluded to prevent leakage.

The team compares k-means for K from 2 through 6 and hierarchical clustering using Ward linkage. Silhouette scores favour K = 3, but a four-group solution is more useful operationally and remains stable under repeated samples. Profiles identify:

1. frequent moderate-value customers with broad category interests;
2. infrequent high-value customers concentrated in selected categories;
3. low-frequency customers with high return rates; and
4. new customers with small but recent orders.

The labels are descriptive rather than moral judgments. A campaign experiment is planned before claiming that the segments improve revenue.

For association analysis, the team mines product-category baskets. A rule involving coffee equipment and premium beans has moderate support and lift above 1, but the team checks whether the rule is driven by a holiday promotion. It is used for a test recommendation rather than an automatic price increase.

For anomaly detection, the team combines transaction amount, device novelty, login geography, and purchase velocity. An Isolation Forest ranks suspicious account-months. Analysts review a sample of high-ranked cases and discover that some are legitimate corporate buyers. The threshold is therefore adjusted, and corporate-account status is added as contextual information.

Outcome

The retailer obtains a stable exploratory segmentation, several testable product associations, and a review queue rather than an automatic fraud verdict. Campaign performance and investigation precision are monitored over time.

Learning points

• The unit of analysis determines what a cluster means.
• A mathematically strong pattern may not be actionable.
• Association is not causation.
• Anomaly scores require human and contextual review.
• Unsupervised systems should be evaluated after deployment because customer behaviour and data-generating processes change.

8. Ethics, Governance, and Professional Practice

Unsupervised models can reproduce social inequalities even without using protected attributes directly. Proxy variables such as postcode, language, device type, or purchasing history may encode protected characteristics. A cluster used for differential pricing, access, or service quality can therefore create discriminatory effects.

Important safeguards include:

• document data sources, transformations, parameters, and intended uses;
• minimize collection and restrict access to sensitive data;
• test whether clusters or anomaly flags differ unfairly across relevant groups;
• provide human review and appeal mechanisms for high-impact decisions;
• distinguish exploratory insight from automated decision-making;
• avoid re-identification when publishing visualizations or profiles;
• monitor model and population drift;
• retain audit logs and versioned datasets;
• explain uncertainty and avoid presenting clusters as natural or permanent categories.

Privacy-preserving aggregation, access controls, retention limits, and appropriate legal review are particularly important when transaction, health, location, or behavioural data are used. Generative AI may assist with code or narrative summaries, but its output must be checked for mathematical correctness, data leakage, fabricated interpretations, and unsupported causal claims.

9. Common Failure Modes and Best Practices

Failure: clustering raw variables with incompatible scales. Remedy: inspect units and standardize or use a justified alternative.

Failure: treating one metric as proof. Remedy: combine internal validation, stability, domain review, and downstream evaluation.

Failure: forcing every observation into a cluster. Remedy: use noise-aware or probabilistic methods where appropriate.

Failure: interpreting PCA components as causal factors. Remedy: describe them as mathematical directions and inspect loadings carefully.

Failure: reporting high-confidence association rules without baseline comparison. Remedy: report support, confidence, lift, and statistical or holdout evidence.

Failure: labelling anomalies as fraud automatically. Remedy: use risk-ranked review and investigate false positives.

Failure: using a two-dimensional visualization as ground truth. Remedy: inspect original variables and quantify stability.

Best practice is to make the entire discovery pipeline reproducible: store preprocessing code, random seeds, algorithm versions, parameter values, data snapshots, and interpretation decisions. Communicate what the model found, what it cannot establish, and what action—if any—is justified.

Chapter Summary

Unsupervised learning discovers structure without a supplied target label. Clustering groups observations according to a chosen similarity concept; k-means favours compact numeric groups, hierarchical methods reveal nested structure, and density-based methods can identify irregular groups and noise. Dimensionality reduction, especially PCA, creates lower-dimensional representations by optimizing a stated criterion such as explained variance. Association analysis identifies recurring item combinations using support, confidence, and lift, while anomaly detection ranks observations that depart from expected global, local, or contextual behaviour.

Reliable pattern discovery depends on sound data preparation, appropriate distance or similarity measures, repeated analysis, and critical validation. Internal metrics alone do not establish usefulness. Patterns must be stable, interpretable, ethically acceptable, and connected to a decision or hypothesis. Association does not imply causation, a visual separation is not necessarily a real population distinction, and an anomaly is not automatically an error or crime. Professional practice therefore combines algorithms with domain knowledge, governance, monitoring, and human judgment.

Key Terms

Unsupervised learning: Learning from data without a supplied target label.

Clustering: Partitioning observations into groups based on similarity.

Centroid: The mean vector representing a cluster in k-means.

k-means: An iterative clustering method minimizing within-cluster squared distance.

Hierarchical clustering: Clustering that creates a nested sequence of merges or splits.

Dendrogram: A tree diagram showing hierarchical cluster merges.

Density-based clustering: Clustering based on dense regions separated by sparse regions.

Silhouette coefficient: A measure comparing within-cluster cohesion with nearest-cluster separation.

Dimensionality reduction: Mapping data into fewer variables or dimensions.

PCA: A linear method producing orthogonal directions of decreasing variance.

Eigenvalue: In PCA, the variance associated with a principal component.

Loading: The contribution of an original variable to a component.

Association rule: A directional co-occurrence statement of the form A → B.

Support: The proportion of transactions containing an itemset.

Confidence: The proportion of A transactions that also contain B.

Lift: Confidence divided by the baseline support of the consequent.

Anomaly: An observation that is unusual relative to a reference population or context.

Isolation Forest: A tree-based method that identifies observations isolated quickly by random partitions.

Concept drift: Change over time in the data-generating process or population.

Review Questions

Short Answer Questions

1. Why is feature scaling important before many clustering algorithms?
2. What does the silhouette coefficient attempt to measure?
3. Distinguish support, confidence, and lift in association analysis.
4. Why can PCA signs be reversed without changing the underlying solution?
5. What is the difference between a point anomaly and a contextual anomaly?

Long Answer Questions

1. Explain the k-means algorithm, its objective function, assumptions, and major limitations.
2. Compare hierarchical clustering, density-based clustering, and k-means for practical data analysis.
3. Describe PCA mathematically and explain how eigenvalues and loadings support interpretation.
4. Explain why association rules require more than a high confidence value to be considered useful.
5. Design a complete validation and governance process for an anomaly-detection system.

Analytical / Application Questions

1. A dataset contains annual income, number of transactions, and average basket value. Propose a preprocessing and clustering plan, including how you would assess whether the resulting groups are useful.
2. In 5,000 transactions, an item A occurs 600 times, item B occurs 1,000 times, and both occur 180 times. Calculate support, confidence for A → B, and lift. Interpret the result without making a causal claim.
3. A t-SNE plot shows four visually separated islands, but the groups change substantially across random seeds. What conclusions are justified, what further analyses are needed, and how should the result be communicated?

Multiple Choice Questions with Answers

Question 1

Which objective does standard k-means minimize?

A. Total between-cluster variance only

B. Within-cluster sum of squared Euclidean distances

C. Number of missing values

D. Association-rule confidence

Correct Answer: B. Within-cluster sum of squared Euclidean distances

Question 2

A silhouette value close to 1 generally indicates that an observation is:

A. Well matched to its cluster and separated from other clusters

B. An extreme outlier in every variable

C. Assigned randomly to a cluster

D. Located in a cluster with zero variance

Correct Answer: A. Well matched to its cluster and separated from other clusters

Question 3

Which clustering method explicitly labels some observations as noise using density conditions?

A. PCA

B. Apriori

C. DBSCAN

D. Ordinary least squares

Correct Answer: C. DBSCAN

Question 4

In PCA, an eigenvalue associated with a component represents:

A. The number of missing observations

B. The variance captured by that component

C. The confidence of an association rule

D. The cluster membership probability

Correct Answer: B. The variance captured by that component

Question 5

In association analysis, lift greater than 1 indicates that:

A. The antecedent always causes the consequent

B. The consequent never occurs alone

C. The consequent is more prevalent with the antecedent than in the overall data

D. The rule has 100% support

Correct Answer: C. The consequent is more prevalent with the antecedent than in the overall data

Question 6

In 1,000 transactions, an itemset occurs 80 times. What is its support?

A. 0.8%

B. 8%

C. 80%

D. 800%

Correct Answer: B. 8%

Question 7

Which is an example of a contextual anomaly?

A. A single impossible sensor value caused by a broken parser

B. High electricity usage at midday in summer

C. High electricity usage at midnight during a normally inactive period

D. A duplicate row in a database

Correct Answer: C. High electricity usage at midnight during a normally inactive period

Question 8

Why should anomaly scores not automatically be treated as fraud decisions?

A. All anomaly methods require labelled fraud data

B. Unusual observations may be legitimate and require contextual review

C. Anomaly methods cannot rank observations

D. Fraud is always normally distributed

Correct Answer: B. Unusual observations may be legitimate and require contextual review

Question 9

Which statement about t-SNE is most appropriate?

A. It guarantees that visual distances represent original global distances

B. It is mainly useful for visualizing local neighbourhood structure

C. It always produces the same result regardless of parameters

D. It proves that visible groups are causal categories

Correct Answer: B. It is mainly useful for visualizing local neighbourhood structure

Question 10

Which practice best supports trustworthy unsupervised learning?

A. Use one metric and ignore domain knowledge

B. Hide preprocessing choices to prevent bias

C. Document preprocessing and test stability under reasonable alternatives

D. Assign permanent labels to clusters immediately

Correct Answer: C. Document preprocessing and test stability under reasonable alternatives