Unsupervised Learning and Pattern Discovery

Introduction

Unsupervised learning discovers structure in data when the observations do not have known target labels. Unlike supervised learning, where a model learns a relationship between predictors and a response variable, unsupervised methods seek groups, lower-dimensional representations, dependencies, or unusual observations. These methods are central to exploratory data analysis because they can reveal patterns that were not anticipated when the data were collected.

In R-based data science, unsupervised learning is applied to customer segmentation, document analysis, image compression, biological discovery, recommender systems, network analysis, process monitoring, and fraud detection. However, discovered patterns are not automatically meaningful. They may arise from scaling choices, missing data, sampling bias, noise, or analyst decisions. Consequently, unsupervised learning requires both computational skill and careful substantive interpretation.

This chapter examines clustering, principal component analysis (PCA), dimensionality reduction, association-rule mining, and anomaly detection. It also discusses validation, reproducibility, ethics, and the translation of discovered structures into professional decisions.

Learning Objectives

By the end of this chapter, the learner should be able to:

1. Explain the purpose and assumptions of major unsupervised-learning methods.

2. Prepare data appropriately for clustering, PCA, association rules, and anomaly detection.

3. Apply k-means, hierarchical clustering, density-based clustering, PCA, and association-rule algorithms in R.

4. Evaluate the stability, separation, interpretability, and usefulness of discovered structures.

5. Select methods for segmentation, profiling, exploratory research, and fraud detection.

6. Identify limitations, ethical risks, and governance requirements associated with pattern discovery.

1. Foundations of Unsupervised Learning

An unsupervised-learning dataset is commonly represented by a matrix X with n observations and p variables. There is no response variable y supplied for training. The objective may be to partition observations into groups, represent them using fewer dimensions, identify co-occurring events, or assign an abnormality score.

Important terminology includes:

• Observation: A unit being analyzed, such as a customer, transaction, or patient.
• Feature: A measured attribute of an observation.
• Similarity or dissimilarity: A rule for judging how alike two observations are.
• Cluster: A set of observations considered relatively homogeneous under a selected criterion.
• Latent structure: An unobserved pattern inferred from measured variables.
• Outlier or anomaly: An observation that is unusually distant, isolated, or inconsistent with a reference population.
• Support: The frequency with which an itemset occurs in transaction data.
• Confidence: The conditional frequency of one itemset given another.
• Lift: The strength of an association relative to independence.

A central methodological principle is that the algorithm does not discover an objective natural structure independently of its design. Distance metric, variable transformation, number of clusters, sampling frame, and analyst purpose all influence the result. Pattern discovery should therefore be treated as an iterative process: understand the data, choose a representation, fit a method, validate the result, interpret it, and test whether it supports a useful decision.

2. Data Preparation and Similarity

2.1 Variable types and preprocessing

Clustering and PCA generally require numerical representations. Categorical variables may be encoded as indicators, but high-cardinality variables can dominate similarity calculations. Mixed-type data may require methods such as Gower dissimilarity, which combines variable-specific comparisons. Dates should usually be transformed into meaningful features such as tenure, recency, weekday, or season rather than used as raw numbers.

Missing values should not be ignored automatically. Mean imputation can reduce variation and create artificial similarity; complete-case analysis can introduce selection bias. Multiple imputation, model-based treatment, or explicit missingness indicators may be more appropriate, depending on the objective.

2.2 Scaling

Suppose two observations have feature vectors x_i and x_j. Euclidean distance is

d(x_i,x_j) = sqrt(sum from k=1 to p of (x_ik - x_jk)^2).

This measure is appropriate when variables are numerical, commensurable, and differences have meaningful magnitudes. If income is measured in thousands and age in years, income may dominate the distance. Standardization replaces each value x_ik with

z_ik = (x_ik - mean(x_k)) / s_k,

where mean(x_k) and s_k are the sample mean and sample standard deviation of feature k. Standardization gives each variable unit variance, assuming s_k is nonzero. It is not always desirable: if magnitude is substantively important, removing scale may discard relevant information. Robust scaling based on medians and interquartile ranges is useful when extreme values are influential.

Other distances include Manhattan distance, which sums absolute coordinate differences, and cosine distance, which focuses on the angle between vectors and is common for text data. The distance must match the data-generating context.

3. Clustering Methods

3.1 Partitioning methods: k-means

K-means partitions n numerical observations into K clusters by minimizing within-cluster squared Euclidean distance. Its objective is

W(K) = sum over clusters C_k and observations x_i in C_k of ||x_i - mu_k||^2,

where mu_k is the arithmetic mean vector of cluster C_k. The algorithm alternates between assigning each observation to its nearest centroid and recomputing each centroid until assignments stabilize or improvement becomes negligible.

K-means is computationally efficient and easy to interpret, but it assumes that clusters are reasonably compact and that means are meaningful. It is sensitive to scaling, outliers, initialization, and the selected K. Different random starts can produce different solutions. In R, a typical workflow is:

```r

x <- scale(customer_data[, c("income", "visits", "spend")])

set.seed(42)

fit <- kmeans(x, centers = 3, nstart = 25)

fit$cluster

fit$centers

```

The `nstart` argument runs the algorithm from multiple initial configurations, reducing—but not eliminating—the risk of a poor local solution. Cluster centers returned for standardized data must be transformed back or interpreted in standard-deviation units.

3.2 Hierarchical clustering

Hierarchical clustering creates a nested sequence of partitions. In agglomerative clustering, each observation begins as a separate cluster. The algorithm repeatedly merges the two most similar clusters until one cluster remains. A dendrogram displays the merge sequence, and a horizontal cut produces a selected number of clusters.

The linkage rule determines the distance between clusters. Single linkage uses the closest pair and can form chaining structures. Complete linkage uses the farthest pair and tends to produce compact clusters. Average linkage averages pairwise distances. Ward’s method merges clusters that produce the smallest increase in within-cluster squared error and is often effective for standardized numerical data.

Hierarchical clustering does not require K at the beginning, which is useful for exploration. However, early merges cannot be undone, and large datasets may require substantial memory. In R:

```r

d <- dist(scale(customer_data[, c("income", "visits", "spend")]))

hc <- hclust(d, method = "ward.D2")

plot(hc)

cluster_id <- cutree(hc, k = 3)

```

The dendrogram is not itself proof that three clusters exist. The cut must be supported by stability, separation, and domain interpretation.

3.3 Density-based clustering

Density-based methods define clusters as dense regions separated by areas of lower density. DBSCAN uses two important parameters: a neighborhood radius epsilon and a minimum number of points. A point is a core point if enough observations occur within its epsilon-neighborhood. Points connected through core points form clusters; isolated observations may be labeled noise.

DBSCAN can discover irregularly shaped groups and identify noise without specifying K. It performs poorly when cluster densities differ substantially or when the feature space is very high-dimensional. Parameter selection should be informed by a k-nearest-neighbor distance plot, domain knowledge, and sensitivity analysis. HDBSCAN extends density-based reasoning to variable-density settings and is available through several R packages.

3.4 Choosing and evaluating clusters

No single criterion determines the correct number of clusters. Common tools include:

• Elbow analysis: Plot total within-cluster sum of squares against K and look for diminishing returns. The elbow may be ambiguous.
• Silhouette width: For observation i, let a(i) be its average distance to observations in its own cluster and b(i) the smallest average distance to another cluster. Its silhouette is

s(i) = [b(i) - a(i)] / max(a(i), b(i)).

Values near 1 indicate good separation, values near 0 indicate overlap, and negative values suggest possible misassignment. The formula assumes a meaningful dissimilarity measure.

• Gap statistic: Compares observed within-cluster dispersion with that expected under a reference null distribution.
• Stability: Refit the method on bootstrap samples or perturbed data and compare assignments using measures such as the adjusted Rand index.
• External validation: If later outcomes or known categories exist, assess whether clusters differ in relevant variables. These variables should not be used improperly to manufacture apparent validation.

A useful cluster is not merely statistically separated. It should be interpretable, sufficiently large, stable, actionable, and appropriate to the decision context.

4. Principal Component Analysis and Dimensionality Reduction

4.1 Purpose of PCA

PCA transforms correlated numerical variables into a smaller set of uncorrelated linear combinations called principal components. It is used for visualization, noise reduction, feature construction, compression, and exploratory investigation of covariance structure.

For centered data matrix X, the first principal direction w_1 is the unit vector that maximizes the variance of Xw_1. Subsequent directions maximize remaining variance subject to being orthogonal to earlier directions. Equivalently, PCA is obtained from the eigenvectors of the covariance matrix (or correlation matrix when variables are standardized). The associated eigenvalues represent the variance captured by each component.

If variables are measured on different scales, PCA on the covariance matrix may be dominated by high-variance variables. PCA on standardized data is equivalent to PCA of the correlation matrix and is often preferred when variables have different units. The choice should be justified, not applied mechanically.

4.2 Interpretation

A component score is the observation’s coordinate on a principal direction. Loadings describe the contribution or association of original variables with a component; exact conventions vary by software, so analysts must document the convention used. A loading sign is arbitrary: multiplying a component by -1 leaves the representation mathematically equivalent.

The explained-variance ratio for component j is

eigenvalue_j / sum of all eigenvalues.

The cumulative ratio indicates how much total variance is represented by the first several components. High explained variance does not guarantee that the retained dimensions preserve the information most relevant to a business or scientific outcome. PCA is unsupervised and optimizes variance, not prediction or causal meaning.

Example in R:

```r

x <- scale(iris[, 1:4])

pca <- prcomp(x, center = TRUE, scale. = TRUE)

summary(pca)

plot(pca$x[, 1], pca$x[, 2], col = iris$Species,

xlab = "PC1", ylab = "PC2")

```

The species labels may be used for interpretation in this illustration, but they were not used to calculate the components.

4.3 Other dimensionality-reduction methods

Multidimensional scaling (MDS) seeks a low-dimensional configuration whose pairwise distances approximate an original dissimilarity matrix. It is useful when distances, rather than variables, are primary. t-SNE emphasizes local neighborhoods and can reveal visual groupings, but its axes have no direct substantive interpretation and results depend on tuning parameters and random initialization. UMAP often preserves local structure while providing better global organization than t-SNE in some settings, but it also depends on parameter choices and should not be treated as proof of clusters.

For exploratory visualization, low-dimensional maps should be accompanied by sensitivity checks and, where possible, analyses in the original feature space. Visual separation can be created by the embedding algorithm itself.

5. Association Rules and Frequent-Pattern Mining

Association-rule mining is designed for transactional data, such as products purchased together or symptoms recorded in patient visits. An itemset is a collection of items. A rule has the form A -> B, where A and B are disjoint itemsets.

For N transactions, support of itemset A is

support(A) = number of transactions containing A / N.

Confidence of A -> B is

confidence(A -> B) = support(A union B) / support(A),

provided support(A) is nonzero. Lift is

lift(A -> B) = confidence(A -> B) / support(B).

Lift greater than 1 indicates positive association relative to independence; lift equal to 1 is consistent with independence; lift below 1 indicates negative association. These measures are descriptive, not causal. A high-confidence rule can be trivial when B is very common, which is why lift and support should also be examined.

The Apriori principle states that if an itemset is infrequent, every larger itemset containing it is also infrequent. This permits pruning of the search space. In R, the `arules` package is commonly used:

```r

library(arules)

rules <- apriori(transactions,

parameter = list(supp = 0.01,

conf = 0.30,

minlen = 2))

inspect(sort(rules, by = "lift")[1:10])

```

Analysts should control for redundant rules, multiple comparisons, seasonality, promotions, and exposure differences. A rule discovered in historical transactions should be tested prospectively before being used for pricing, recommendations, or inventory decisions.

6. Anomaly Detection

Anomaly detection identifies observations that differ substantially from a reference population. A point may be globally unusual, unusual only within a subgroup, or unusual in a temporal sequence. These distinctions matter: a high-value transaction may be normal for a corporate account but anomalous for a student account.

Methods include:

• Statistical methods: z-scores, robust distances, and parametric probability models. They require assumptions about distributions and dependence.
• Distance-based methods: flag observations with unusually large distance from neighbors or a center.
• Local Outlier Factor: compares local density around an observation with the density around its neighbors.
• Isolation Forest: isolates observations through random partitioning; anomalies tend to require fewer partitions.
• One-class classification: learns a boundary around normal observations.
• Autoencoders: neural networks learn to reconstruct normal data; high reconstruction error may indicate an anomaly.

An anomaly score is not automatically a fraud probability. Thresholds should reflect investigation capacity, costs of false positives and false negatives, and changing base rates. In fraud detection, labels may be delayed or selectively observed, creating evaluation bias. Time-based validation and analyst feedback are essential.

7. Worked Application: Customer Segmentation for a Digital Bank

Background

This illustrative case concerns a digital bank seeking to improve customer communication. The bank has twelve months of customer-level data: monthly transaction count, average balance, digital logins, card spending, service contacts, and account tenure. The aim is not to label customers as good or bad, but to identify behaviorally meaningful profiles.

Problem and preparation

The analysts remove direct identifiers, aggregate transactions to the customer level, investigate missing balances, and winsorize extreme spending only after confirming that the values are not data-entry errors. Highly skewed monetary variables are log-transformed. Numerical variables are standardized because their units and ranges differ. Customers with fewer than three months of activity are analyzed separately to avoid confusing limited observation with low engagement.

Analysis

The team compares k-means, Ward hierarchical clustering, and a Gaussian-mixture model. Candidate solutions with two through six clusters are assessed using silhouette width, bootstrap stability, cluster size, and business interpretability. A four-cluster k-means solution is selected because it is stable and supports distinct interventions, although the silhouette score is only moderate, indicating overlap.

The profiles are summarized in original units rather than standardized units:

1. Digitally engaged transactors: frequent card use, high logins, moderate balances, few service contacts.

2. Balance-oriented customers: high balances, low transaction frequency, infrequent logins.

3. Emerging users: short tenure, low activity, and high variability.

4. Service-intensive customers: frequent contacts and moderate activity.

PCA is used only for visualization. The first two components display partial separation, but the team does not claim that the plot proves four natural groups. Association analysis on product holdings shows that travel insurance and foreign-currency features co-occur, but the relationship is examined by tenure and season before campaign design.

Outcome

The bank develops different communication strategies: digital feature education for emerging users, relationship-management offers for balance-oriented customers, self-service support for digitally engaged transactors, and root-cause service improvements for service-intensive customers. A controlled pilot compares outcomes with a randomly selected control group. The segments are not used to deny services, alter credit eligibility, or infer sensitive characteristics.

Learning points

The case demonstrates that segmentation is a decision-support tool rather than an unquestionable taxonomy. Preparation, comparison of algorithms, stability testing, substantive profiling, and prospective evaluation are all necessary. It also shows why customer segments should not be treated as permanent identities: behavior changes, and models require monitoring and periodic redevelopment.

8. Professional, Ethical, and Governance Considerations

Unsupervised methods can reproduce historical inequities even without using protected attributes. Proxy variables such as postcode, device type, language, or purchasing behavior may encode socioeconomic or demographic information. A segmentation that appears commercially useful may result in exclusionary pricing, reduced service quality, or discriminatory surveillance.

Governance should include a documented purpose, data lineage, preprocessing decisions, distance metrics, hyperparameters, random seeds, validation results, and known limitations. Access to transaction and behavioral data should follow privacy, security, retention, and minimization principles. Analysts should distinguish exploratory findings from evidence suitable for operational decisions.

Interpretability is particularly important when clusters or anomaly scores affect people. Users should know whether a decision is based on a statistical profile, and organizations should provide review and appeal mechanisms where appropriate. Differential privacy, aggregation, access controls, and secure computation may reduce exposure, but technical protections do not remove the need for lawful and fair use.

9. Contemporary Developments and Best Practice

Modern workflows increasingly combine unsupervised learning with representation learning, graph analytics, streaming data, and human-in-the-loop review. Embeddings from language and vision models enable clustering of documents, images, and multimodal records, but embedding spaces inherit model biases and may be difficult to interpret. Streaming anomaly detection must address concept drift, delayed labels, and changing normal behavior.

Recommended practice is to:

1. Define the decision or research question before selecting an algorithm.

2. Inspect distributions, missingness, duplicates, leakage, and sampling bias.

3. Choose variables and similarity measures substantively.

4. Scale and transform data transparently.

5. Compare multiple plausible methods and parameter settings.

6. Evaluate separation, stability, sensitivity, and practical usefulness.

7. Profile clusters in original units and avoid causal language for descriptive associations.

8. Validate recommendations prospectively or with controlled experiments.

9. Monitor drift, subgroup performance, false alarms, and unintended effects.

10. Document reproducible code, data versions, random seeds, and governance decisions.

Chapter Summary

Unsupervised learning discovers structure without a predefined target. Clustering partitions observations according to a distance or density principle; k-means is efficient but sensitive to scale, initialization, outliers, and the number of clusters. Hierarchical and density-based methods provide alternatives for nested, irregular, or noisy structures. Cluster quality must be assessed through separation, stability, interpretation, and usefulness rather than a single numerical score.

PCA represents variance through orthogonal linear components and supports visualization, compression, and feature construction. Its components are not automatically causal or substantively meaningful. MDS, t-SNE, and UMAP provide additional representations but require cautious interpretation. Association rules quantify recurring item combinations using support, confidence, and lift; they describe co-occurrence rather than causation. Anomaly detection identifies unusual observations through statistical, distance, density, isolation, or reconstruction-based methods, but thresholds and evaluation must reflect operational costs and changing populations.

Successful pattern discovery depends on data preparation, appropriate similarity definitions, validation, domain knowledge, ethical governance, and prospective testing. In professional settings, discovered patterns should inform decisions while remaining provisional, monitored, and open to revision.

Key Terms

Unsupervised learning: Learning structure from data without a supplied target label.

Clustering: Grouping observations according to similarity or dissimilarity.

Euclidean distance: Square-root distance based on squared differences across features.

Standardization: Rescaling a variable to mean zero and standard deviation one.

K-means: Partitioning algorithm minimizing within-cluster squared distance to centroids.

Hierarchical clustering: Clustering method that creates a nested sequence of merges or splits.

Dendrogram: Tree-like display of hierarchical clustering operations.

DBSCAN: Density-based method that identifies dense groups and noise.

Silhouette width: Measure comparing an observation’s within-cluster and nearest-cluster distances.

Principal component: Orthogonal linear combination capturing variance in PCA.

Loading: Contribution or association of an original variable with a component.

Dimensionality reduction: Representing data with fewer variables or coordinates.

Association rule: Descriptive implication of the form A -> B in transactional data.

Support: Proportion of transactions containing an itemset.

Confidence: Proportion containing B among transactions containing A.

Lift: Confidence divided by the support of the consequent.

Anomaly: Observation that is unusually isolated, distant, or inconsistent.

Concept drift: Change over time in the distribution or meaning of data.

Stability: Consistency of a discovered structure under resampling or perturbation.

Review Questions

Short Answer Questions

1. Why is scaling often necessary before k-means clustering?

2. What does a silhouette value near zero suggest?

3. Distinguish PCA from supervised feature selection.

4. Define support, confidence, and lift for an association rule.

5. Why is an anomaly score not necessarily a fraud probability?

Long Answer Questions

1. Compare k-means, hierarchical clustering, and DBSCAN in terms of assumptions, advantages, and limitations.

2. Explain how PCA is derived from the covariance or correlation matrix and discuss the interpretation of eigenvalues and loadings.

3. Develop a rigorous workflow for evaluating the number and quality of clusters in a customer-segmentation project.

4. Explain why association rules are descriptive rather than causal, and discuss how an organization should validate a discovered rule.

5. Discuss ethical, privacy, and governance risks in using unsupervised learning for customer profiling.

Analytical / Application Questions

1. A dataset contains income in dollars, age in years, and number of transactions. Propose a preprocessing and distance strategy for clustering, and justify each decision.

2. A fraud-detection model produces many alerts for a newly launched product. Design an investigation to determine whether the alerts represent fraud, a data-quality problem, or legitimate concept drift.

3. A two-dimensional UMAP plot shows three visually separate groups, but clustering in the original variables is unstable. How should the analyst report and investigate this apparent contradiction?

Multiple Choice Questions with Answers

Question 1

Which objective does k-means minimize?

A. Total between-cluster variance only

B. Within-cluster sum of squared distances to centroids

C. The number of observations in the largest cluster

D. The average confidence of association rules

Correct Answer: B. Within-cluster sum of squared distances to centroids

Question 2

What is the main effect of standardizing a numerical variable using its sample mean and standard deviation?

A. It gives the variable mean zero and standard deviation one

B. It removes all outliers

C. It converts the variable into a categorical feature

D. It guarantees normally distributed values

Correct Answer: A. It gives the variable mean zero and standard deviation one

Question 3

A silhouette value close to 1 generally indicates that an observation is:

A. Equally close to every cluster

B. Closer to another cluster than to its assigned cluster

C. Well matched to its own cluster and separated from alternatives

D. An extreme outlier within its cluster

Correct Answer: C. Well matched to its own cluster and separated from alternatives

Question 4

In PCA, an eigenvalue associated with a component represents:

A. The number of clusters selected

B. The variance captured by that component

C. The percentage of missing values

D. The classification accuracy of the component

Correct Answer: B. The variance captured by that component

Question 5

If lift(A -> B) equals 1, the rule is most consistent with:

A. Perfect negative association

B. Independence between A and B

C. A causal relationship from A to B

D. A rule with zero support

Correct Answer: B. Independence between A and B

Question 6

Which clustering method can identify noise points without requiring the analyst to specify the number of clusters directly?

A. DBSCAN

B. Ordinary least squares

C. Naive Bayes

D. PCA

Correct Answer: A. DBSCAN

Question 7

Why can a high-confidence association rule be misleading when considered alone?

A. Confidence cannot be calculated from transaction data

B. The consequent may be very common, making the rule appear strong

C. High confidence proves causation

D. Confidence is always lower than support

Correct Answer: B. The consequent may be very common, making the rule appear strong

Question 8

Which statement about PCA component signs is correct?

A. A component is invalid if all its loadings are negative

B. The sign can be reversed without changing the represented structure

C. Signs indicate statistical significance automatically

D. Signs determine the number of observations

Correct Answer: B. The sign can be reversed without changing the represented structure

Question 9

Why should anomaly-detection thresholds consider operational costs?

A. Thresholds determine the units of every feature

B. False positives and false negatives have different consequences

C. Thresholds eliminate concept drift

D. Costs are required to compute Euclidean distance

Correct Answer: B. False positives and false negatives have different consequences

Question 10

Which practice best supports reliable cluster interpretation?

A. Using only a visually attractive two-dimensional plot

B. Selecting the solution with the largest number of clusters

C. Combining stability, separation, original-unit profiling, and domain usefulness

D. Treating every cluster as a permanent customer identity

Correct Answer: C. Combining stability, separation, original-unit profiling, and domain usefulness