Unsupervised Learning and Dimensionality Reduction

Introduction and Context

Supervised learning uses labelled examples, but many important datasets contain no target variable. Customer transactions may not be labelled by customer type; sensor records may not identify faults; and images may lack semantic categories. Unsupervised learning addresses this situation by discovering structure, regularities, or unusual observations from input data alone.

Two major unsupervised-learning tasks are clustering and dimensionality reduction. Clustering groups observations that are similar according to a chosen distance or similarity measure. Dimensionality reduction represents data with fewer variables while attempting to preserve important information. These techniques support customer segmentation, document organisation, image analysis, exploratory data analysis, visualization, compression, and anomaly detection.

Unsupervised results are not automatically meaningful. A clustering algorithm always produces some partition if instructed to do so, but that partition may be statistically unstable, operationally irrelevant, or caused by poor feature scaling. Effective practice therefore combines algorithmic output with domain knowledge, validation, visualization, and careful interpretation.

Learning Objectives

By the end of this chapter, a student should be able to:

• explain the purpose and assumptions of unsupervised learning;
• apply k-means, hierarchical, and density-based clustering;
• select distance measures and preprocess features appropriately;
• evaluate cluster quality using internal and practical criteria;
• explain and apply principal component analysis (PCA);
• use dimensionality reduction for visualization and preprocessing;
• distinguish clustering from anomaly detection;
• assess limitations, ethical risks, and deployment considerations.

1. Foundations of Unsupervised Learning

Let a dataset be represented by a matrix \(X\) with \(n\) observations and \(p\) features. In supervised learning, a response variable \(y\) is available. In unsupervised learning, the algorithm receives \(X\) but no explicitly supplied labels. It seeks patterns such as groups, low-dimensional structure, or observations that differ substantially from the rest.

Important terminology includes:

• Observation or instance: one data record, such as a customer or image.
• Feature: a measured variable describing an observation.
• Cluster: a set of observations considered similar under a specified criterion.
• Centroid: the arithmetic mean of observations assigned to a cluster.
• Partition: an allocation of observations to clusters.
• Embedding: a representation of data in another space, often of lower dimension.
• Anomaly: an observation that is unusual relative to a reference population.

Similarity is not an intrinsic property of a dataset; it depends on feature selection, measurement scales, preprocessing, and the distance function. For two vectors \(x=(x_1,\ldots,x_p)\) and \(z=(z_1,\ldots,z_p)\), Euclidean distance is

\[

d(x,z)=\sqrt{\sum_{j=1}^{p}(x_j-z_j)^2}.

\]

This distance is appropriate for numerical features when differences across dimensions are meaningfully comparable. Squared Euclidean distance is often used computationally because it gives the same nearest-neighbour ordering as Euclidean distance. Manhattan distance, \(\sum_j |x_j-z_j|\), may be more robust to individual large coordinate differences. Categorical variables require suitable measures, such as matching-based distances or carefully encoded representations.

Feature scaling is usually essential. Standardization transforms feature \(x_j\) to

\[

z_j=\frac{x_j-\bar{x}_j}{s_j},

\]

where \(\bar{x}_j\) is the sample mean and \(s_j\) is the sample standard deviation. This produces a feature with sample mean approximately zero and sample standard deviation one, assuming \(s_j>0\). Without scaling, a variable measured in large numerical units can dominate distance calculations. Standardization is not always desirable: if the original units have a justified importance, or if robust scaling is required because of extreme outliers, another transformation may be preferable.

2. k-Means Clustering

2.1 Principle

k-means partitions data into a preselected number \(k\) of clusters. Each observation is assigned to the nearest centroid, and each centroid is updated as the mean of its assigned observations. The standard objective is to minimize the within-cluster sum of squares (WCSS):

\[

J=\sum_{r=1}^{k}\sum_{x_i\in C_r}\|x_i-\mu_r\|_2^2,

\]

where \(C_r\) is cluster \(r\), \(\mu_r\) is its centroid, and \(\|\cdot\|_2\) denotes Euclidean norm. The objective measures compactness, not necessarily real-world usefulness.

2.2 Algorithm

A standard procedure is:

1. Choose \(k\) initial centroids, commonly through random initialization or k-means++.

2. Assign each observation to the closest centroid.

3. Recalculate every centroid as the coordinate-wise mean of its assigned observations.

4. Repeat assignment and update steps until assignments stop changing, centroid movement is small, or a maximum iteration count is reached.

Each iteration does not increase the WCSS, but the algorithm generally reaches a local rather than global optimum. Multiple initializations are therefore recommended. k-means++ selects initial centres in a way that tends to spread them across the data and often improves reliability.

2.3 Worked Example

Consider one-dimensional observations \(1,2,3,10,11,12\), with \(k=2\). Suppose initial centroids are 2 and 11. The first assignment is \(C_1=\{1,2,3\}\) and \(C_2=\{10,11,12\}\). Updated centroids are

\[

\mu_1=(1+2+3)/3=2, \qquad \mu_2=(10+11+12)/3=11.

\]

Assignments do not change, so the algorithm has converged. The WCSS is

\[

[(1-2)^2+(2-2)^2+(3-2)^2]+[(10-11)^2+(11-11)^2+(12-11)^2]=4.

\]

The result is intuitively sensible because the observations form two compact groups. In higher dimensions, the same process operates on vectors. Notice that a low WCSS alone does not prove that two clusters are the correct interpretation; increasing \(k\) cannot increase the optimized WCSS and often reduces it.

2.4 Strengths and Limitations

k-means is computationally efficient, easy to explain, and effective for roughly spherical, similarly sized clusters. Its limitations include:

• the need to select \(k\) in advance;
• sensitivity to scaling and initialization;
• sensitivity to outliers because means are easily displaced;
• difficulty with elongated, nested, or differently sized clusters;
• dependence on the chosen distance geometry;
• forced assignment of every observation to a cluster.

The last point is especially important in anomaly detection: an extreme observation may simply be assigned to the nearest ordinary cluster rather than identified as anomalous.

3. Hierarchical Clustering

Hierarchical clustering creates a tree-like structure called a dendrogram, rather than requiring a single partition at the outset. In agglomerative hierarchical clustering, each observation begins as its own cluster. The algorithm repeatedly merges the two closest clusters until only one remains. Cutting the dendrogram at a selected height produces a desired number of groups.

The distance between clusters depends on the linkage rule:

• Single linkage: minimum pairwise distance between two clusters. It can detect chain-like structures but is vulnerable to chaining.
• Complete linkage: maximum pairwise distance. It tends to produce compact clusters but can be influenced by distant points.
• Average linkage: average pairwise distance, providing a compromise.
• Ward linkage: merges the pair producing the smallest increase in total within-cluster squared error. It is generally associated with Euclidean numerical data.

A dendrogram height represents the dissimilarity at which a merge occurs, subject to the linkage definition. It should not automatically be interpreted as a probability or a causal threshold. Hierarchical methods are useful when analysts want to inspect clustering at several granularities. However, early merges cannot normally be undone, and large datasets may make the method computationally expensive.

4. Density-Based Clustering

Density-based methods define clusters as regions of relatively high observation density separated by low-density regions. DBSCAN is a widely used example. It uses two parameters:

• \(\varepsilon\) (epsilon): the neighbourhood radius;
• MinPts: the minimum number of observations required for a dense neighbourhood.

An observation is a core point if its epsilon neighbourhood contains at least MinPts observations, typically counting the point itself. A point that is not core but lies within the neighbourhood of a core point is a border point. Other observations are labelled noise. DBSCAN expands clusters through connected core points and their reachable border points.

DBSCAN can discover non-spherical clusters and explicitly identify noise, making it useful for spatial data and some anomaly-detection tasks. It does not require the number of clusters in advance. Its weaknesses include sensitivity to \(\varepsilon\) and MinPts, difficulty when clusters have substantially different densities, and reduced effectiveness in high-dimensional spaces where distances become less discriminative.

A practical approach is to inspect a k-nearest-neighbour distance plot. A sharp change in the sorted distances can suggest a candidate epsilon, but the choice must be tested for stability and domain plausibility. HDBSCAN extends density-based ideas by constructing a hierarchy of density-connected clusters and can be more flexible when density varies.

5. Choosing and Evaluating Clusters

5.1 Selecting the Number of Clusters

The elbow method plots optimized WCSS against \(k\). The analyst looks for a point after which additional clusters yield relatively small improvements. It is a heuristic, not a theorem, and the elbow may be ambiguous.

The silhouette coefficient evaluates whether an observation is closer to its own cluster than to other clusters. For observation \(i\), let \(a(i)\) be its mean distance to observations in its own cluster and \(b(i)\) the smallest mean distance to observations in another cluster. Then

\[

s(i)=\frac{b(i)-a(i)}{\max\{a(i),b(i)\}}.

\]

For observations in non-singleton clusters, the value lies between -1 and 1. Values near 1 indicate good separation and cohesion; values near 0 indicate overlap; negative values suggest possible misassignment. The mean silhouette is useful for comparison under the same distance and preprocessing choices, but it can favour particular cluster shapes and should not replace domain validation.

The Calinski–Harabasz index compares between-cluster dispersion with within-cluster dispersion, generally favouring higher values. The Davies–Bouldin index measures average similarity between each cluster and its most similar competitor, generally favouring lower values. Exact numerical values are meaningful mainly when comparing methods on the same dataset and preprocessing pipeline.

5.2 Stability and External Validation

A credible clustering should be reasonably stable under resampling, modest parameter changes, or alternative initializations. Bootstrap resampling, repeated runs, and comparison of assignments can reveal instability. If labels become available later, external measures such as adjusted Rand index or normalized mutual information may be used, while remembering that business labels may be incomplete or themselves constructed from similar variables.

Interpretability is also an evaluation criterion. A cluster with strong statistical separation but no actionable distinction may have limited value. Analysts should profile clusters using variables not used to construct them, examine sample sizes, and test whether decisions based on the clusters improve outcomes.

6. Principal Component Analysis

6.1 Purpose and Intuition

PCA transforms correlated numerical features into a new set of orthogonal directions called principal components. The first component captures the greatest possible variance among all unit-length directions; the second captures the greatest remaining variance subject to being orthogonal to the first, and so on.

PCA can reduce noise and computational cost, address multicollinearity, and support two- or three-dimensional visualization. It is not a general-purpose feature-selection method: each component is usually a combination of many original variables, so interpretability can decrease.

6.2 Mathematical Formulation

After appropriate centering—and often standardization—let \(X_c\) be the centred \(n\times p\) data matrix. The sample covariance matrix is

\[

S=\frac{1}{n-1}X_c^T X_c.

\]

The first principal direction \(w_1\) solves

\[

\max_{\|w\|_2=1} w^T S w.

\]

The solution is an eigenvector of \(S\) associated with its largest eigenvalue \(\lambda_1\). Subsequent directions are eigenvectors associated with descending eigenvalues and are mutually orthogonal. The score of observation \(x_i\) on component \(j\) is obtained by projecting the centred observation onto \(w_j\):

\[

t_{ij}=(x_i-\bar{x})^T w_j.

\]

The proportion of total variance explained by component \(j\) is

\[

\frac{\lambda_j}{\sum_{\ell=1}^{p}\lambda_\ell}.

\]

These statements assume a covariance-based PCA and finite, appropriately measured variables. If features have very different units, PCA on the covariance matrix may be dominated by high-variance units; standardizing first makes PCA equivalent to applying PCA to the correlation matrix.

6.3 Small Worked Example

Suppose a standardized dataset has three PCA eigenvalues 2.1, 0.7, and 0.2. Their total is 3.0, equal to the number of standardized features. The explained-variance ratios are 70%, 23.33%, and 6.67%, respectively. Retaining the first two components preserves 93.33% of the variance according to this criterion. This does not mean that 93.33% of every predictive signal is retained: variance preservation and prediction preservation are different objectives.

PCA can be computed through eigen-decomposition of the covariance or correlation matrix, or through singular value decomposition (SVD), which is often numerically preferable. The signs of component vectors are arbitrary: multiplying a component by -1 leaves the represented subspace unchanged.

6.4 Interpretation and Limitations

Loadings describe how strongly original features contribute to a component. A component with large positive loadings for income and spending and a large negative loading for debt might represent a financial-contrast dimension, but interpretation requires domain knowledge. Loadings should not be interpreted as causal effects.

PCA is linear and sensitive to scaling and outliers. It may fail to represent curved manifolds, nonlinear relationships, or rare but important directions. Kernel PCA, autoencoders, and manifold-learning methods can model nonlinear structure, but they introduce additional assumptions, hyperparameters, and interpretability concerns.

7. Visualization and the Clustering Pipeline

A responsible workflow is:

1. Define the practical question and unit of analysis.

2. Inspect data quality, missingness, duplicates, outliers, and leakage.

3. Select features based on measurement validity and the intended use.

4. Encode categorical variables appropriately and scale numerical variables when justified.

5. Split time-dependent data or use a deployment-realistic validation design when necessary.

6. Fit multiple candidate methods and parameter settings.

7. Evaluate internal quality, stability, interpretability, and business or scientific usefulness.

8. Visualize results and profile clusters using independent variables.

9. Document preprocessing, random seeds, parameters, and limitations.

10. Monitor whether cluster composition and meanings change over time.

Scatterplots of two original features can reveal shape and overlap. PCA scores provide a compact linear visualization, but apparent separation in a two-dimensional projection may disappear in the full space. t-SNE and UMAP are useful exploratory visualization tools for nonlinear structure, yet distances and cluster sizes in their maps can be misleading. They should not be treated as definitive evidence of clusters or used without considering parameter sensitivity and random initialization.

A common architecture is to standardize data, apply PCA to reduce noise or computational burden, and then cluster the retained scores. This can work well, but PCA may remove low-variance directions that are important for a rare subgroup. The number of retained components should therefore be selected using the analytical objective, not only a variance threshold.

8. Applications: Segmentation and Anomaly Detection

8.1 Customer Segmentation

A retailer might cluster customers using recency, purchase frequency, average order value, and category diversity. The resulting groups could support differentiated communication or service design. However, clustering should not be used to infer sensitive traits from proxies or to impose unfair prices. Segment descriptions should be based on validated behavioural patterns, not stereotypes.

8.2 Anomaly Detection

Anomaly detection seeks observations that are rare, isolated, or inconsistent with a reference distribution. Density-based clustering can label low-density points as noise. Distance from a k-means centroid can provide a rudimentary score, but it is affected by cluster size and shape. More specialized methods include isolation forests, one-class support vector machines, and local outlier factor.

An anomaly is not necessarily an error, fraud, or threat. It may be a genuine new pattern, a sensor failure, or a data-entry problem. Operational systems need thresholds, investigation procedures, feedback loops, and false-positive monitoring. In safety-critical settings, unsupervised scores should normally support—not silently replace—human or rule-based review.

9. Illustrative Case Study: Regional Retail Segmentation

Background

A fictional online retailer operates across several regions and wants to design service strategies. It has twelve months of transaction data for 50,000 customers, including order frequency, average basket value, return rate, discount usage, and days since last purchase.

Problem

The management team wants meaningful customer groups, but it does not want the algorithm to reproduce region, age, or other sensitive attributes. It also needs segments that remain useful for at least one quarter.

Analysis

The data team removes direct identifiers, checks duplicate customer records, and aggregates transactions at customer level. Since frequency and monetary variables are strongly right-skewed, they use a documented log transformation where appropriate, then standardize continuous features. They exclude region from the clustering features but retain it for fairness auditing and interpretation. Missing values are handled using a predefined procedure rather than ad hoc deletion.

They compare k-means for several values of \(k\), Ward hierarchical clustering, and HDBSCAN. Repeated k-means runs are used to assess initialization stability. Silhouette values alone favour a small number of broad groups, while business users prefer a more detailed but still interpretable segmentation. The team therefore considers stability, minimum segment size, campaign feasibility, and differences in outcomes on a holdout period.

Approach and Solution

A four-group k-means solution is selected provisionally because it is stable, easy to assign to new customers, and produces actionable profiles: recent frequent purchasers, high-value occasional purchasers, discount-sensitive customers, and low-engagement customers. The profiles are generated from both clustering features and independent descriptive variables. The team tests whether region membership, service quality, and campaign response differ substantially across groups. It avoids using protected attributes to determine treatment and requires approval for any differential offer.

Outcome

A pilot campaign is evaluated against a control group. The value of a segment is judged by incremental response and customer experience, not by cluster compactness alone. After three months, the team re-runs the stability analysis because a major pricing change altered customer behaviour. Segment assignments are monitored for drift, and customers can move between segments rather than receiving permanent labels.

Learning Points

The case illustrates that clustering is a decision-support process, not merely an algorithmic operation. Data preparation, parameter selection, validation, fairness auditing, experimentation, and monitoring are all necessary. Excluding sensitive variables does not guarantee fairness because other features may act as proxies.

10. Challenges, Ethics, and Governance

Unsupervised models can encode historical inequality even without labels. A segmentation may systematically place a vulnerable population into a group receiving poorer service. Privacy risks arise when detailed behavioural data are combined across sources. Re-identification may be possible even after direct identifiers are removed.

Governance should include purpose limitation, data minimization, access control, retention rules, documented feature provenance, and review of downstream decisions. Analysts should record why a distance measure, number of clusters, and threshold were selected. Individuals affected by automated profiling may require explanation, appeal, or human review depending on the jurisdiction and application. High-impact uses—such as credit, employment, healthcare, housing, or policing—require especially strong legal, fairness, validity, and accountability controls.

Technical risks include data drift, changing measurement processes, missing-not-at-random data, unstable clusters, and misuse of visualizations. A model card or similar record should describe intended use, data scope, preprocessing, evaluation, known failure modes, and monitoring plans.

11. Contemporary Developments and Best Practices

Modern unsupervised learning increasingly combines representation learning with clustering. Autoencoders learn compressed representations using neural networks; contrastive methods learn embeddings by encouraging related examples to be close; and foundation-model embeddings enable clustering of text, images, and multimodal records. These representations can be powerful, but their semantics, training-data biases, licensing conditions, and computational costs must be examined.

Best practices include:

• formulate a measurable use case before clustering;
• prevent leakage from future information or outcome variables;
• scale and transform features deliberately;
• compare algorithms with appropriate assumptions;
• use repeated initialization and stability analysis;
• inspect cluster sizes, boundaries, and outliers;
• validate with domain experts and downstream outcomes;
• avoid treating PCA plots or a single metric as proof;
• document choices and monitor drift;
• include privacy, fairness, and human-oversight controls.

Chapter Summary

Unsupervised learning discovers structure without supplied target labels. Clustering methods differ in their assumptions: k-means optimizes within-cluster squared error and is effective for compact, roughly spherical groups; hierarchical clustering provides a multilevel dendrogram; and density-based methods identify dense regions and can label noise. Scaling, distance selection, initialization, and parameter choices strongly affect results.

Cluster quality should be assessed using cohesion, separation, silhouette or related indices, stability, interpretability, and practical outcomes. PCA is a linear dimensionality-reduction method based on eigenvectors of a covariance or correlation matrix. It creates orthogonal components ordered by explained variance, but it may reduce interpretability and can miss nonlinear or low-variance structure. Visualization is valuable for exploration but can distort high-dimensional relationships.

Applications include segmentation, exploratory analysis, compression, visualization, and anomaly detection. Responsible use requires attention to privacy, fairness, drift, validation, documentation, and human oversight. The central principle is that an unsupervised output is a hypothesis about structure, not an unquestionable fact.

Key Terms

Unsupervised learning: Learning patterns from data without supplied target labels.

Clustering: Grouping observations according to a similarity or distance criterion.

Centroid: The arithmetic mean vector of observations assigned to a cluster.

WCSS: Sum of squared distances from observations to their assigned cluster centroids.

k-means: A centroid-based algorithm that minimizes WCSS for a chosen number of clusters.

k-means++: An initialization method that spreads initial centroids probabilistically.

Hierarchical clustering: Clustering that builds a nested hierarchy of groups.

Dendrogram: A tree diagram showing hierarchical merges or splits.

Linkage: The rule used to define distance between clusters.

Density-based clustering: Clustering based on dense regions separated by sparse regions.

DBSCAN: A density-based algorithm using epsilon and MinPts to identify clusters and noise.

Anomaly: An observation unusual relative to a reference population.

PCA: A linear transformation producing orthogonal directions ordered by variance.

Principal component: A direction in feature space onto which observations are projected.

Eigenvalue: In PCA, the variance associated with a principal direction.

Loading: The coefficient of an original feature in a principal component.

Explained variance ratio: The proportion of total variance represented by a component.

Silhouette coefficient: A score comparing within-cluster cohesion with nearest-cluster separation.

Embedding: A representation of data in a transformed feature space.

Data drift: A change in data distributions or relationships over time.

Review Questions

Short Answer Questions

1. Why is feature scaling often important before distance-based clustering?

2. State the main objective minimized by k-means.

3. What information does a dendrogram provide?

4. Distinguish a core point from a noise point in DBSCAN.

5. What does the explained-variance ratio of a principal component measure?

Long Answer Questions

1. Explain the k-means algorithm, including its objective, convergence behaviour, and major limitations.

2. Compare k-means, hierarchical clustering, and DBSCAN with respect to assumptions, parameters, cluster shapes, and noise handling.

3. Describe PCA mathematically and explain how eigenvalues, eigenvectors, scores, and loadings are related.

4. Discuss how internal metrics, stability, domain knowledge, and downstream outcomes should be combined to evaluate clusters.

5. Explain the ethical and governance risks of using unsupervised learning for customer segmentation or high-impact decisions.

Analytical / Application Questions

1. A dataset contains annual income measured in dollars and purchase frequency measured as a count. Explain how you would preprocess it before k-means and justify your choices.

2. DBSCAN labels many observations as noise, while k-means produces three apparently compact groups. Design an investigation to determine whether the result reflects poor DBSCAN parameters, varying density, genuine anomalies, or unsuitable features.

3. A PCA model on four standardized features has eigenvalues 2.4, 1.0, 0.4, and 0.2. Calculate the variance explained by the first two components and discuss whether retaining them is automatically appropriate for anomaly detection.

Multiple Choice Questions with Answers

Question 1

What quantity does standard k-means minimize?

A. Total between-cluster variance only

B. Within-cluster sum of squared distances

C. The number of observations marked as noise

D. The total number of principal components

Correct Answer: B. Within-cluster sum of squared distances

Question 2

Which clustering method explicitly identifies observations as noise using density?

A. DBSCAN

B. Ward hierarchical clustering

C. Standard PCA

D. Ordinary least squares

Correct Answer: A. DBSCAN

Question 3

What is the primary purpose of feature standardization before Euclidean clustering?

A. To create target labels

B. To ensure every feature has identical correlation

C. To prevent large measurement scales from dominating distance

D. To guarantee globally optimal k-means results

Correct Answer: C. To prevent large measurement scales from dominating distance

Question 4

In agglomerative hierarchical clustering, what happens initially?

A. All observations form one cluster

B. Each observation begins as its own cluster

C. Only outliers form clusters

D. PCA components are calculated first

Correct Answer: B. Each observation begins as its own cluster

Question 5

A silhouette coefficient close to 1 generally indicates that an observation is:

A. Closer to another cluster than to its own

B. Equally distant from all clusters

C. Well matched to its own cluster and separated from others

D. Definitely an outlier

Correct Answer: C. Well matched to its own cluster and separated from others

Question 6

In PCA, the first principal component is the unit direction that:

A. Minimizes total variance

B. Maximizes projected variance

C. Maximizes the number of clusters

D. Guarantees causal interpretation

Correct Answer: B. Maximizes projected variance

Question 7

If standardized PCA has eigenvalues 2.1, 0.7, and 0.2, what percentage of variance is explained by the first component?

A. 21%

B. 30%

C. 70%

D. 93.3%

Correct Answer: C. 70%

Question 8

Which linkage is particularly associated with minimizing increases in within-cluster squared error during hierarchical merging?

A. Single linkage

B. Complete linkage

C. Average linkage

D. Ward linkage

Correct Answer: D. Ward linkage

Question 9

Why can a low WCSS value be misleading when selecting k?

A. WCSS always increases as k increases

B. WCSS generally decreases as more clusters are allowed

C. WCSS is defined only for categorical data

D. WCSS identifies causal relationships

Correct Answer: B. WCSS generally decreases as more clusters are allowed

Question 10

Which statement about PCA is correct?

A. Its component signs have an intrinsic meaning that cannot change

B. It always preserves predictive performance

C. It creates orthogonal directions ordered by captured variance

D. It requires labelled target values

Correct Answer: C. It creates orthogonal directions ordered by captured variance