Unsupervised Learning and Segmentation

Introduction

Unsupervised learning discovers structure in data when no target variable or labelled outcome is available. In data warehousing and data mining, this is especially valuable because enterprise repositories contain large volumes of behavioural, transactional, textual, and operational data that have not been classified in advance. Clustering can reveal customer segments, groups of products with similar demand, anomalous operating conditions, or regions with comparable risk profiles. Representation techniques such as principal component analysis (PCA) reduce dimensionality so that these structures can be analysed, visualised, and communicated.

Unsupervised analysis is exploratory rather than automatically causal. A cluster is a pattern induced by a dataset, a distance measure, and an algorithm; it is not necessarily a naturally existing group. Consequently, a credible segmentation requires both quantitative validation and substantive interpretation.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Explain the purpose, assumptions, and workflow of unsupervised learning.
2. Apply partitioning methods such as k-means and k-medoids.
3. Compare hierarchical and density-based clustering.
4. Select appropriate distance measures and preprocessing methods.
5. Evaluate cluster quality using internal and external validity measures.
6. Use PCA and related representation techniques for segmentation and visualisation.
7. Interpret clusters responsibly in a data-warehouse and business setting.

1. Unsupervised Learning in the Data-Warehouse Context

A typical warehouse integrates dimensions such as customer, product, geography, time, channel, and promotion with measurable facts such as revenue, quantity, margin, and service incidents. A segmentation dataset is usually created by aggregating facts at an appropriate grain. For example, one row may represent one customer and columns may include annual revenue, order frequency, average basket value, return rate, and recency.

The analytical workflow is:

1. Define the unit of analysis and business purpose.
2. Extract and aggregate warehouse facts.
3. Clean missing, erroneous, and extreme values.
4. Transform and scale variables.
5. Select a representation and similarity measure.
6. Fit one or more clustering algorithms.
7. Validate stability, separation, and usefulness.
8. Profile and name clusters using independent variables.
9. Deploy results with monitoring and governance.

A crucial distinction is between descriptive segmentation and predictive classification. Clustering discovers groups without labels; classification assigns new observations to previously defined labelled classes. A discovered segment should not be treated as a guaranteed future response group unless its behavioural relevance has been tested.

2. Similarity, Distance, and Preprocessing

Clustering depends fundamentally on how similarity is defined. For numeric vectors x and y with p variables, Euclidean distance is:

Here xᵢ and yᵢ are measurements on variable i. Euclidean distance is appropriate when variables are numeric, comparably scaled, and differences in each dimension have meaningful magnitude. Squared Euclidean distance is often used computationally because it preserves the ordering of distances while avoiding square roots.

Manhattan distance is:

It can be more robust than Euclidean distance when coordinate-wise deviations are additive or when moderate outliers should have less influence. For binary attributes, Jaccard similarity is useful when joint presence matters more than joint absence:

where a is the number of attributes present in both records, b is present only in A, and c is present only in B. The corresponding Jaccard distance is 1 − J(A,B).

Scaling and transformation

Suppose income is measured in thousands and order count is measured from 0 to 10. Without scaling, income may dominate distance. Standardisation uses:

where μᵢ is the sample mean and σᵢ is the sample standard deviation of variable i. This creates variables with mean approximately 0 and standard deviation approximately 1. It assumes that mean and standard deviation are representative and can be distorted by extreme observations. Robust scaling may instead use the median and interquartile range:

Log transformation, such as x' = log(1 + x), is useful for non-negative, strongly right-skewed measures such as revenue. Categorical variables require an appropriate representation, such as one-hot encoding or a mixed-data distance. One-hot encoding with ordinary Euclidean distance can unintentionally assign arbitrary geometry; therefore, Gower distance is often preferable for mixed numeric, categorical, and binary data.

Missing values should not be silently replaced with zero. Imputation, missingness indicators, or a deliberate exclusion rule may be appropriate depending on the data-generating process. Duplicate records, leakage from future periods, and post-outcome variables can produce misleading segments.

3. Partitioning Methods

3.1 k-means clustering

The k-means algorithm partitions n observations into k clusters by minimising within-cluster squared Euclidean error:

Cⱼ is cluster j, μⱼ is its arithmetic mean vector, and k is selected by the analyst. The algorithm alternates between two steps:

1. Assign each observation to the nearest centroid.
2. Recalculate each centroid as the mean of observations assigned to it.

It stops when assignments, centroids, or the objective function change negligibly. The method is computationally efficient and works well for compact, approximately spherical clusters. It is sensitive to scale, outliers, initialisation, and the chosen value of k. Multiple random starts or k-means++ initialisation reduce, but do not eliminate, local-minimum problems.

Worked example: one-dimensional k-means

Consider standardised one-dimensional observations 1, 2, 3, 8, 9 and 10, with k = 2. If the assignments are C₁ = {1,2,3} and C₂ = {8,9,10}, the centroids are:

The objective is (1−2)² + (2−2)² + (3−2)² + (8−9)² + (9−9)² + (10−9)² = 4. An alternative grouping, C₁ = {1,2,8} and C₂ = {3,9,10}, has centroids 11÷3 and 22÷3 and a much larger within-cluster error. The example illustrates why centroids represent averages and why separation relative to within-cluster spread matters.

3.2 k-medoids

k-medoids resembles k-means but chooses an actual observation—the medoid—as the representative of each cluster. A common objective is the sum of distances from observations to their medoid:

where mⱼ is an observed point in Cⱼ. Because the representative is a real record and arbitrary distance measures can be used, k-medoids is often more interpretable and more resistant to outliers than k-means. It is generally more computationally expensive.

4. Hierarchical Clustering

Hierarchical clustering constructs a nested sequence of groups, represented by a dendrogram. In agglomerative clustering, every observation begins as a singleton and the two closest clusters are repeatedly merged. Divisive clustering begins with one cluster and recursively splits it, but is less commonly used.

The distance between clusters depends on the linkage rule:

A dendrogram does not itself determine the correct number of clusters. Cutting it at a selected height creates a partition, which must be justified by validation and domain knowledge. Hierarchical methods are valuable when analysts need multiple resolutions—for example, broad customer families and more detailed subsegments.

5. Density-Based Clustering

Density-based methods define clusters as dense regions separated by sparse regions. DBSCAN uses two parameters: ε, the neighbourhood radius, and MinPts, the minimum number of points required for a dense neighbourhood. A point is a core point if its ε-neighbourhood contains at least MinPts points, usually counting the point itself. Points reachable from core points form clusters; isolated observations may be labelled noise.

DBSCAN can discover non-spherical shapes and explicitly identify noise, unlike k-means. However, it struggles when clusters have very different densities or when distance becomes less meaningful in high dimensions. Parameter selection is domain-sensitive. A k-distance plot can help identify a plausible ε, but it is not a substitute for substantive validation.

HDBSCAN extends density-based reasoning by constructing a hierarchy of density-connected groups and selecting relatively stable clusters. It can be more effective than DBSCAN when density varies, although its output still depends on representation and parameter choices.

6. Choosing the Number of Clusters and Validating Results

6.1 Internal validity

Internal measures use only the data and the assigned labels. The silhouette value for observation i is:

where a(i) is the average distance from i to points in its own cluster and b(i) is the smallest average distance from i to any other cluster. Values approach 1 when the point is well assigned, are near 0 at boundaries, and may be negative when another cluster appears closer. The mean silhouette is useful for comparison, not as an absolute proof of correctness.

The Calinski–Harabasz index compares between-cluster dispersion with within-cluster dispersion. Higher values generally indicate compact and well-separated groups, but values are not comparable without care across substantially different datasets or distance definitions. The Davies–Bouldin index averages the similarity of each cluster to its most similar competitor; lower values are preferred.

The elbow method plots within-cluster sum of squares against k and looks for diminishing returns. It is intuitive but subjective: many real datasets have no clear elbow. A practical selection combines several values of k, stability under resampling, interpretability, operational actionability, and cost of maintaining segments.

6.2 External and stability validity

If trusted labels exist, external measures such as adjusted Rand index or normalized mutual information can compare discovered clusters with those labels. These measures do not prove that the labels are theoretically correct; they quantify agreement.

Stability asks whether similar clusters arise under bootstrap samples, different initialisations, time periods, or reasonable preprocessing choices. A segmentation that disappears when a small set of observations is removed should not drive major decisions. Analysts should report uncertainty and avoid presenting cluster boundaries as exact natural laws.

6.3 Profiling and validation with independent variables

Variables used to create clusters should be separated from variables used to profile them when possible. For example, customer value may define the clustering, while later purchase response, complaint rate, or retention can assess usefulness. Comparing outcomes across clusters requires appropriate statistical tests and effect sizes, not merely a table of means. Multiple comparisons, confounding, and selection bias must be considered.

7. Principal Component Analysis and Representation

High-dimensional data are difficult to visualise and may contain correlated variables. PCA creates orthogonal linear combinations called principal components. Given a centred data matrix X, the covariance matrix is:

where n is the number of observations and X contains centred variables. PCA finds eigenvectors vⱼ and eigenvalues λⱼ satisfying:

The first component has the largest variance among all unit-length linear combinations; each subsequent component maximises remaining variance subject to orthogonality. The score of observation x on component j is:

The proportion of variance explained by the first r components is:

PCA is sensitive to scaling, outliers, and linear relationships. If variables have different units, PCA is usually performed on the correlation matrix, equivalent to standardising first. A high explained-variance percentage does not guarantee that a low-dimensional plot preserves business-relevant separation.

PCA can support clustering in two ways: visualising existing clusters in two or three dimensions, or reducing noise before clustering. The second use must be justified carefully because discarded components may contain small but meaningful segmentation signals. PCA components are not automatically causal factors; loadings describe statistical associations.

Other representation methods include:

• Factor analysis, which models observed variables as indicators of latent factors plus error and is often used for constructs such as attitudes.
• Multidimensional scaling, which seeks coordinates preserving pairwise dissimilarities.
• Non-negative matrix factorisation, which represents non-negative data as additive, parts-based components.
• t-SNE, useful for local-neighbourhood visualisation but sensitive to settings and unsuitable for interpreting global distances or cluster sizes.
• UMAP, which often preserves local structure while offering useful global visualisations, though its geometry remains parameter-dependent.
• Autoencoders, neural networks trained to reconstruct inputs through a lower-dimensional bottleneck; they can learn nonlinear representations but require careful regularisation and monitoring.

A visual island in t-SNE or UMAP is not by itself evidence of a robust cluster. Clustering should be performed and validated in a representation whose distances have a defensible interpretation.

8. Illustrative Case: Retail Customer Segmentation

Background

A fictional omnichannel retailer stores order, customer, product, promotion, and service data in a warehouse. Management wants to improve campaign relevance without creating unfair or intrusive targeting practices.

Problem

The marketing team proposes four segments based only on total annual sales. This produces large-value groups but fails to distinguish frequent low-value customers, promotion-dependent customers, and customers with high returns.

Analysis

The data team creates one row per active customer for the previous twelve months. Candidate variables include recency, order frequency, average order value, category diversity, discount share, return rate, and contribution margin. Monetary variables are log-transformed; variables are robust-scaled; customers with fewer than two completed orders are analysed separately because their behavioural measures are unreliable. No future campaign response is included in the clustering data.

The team compares k-means, k-medoids using a mixed-data distance, hierarchical Ward clustering, and HDBSCAN. k-means with k = 5 has a strong mean silhouette and stable assignments across several initialisations. However, a four-cluster solution is more stable across monthly samples and easier for managers to operationalise. PCA is used to visualise the solution, while clustering is performed on the full validated feature set rather than only the first two components.

Approach and solution

The four clusters are profiled using variables not used to form them, including next-quarter retention, customer-service contacts, and response to a controlled pilot campaign. The resulting profiles are interpreted as: high-margin loyal customers, frequent promotion-sensitive customers, occasional high-basket customers, and high-return at-risk customers. These names describe observed patterns rather than immutable customer identities.

A randomised campaign test is then conducted within each segment. The treatment group receives a tailored offer and the control group receives the standard communication. Incremental response, margin after discount, and complaint rates are measured. Segment membership is recomputed periodically, with versioned definitions and an audit trail.

Outcome

The retailer finds that tailoring offers increases response in the promotion-sensitive group but reduces margin if the discount is too large. The high-margin group responds better to early access than to price reductions. The high-return group requires service and product-quality investigation rather than aggressive promotion. The company therefore uses segments as decision-support inputs, not as automatic triggers.

Learning points

The case demonstrates that segmentation is a pipeline, not a single algorithm. Grain definition, leakage control, scaling, algorithm comparison, stability, experimental validation, and ethical governance are all necessary. It also shows why a statistically coherent segment may have little value unless an organisation can act on it and measure consequences.

9. Implementation, Governance, and Professional Practice

A production segmentation system should record the data snapshot, feature definitions, transformations, distance measure, algorithm, random seed, parameters, validation results, and cluster labels. Warehouse pipelines should distinguish slowly changing customer attributes from time-varying behavioural facts. Segment drift can be monitored through feature distributions, cluster proportions, centroid movement, and assignment stability.

Ethical risks include discriminatory targeting, use of sensitive proxies, opaque profiling, exclusion from essential services, and inappropriate inference of traits. Privacy principles require purpose limitation, data minimisation, access control, retention limits, and appropriate consent or legal basis. Analysts should inspect whether protected groups are disproportionately assigned to high-risk or low-benefit segments. Where segments influence credit, employment, insurance, healthcare, or other regulated decisions, applicable law may require explanation, impact assessment, human oversight, and contestability.

Best practice is to:

• define a decision and success measure before clustering;
• use time-appropriate data and prevent leakage;
• test several algorithms and representations;
• report sensitivity to scaling, distance, and parameters;
• retain noise and uncertainty rather than forcing every point into a group;
• validate actionability with controlled experiments;
• document limitations and monitor drift;
• provide human review for consequential decisions.

Challenges and Limitations

The curse of dimensionality makes distances increasingly similar as dimensions increase, weakening nearest-neighbour reasoning. Feature selection, domain-informed aggregation, and dimensionality reduction can help, but each changes the problem. Outliers may represent errors, rare valuable customers, or emerging risks; automatic removal can destroy insight. Nonstationary markets can make a previously useful segmentation obsolete. Correlation can cause a construct to be counted multiple times, while imbalanced cluster sizes can distort validation metrics.

Most importantly, clusters are not explanations. They describe similarity under selected variables and assumptions. Causal claims require causal designs, experiments, or appropriate observational methods. A responsible analyst communicates both what the segmentation reveals and what it cannot establish.

Chapter Summary

Unsupervised learning discovers structure without a predefined target. In data warehousing, effective segmentation begins with a defensible unit of analysis, clean time-bounded features, suitable transformations, and an explicitly chosen similarity measure. k-means is efficient for compact numeric clusters; k-medoids is more robust and interpretable; hierarchical methods provide nested solutions; and density-based methods identify irregular groups and noise.

Cluster quality should be assessed through internal validity, external agreement where labels exist, stability, interpretability, and operational outcomes. PCA decomposes covariance into orthogonal components and supports dimensionality reduction and visualisation, but explained variance and visual separation do not automatically establish meaningful clusters. Nonlinear representations require particularly cautious interpretation.

A professional segmentation process combines algorithmic analysis with domain expertise, experimental validation, documentation, privacy protection, fairness assessment, and drift monitoring. Clusters are analytical representations—not fixed identities or causal explanations.

Key Terms

Unsupervised learning: Learning patterns from data without a predefined target label.

Segmentation: Dividing observations into meaningful groups based on similarity.

Feature: A variable used to represent an observation.

Distance measure: A numerical rule for quantifying dissimilarity between observations.

k-means: A partitioning algorithm that minimises within-cluster squared distance to centroids.

Centroid: The arithmetic mean vector representing a k-means cluster.

k-medoids: A partitioning method whose representatives are actual observations.

Hierarchical clustering: Clustering that builds a nested hierarchy of groups.

Dendrogram: A tree-like representation of hierarchical merges or splits.

Linkage: A rule defining distance between clusters.

Density-based clustering: Clustering based on dense regions separated by sparse regions.

DBSCAN: A density-based algorithm that uses neighbourhood radius and minimum points.

Silhouette coefficient: A measure comparing within-cluster and nearest-other-cluster distances.

Cluster stability: The degree to which clustering results persist under perturbations.

Principal component: An orthogonal linear combination explaining variance in PCA.

Eigenvalue: The variance associated with a principal component.

Loading: The coefficient linking an original variable to a component.

Dimensionality reduction: Representing data with fewer variables or coordinates.

Concept drift: Change in the underlying data pattern over time.

Data leakage: Use of information unavailable at the time a real decision would be made.

Review Questions

Short Answer Questions

1. Why must variables usually be scaled before applying k-means?
2. Distinguish a centroid from a medoid.
3. What does a negative silhouette value suggest?
4. State one advantage and one limitation of DBSCAN.
5. Why is PCA not a causal modelling technique?

Long Answer Questions

1. Explain the complete workflow for developing a warehouse-based customer segmentation, including data preparation, modelling, validation, and deployment.
2. Compare k-means, k-medoids, hierarchical clustering, and DBSCAN in terms of assumptions, strengths, and limitations.
3. Derive and interpret the silhouette coefficient, explaining how it can support selection of a clustering solution.
4. Explain PCA using the covariance matrix, eigenvectors, eigenvalues, scores, and explained variance.
5. Discuss ethical, privacy, fairness, and governance issues in using unsupervised segments for business decisions.

Analytical / Application Questions

1. A retailer has revenue, order count, discount share, and return rate with highly skewed distributions and different units. Design a preprocessing and clustering strategy, defending each choice.
2. A five-cluster solution has a higher silhouette score than a four-cluster solution, but its assignments are unstable across months and managers cannot act on one cluster. Which solution should be selected, and what additional evidence would you collect?
3. A UMAP plot shows three visually separate islands, but DBSCAN identifies one cluster and noise while k-means identifies four groups. Analyse possible reasons for the disagreement and propose a validation plan.

Multiple Choice Questions with Answers

Question 1

Which objective does standard k-means minimise?

A. The sum of within-cluster squared distances to centroids

B. The maximum distance between any two observations

C. The number of principal components

D. The sum of distances to the global mean only

Correct Answer: A. The sum of within-cluster squared distances to centroids

Question 2

In k-medoids, a cluster representative is:

A. Always the arithmetic mean

B. An actual observation in the cluster

C. The observation farthest from all others

D. A randomly generated vector

Correct Answer: B. An actual observation in the cluster

Question 3

A silhouette value close to 1 generally indicates that an observation:

A. Is equally close to every cluster

B. Is closer to another cluster than its own

C. Is well matched to its own cluster and separated from others

D. Is necessarily an outlier

Correct Answer: C. Is well matched to its own cluster and separated from others

Question 4

Which clustering method is specifically designed to identify noise points and non-spherical dense regions?

A. DBSCAN

B. k-means

C. Ward clustering

D. PCA

Correct Answer: A. DBSCAN

Question 5

In PCA, the eigenvalue associated with a component represents:

A. The number of clusters

B. The variance explained by that component

C. The sample median

D. The distance between two records

Correct Answer: B. The variance explained by that component

Question 6

Why is standardisation commonly applied before PCA when variables have different units?

A. To make all observations identical

B. To prevent variables with large measurement scales from dominating variance

C. To eliminate all correlation

D. To guarantee causal components

Correct Answer: B. To prevent variables with large measurement scales from dominating variance

Question 7

For Jaccard similarity of binary attributes, joint absence is generally:

A. Counted twice

B. Ignored in the numerator and denominator

C. Treated as a principal component

D. Converted into a centroid

Correct Answer: B. Ignored in the numerator and denominator

Question 8

What is the main purpose of cluster-stability analysis?

A. To guarantee that clusters are causal

B. To test whether similar assignments persist under reasonable perturbations

C. To increase the number of variables automatically

D. To replace domain interpretation

Correct Answer: B. To test whether similar assignments persist under reasonable perturbations

Question 9

Why can a high explained-variance percentage in PCA fail to guarantee useful segmentation?

A. Variance is unrelated to every possible business objective

B. PCA always uses categorical variables incorrectly

C. It proves that clusters are natural kinds

D. It prevents visualisation

Correct Answer: A. Variance is unrelated to every possible business objective

Question 10

Including future campaign response as an input feature when constructing pre-campaign customer segments creates:

A. Robust scaling

B. Data leakage

C. A medoid

D. Density separation

Correct Answer: B. Data leakage