Clustering and Unsupervised Learning

Introduction and Context

Supervised learning uses labelled examples: a target variable is known and a model learns to predict it. In many data-science situations, however, useful labels do not exist. A retailer may know what customers purchased but not their customer types; a hospital may record measurements without diagnostic categories; and a manufacturing plant may collect sensor readings without predefined fault classes. Unsupervised learning addresses this problem by identifying structure within data without a supplied target variable.

Clustering is the principal unsupervised-learning task. It groups observations so that members of the same group are relatively similar and members of different groups are relatively dissimilar. The groups are not automatically meaningful: interpretation requires subject knowledge, validation and careful examination of the variables used.

This chapter examines k-means clustering, hierarchical clustering and density-based methods, with emphasis on similarity measures, preprocessing, evaluation and practical segmentation.

Learning Objectives

After studying this chapter, you should be able to:

• explain the purpose and assumptions of unsupervised learning;
• distinguish partitioning, hierarchical and density-based clustering;
• calculate and interpret common distance and similarity measures;
• apply k-means and describe its optimization process;
• interpret dendrograms and density-based clusters;
• select an appropriate number of clusters and evaluate cluster quality;
• apply clustering to customer, operational and scientific data;
• identify limitations, ethical risks and governance requirements.

1. Foundations of Unsupervised Learning

Let a dataset contain n observations and p measured features. It can be represented as a matrix X, where each row is an observation and each column is a feature. In clustering, the desired output is a partition or structure derived from X rather than a prediction of a known response variable.

A clustering result should normally satisfy two broad principles:

1. Within-cluster cohesion: observations assigned to one cluster should be similar.
2. Between-cluster separation: different clusters should be distinguishable according to the selected representation and similarity measure.

These principles are not absolute. A method may identify overlapping groups, nested groups or irregularly shaped groups. Consequently, clustering is not simply an objective discovery of natural categories. It is a modelling choice affected by variables, scaling, distance functions, algorithms and analyst interpretation.

Important terminology

• An observation is one item being clustered, such as a customer or experiment.
• A feature is a measured attribute of an observation.
• A cluster is a group formed by an algorithm.
• A partition assigns each observation to one of several groups.
• A prototype is a representative point, such as a k-means centroid or medoid.
• Hard clustering assigns an observation to one cluster.
• Soft or fuzzy clustering assigns degrees of membership to multiple clusters.
• A distance expresses dissimilarity; a similarity expresses closeness.
• A hyperparameter is selected by the analyst, such as k or a density radius.

2. Preparing Data for Clustering

Variable selection

Clustering reflects the variables supplied to it. Including irrelevant variables can hide meaningful structure, while excluding a crucial variable can produce misleading groups. Features should therefore be selected according to the business or scientific question. Identifiers, such as customer numbers, generally must not be used as analytical features.

Highly redundant variables can give one underlying concept disproportionate influence. Correlation analysis, domain review or dimensionality-reduction methods may help identify redundancy, although removing correlated variables should be justified rather than automatic.

Missing values and outliers

Most distance-based algorithms cannot use missing values directly. Options include imputation, use of a distance measure designed for incomplete data, or removal when missingness is limited and plausibly random. Imputation must be performed consistently and documented.

Outliers can strongly affect centroids and distances. Analysts should determine whether an extreme value is an error, a rare but valid case, or evidence of a separate group. Blind deletion may remove the most important cases.

Scaling

Suppose income is measured in thousands and age in years. Without scaling, the feature with the larger numerical range may dominate Euclidean distance. A common standardization is the z-score:

Here xᵢⱼ is the value of feature j for observation i, μⱼ is the feature mean and σⱼ is its standard deviation. The transformed feature is dimensionless, has mean approximately 0 and standard deviation approximately 1 when calculated from the same data. Standardization is especially common for k-means, but it can reduce the influence of meaningful scale differences. Robust scaling using the median and interquartile range may be preferable when distributions contain extreme values.

For a feature with known lower and upper bounds, min–max scaling is:

It maps observed values to approximately [0, 1], provided the denominator is non-zero. New observations outside the training range may fall outside this interval.

Categorical variables require special treatment. One-hot encoding can make nominal categories usable, but ordinary Euclidean distance may then be inappropriate. Mixed-type data may require a mixed distance, such as Gower distance, or a method designed for categorical variables.

3. Similarity and Distance Measures

The measure of resemblance is central to clustering. It should reflect the meaning and measurement scale of the variables.

Euclidean distance

For two p-dimensional observations x and y, Euclidean distance is:

All features should be numeric and comparably scaled for this measure to be sensible. Euclidean distance is sensitive to outliers and tends to favour compact, approximately spherical clusters.

Manhattan distance

Manhattan distance measures total coordinate-wise difference. It can be more robust than Euclidean distance in some high-dimensional or outlier-prone settings, although it remains dependent on scaling.

Cosine similarity

For non-zero vectors, cosine similarity is:

where x · y is the dot product and ||x|| is the vector length. It measures orientation rather than magnitude and is common for text-term vectors. Two documents with similar word proportions may have high cosine similarity even if one is much longer.

Jaccard similarity

For binary sets A and B, Jaccard similarity is:

It ignores shared absences and is useful when the presence of attributes matters more than their absence. For example, it may compare products containing features, rather than products lacking features.

Distance choice as a modelling decision

Distance selection should follow the data-generating context. Euclidean distance may suit standardized physical measurements; cosine similarity may suit documents; Jaccard similarity may suit sparse binary attributes. Analysts should test whether conclusions remain stable under reasonable alternative measures.

4. K-Means Clustering

Concept and objective

K-means partitions n observations into k clusters. Each cluster is represented by the arithmetic mean of its assigned observations, called a centroid. The algorithm seeks to minimize the within-cluster sum of squared Euclidean distances:

Here Cᵣ is cluster r, μᵣ is its centroid, xᵢ is an observation and ||·|| is Euclidean norm. The objective is non-negative and has units equal to the square of the feature units if raw measurements are used; after standardization it is dimensionless.

The objective is also called inertia or within-cluster sum of squares. Minimizing it does not prove that the clusters are meaningful. Increasing k cannot increase the minimum training objective, so a small objective alone is not sufficient for choosing k.

Lloyd's algorithm

The common iterative procedure is:

1. Select k initial centroids.
2. Assign each observation to the nearest centroid.
3. Recalculate each centroid as the mean of observations assigned to it.
4. Repeat steps 2 and 3 until assignments stop changing, centroid movement is small, or a maximum iteration limit is reached.

For fixed assignments, the arithmetic mean minimizes the sum of squared distances within a cluster. The assignment step and update step therefore do not increase the objective, although the method may converge to a local minimum rather than the global minimum. Multiple random starts, often called k-means++, can improve the chance of finding a good solution.

Worked example

Consider four standardized two-dimensional observations: A = (1, 1), B = (1, 2), C = (5, 5) and D = (6, 5), with k = 2. Suppose initial centroids are A and C.

Distances to A and C assign A and B to the first cluster, and C and D to the second. The updated centroids are:

The squared distances within cluster 1 are 0.25 and 0.25; within cluster 2 they are 0.25 and 0.25. Thus the objective is J = 1.00. The groups are visually and substantively plausible in this simple example, but a real analysis would still assess stability and domain meaning.

Strengths and weaknesses

K-means is fast, easy to explain and effective for large numeric datasets with compact groups. Its limitations include the need to specify k, sensitivity to scaling and outliers, dependence on initialization, and poor performance for elongated, nested or differently sized clusters. It also assigns every observation to a cluster, even when an observation is noise or lies between groups.

K-medoids is a related method that represents each cluster by an actual observation rather than a mean. It is generally more robust to outliers and can work with arbitrary dissimilarities, although it is often more computationally expensive.

5. Hierarchical Clustering

Hierarchical clustering creates a tree of relationships rather than requiring one final partition at the start. The result is displayed as a dendrogram. A horizontal cut through the dendrogram produces a chosen number of clusters.

Agglomerative procedure

The most common form is agglomerative clustering:

1. Begin with each observation as its own cluster.
2. Find the two closest clusters.
3. Merge them.
4. Recalculate distances between the new cluster and all remaining clusters.
5. Continue until one cluster remains.

The choice of linkage determines how cluster-to-cluster distance is calculated.

For Ward linkage, Euclidean geometry and compatible scaling are normally assumed. Single linkage can identify elongated structures but is vulnerable to bridging points. Complete linkage is more sensitive to extreme separation. The dendrogram records the sequence of merges, but the analyst must still decide where to cut it.

Hierarchical methods are useful when nested structure matters, when analysts want to inspect several possible cluster counts, or when the dataset is moderate in size. Their disadvantages include computational cost, sensitivity to distance and linkage choices, and the fact that early merges cannot normally be undone.

6. Density-Based Clustering

Density-based methods define clusters as regions of high observation density separated by regions of low density. They can identify irregular shapes and label isolated observations as noise.

DBSCAN

DBSCAN uses two principal parameters:

• ε (epsilon): the radius of a neighbourhood;
• MinPts: the minimum number of observations required for a dense neighbourhood.

An observation is a core point if its ε-neighbourhood contains at least MinPts observations, according to the chosen convention. Points reachable through connected core-point neighbourhoods form a cluster. Border points lie within the neighbourhood of a core point but may not themselves be core points. Remaining observations are labelled noise.

DBSCAN does not require the number of clusters in advance and can discover non-spherical shapes. It is valuable for geographic data, anomaly detection and spatial sensor measurements. However, selecting ε and MinPts can be difficult, and DBSCAN struggles when clusters have very different densities or when dimensions are high. Standardization and a meaningful distance measure remain important.

HDBSCAN extends the density-based idea by examining a hierarchy of density levels and can be more flexible when density varies. It is useful in practice, but its outputs still require parameter review and domain validation.

7. Choosing the Number and Quality of Clusters

No single metric establishes that a clustering solution is correct. Good practice combines quantitative diagnostics, visualization, stability testing and domain interpretation.

Elbow method

For k-means, analysts plot the minimized within-cluster sum of squares against k. The curve generally decreases as k increases. An “elbow” may indicate a point beyond which additional clusters provide diminishing improvement. The elbow can be ambiguous and should not be treated as an automatic answer.

Silhouette coefficient

For observation i, let a(i) be its average distance to observations in its own cluster. Let b(i) be the smallest average distance from i to observations in any other cluster. The silhouette coefficient is:

When a(i) and b(i) are positive distances, s(i) lies between −1 and 1. Values near 1 indicate good separation, values near 0 indicate overlap, and negative values suggest that the observation may be assigned to the wrong cluster. The mean silhouette is an internal evaluation measure; it does not measure business usefulness.

Calinski–Harabasz and Davies–Bouldin indices

The Calinski–Harabasz index compares between-cluster dispersion with within-cluster dispersion, adjusted for the number of observations and clusters. Higher values are generally preferred for comparable solutions. The Davies–Bouldin index averages the similarity of each cluster to its most similar alternative cluster; lower values are generally preferred. Definitions vary slightly across implementations, so software documentation should be checked.

Stability and external validation

A robust solution should not change dramatically when the data are resampled, when a reasonable initialization changes, or when small preprocessing decisions vary. Bootstrap or repeated-sample analyses can assess stability. If external information exists—such as later retention, known geography or expert classifications—it can be used after clustering to assess usefulness, while avoiding leakage into an exploratory analysis.

Interpretability

Each cluster should be profiled using original, interpretable variables. Report size, central tendency, spread and relevant categorical proportions. Avoid labelling groups with unsupported psychological or social claims. A cluster is a data-derived segment, not automatically a real-world identity.

8. Applied Case: Customer Segmentation

The following is an illustrative case rather than a report of a real company.

Background and problem

A subscription retailer wants to design differentiated service policies. It has twelve months of customer records containing purchase frequency, average order value, months since last purchase, number of product categories purchased and support contacts. No customer labels are available.

Analysis

The data-science team first removes account identifiers and checks whether multiple rows represent the same customer. Missing values are documented and imputed only where justified. Purchase frequency and order value are right-skewed, so the team considers a logarithmic transformation after confirming that values are non-negative. Features are then standardized so that order value does not dominate the distance calculation.

The team compares k-means for k from 2 to 7, hierarchical clustering using Ward linkage, and HDBSCAN. Mean silhouette, cluster stability across repeated samples, cluster size and managerial interpretability are reviewed together. A solution with four k-means clusters has slightly lower silhouette than a three-cluster solution but produces stable and actionable segments. The decision is therefore not based on the metric alone.

Approach and solution

The four segments are profiled as follows:

These descriptions are interpretations of feature patterns, not labels discovered by the algorithm. The team validates them using a later period of data. It measures retention, profit, complaint rates and offer response rather than assuming that a segment is valuable because it is large.

Outcome and learning points

The segmentation supports targeted experiments, but the organisation avoids treating clusters as permanent. Customers may move between segments as behaviour changes. The model is retrained periodically, and campaign performance is compared with a control group. This prevents the common error of claiming that clustering caused an improvement when no counterfactual comparison exists.

The case demonstrates that clustering is a complete workflow: question definition, feature design, preprocessing, algorithm comparison, evaluation, interpretation, action and monitoring.

9. Operational and Scientific Applications

Operational data

Clustering can group stores by demand pattern, machines by sensor behaviour, delivery routes by delay profile or incidents by operational signature. Time-series data require care. Treating each timestamp as an independent row may ignore temporal dependence. Useful alternatives include summary features, seasonal profiles, time-series distances or feature extraction from frequency and trend components.

For anomaly detection, a small or low-density cluster may be operationally important, but it may also reflect data-quality problems. Domain experts should inspect examples before changing a process.

Scientific data

In biology, clustering can group genes, cells or samples according to expression profiles. In earth science, it can identify regions with similar climate or mineral characteristics. Scientific conclusions require attention to sampling design, measurement error, batch effects, multiple representations and reproducibility. A computational cluster is not automatically a biological species, disease subtype or causal mechanism.

10. Ethical, Governance and Professional Issues

Clustering can create sensitive profiles even when protected attributes are excluded. Proxy variables, such as location or purchasing behaviour, may reproduce socioeconomic or demographic distinctions. Potential harms include discriminatory pricing, exclusion from services and intrusive targeting.

Responsible practice includes:

• documenting the purpose, data sources, transformations, distance measure and algorithm;
• assessing whether features are necessary and proportionate;
• testing cluster composition and outcomes across relevant groups;
• limiting access to sensitive segment labels;
• explaining that segments are probabilistic analytical constructs;
• offering review or correction mechanisms where decisions affect people;
• monitoring drift and unintended consequences;
• complying with applicable privacy, consumer-protection and data-governance requirements.

Unsupervised learning can appear objective because no target label is supplied. In reality, analyst choices determine what “similar” means. Transparency about these choices is therefore an essential professional obligation.

11. Contemporary Developments and Best Practice

Modern workflows increasingly combine clustering with dimensionality reduction for visualization, representation learning for complex data and scalable distributed computation. Embeddings can place text, images or products into a vector space before clustering, but the embedding itself introduces assumptions and potential bias. Cluster visualizations in two dimensions may distort high-dimensional relationships and should not replace quantitative checks.

A practical workflow is:

1. Define the decision or scientific question.
2. Select variables with domain experts.
3. Clean, transform and scale data appropriately.
4. Choose a distance measure consistent with the data.
5. Compare suitable algorithms and parameter settings.
6. Evaluate cohesion, separation and stability.
7. Profile and name clusters cautiously.
8. Validate against outcomes or expert judgement.
9. Deploy only with monitoring, governance and a review plan.

Chapter Summary

Clustering is an unsupervised-learning approach that discovers structure without a supplied target variable. Its results depend on feature selection, preprocessing, scaling, similarity measures and algorithmic assumptions. K-means efficiently creates compact partitions using centroid updates and minimizes within-cluster squared error, but requires k and is sensitive to scale, outliers and initialization. Hierarchical clustering provides a dendrogram and supports analysis of nested relationships through different linkage rules. Density-based methods such as DBSCAN identify dense, potentially irregular groups and can label noise, but require careful parameter selection and may struggle with varying densities.

Cluster evaluation should combine elbow analysis, silhouette or related indices, stability testing, visualization, domain expertise and—where available—external outcomes. Applications include customer segmentation, operational analysis and scientific discovery. Ethical use requires awareness that clusters are constructed analytical representations, not unquestionable natural categories. Effective practice therefore links technical modelling with interpretation, validation, monitoring and governance.

Key Terms

Unsupervised learning: Learning from data without a supplied target label.

Clustering: Grouping observations according to similarity or distance.

Observation: One entity or record represented in a dataset.

Feature: A measured attribute used to represent an observation.

Partition: An assignment of observations to groups.

Centroid: The arithmetic mean point representing a k-means cluster.

Medoid: An actual observation selected as a representative cluster point.

Euclidean distance: Straight-line distance between numeric vectors.

Manhattan distance: Sum of absolute coordinate-wise differences.

Cosine similarity: Similarity based on the angle between two vectors.

Jaccard similarity: Intersection divided by union for sets or binary attributes.

K-means: A centroid-based partitioning algorithm.

Dendrogram: A tree diagram showing hierarchical merges or splits.

Linkage: A rule for measuring distance between clusters.

Density-based clustering: Clustering based on high-density connected regions.

DBSCAN: A density-based algorithm using neighbourhood radius and minimum points.

Silhouette coefficient: A measure comparing within-cluster and nearest-other-cluster distances.

Inertia: The k-means within-cluster sum of squared distances.

Cluster stability: The consistency of clusters under resampling or modelling changes.

Segmentation: Dividing a population into analytically useful groups.

Review Questions

5 Short Answer Questions

1. What distinguishes supervised learning from unsupervised learning?
2. Why should numeric features commonly be scaled before k-means?
3. State one strength and one limitation of hierarchical clustering.
4. What roles do ε and MinPts play in DBSCAN?
5. What does a negative silhouette coefficient suggest?

5 Long Answer Questions

1. Explain the k-means objective and describe Lloyd’s algorithm, including why initialization can affect the result.
2. Compare single, complete, average and Ward linkage in hierarchical clustering.
3. Discuss how the choice of distance measure should reflect data type and application context.
4. Explain why cluster-number selection should combine internal metrics, stability and domain interpretation.
5. Design a responsible end-to-end workflow for customer segmentation using an unsupervised method.

3 Analytical / Application Questions

1. A dataset contains annual income, age, number of purchases and a binary indicator for mobile-app use. Propose a preprocessing and distance strategy, and justify how you would prevent income from dominating the analysis.
2. DBSCAN identifies one large cluster, two small clusters and many noise points in machine-sensor data. Give at least three possible explanations and describe checks you would perform before declaring the small clusters to be faults.
3. A retailer chooses a four-cluster solution because it has a slightly lower silhouette than a three-cluster solution but produces stable, interpretable segments and better performance in a controlled campaign. Evaluate this decision and identify safeguards needed before deployment.

Multiple Choice Questions with Answers

Question 1

Which statement best describes unsupervised learning?

A. It predicts a known target from labelled examples.

B. It discovers patterns without a supplied target label.

C. It always produces a binary classification.

D. It requires human labels for every observation.

Correct Answer: B. It discovers patterns without a supplied target label.

Question 2

What is the main purpose of standardizing features before Euclidean-distance clustering?

A. To guarantee the global optimum.

B. To remove all outliers.

C. To prevent numerical scale from dominating distance.

D. To convert categorical variables into labels.

Correct Answer: C. To prevent numerical scale from dominating distance.

Question 3

In k-means, what does a centroid represent?

A. The most distant observation in a cluster.

B. The arithmetic mean of observations assigned to a cluster.

C. A point labelled as noise.

D. The median distance between all clusters.

Correct Answer: B. The arithmetic mean of observations assigned to a cluster.

Question 4

Which k-means objective is minimized?

A. The sum of squared distances from observations to their assigned centroids.

B. The number of features in the dataset.

C. The maximum distance between all observations.

D. The number of missing values.

Correct Answer: A. The sum of squared distances from observations to their assigned centroids.

Question 5

Which linkage uses the maximum pairwise distance between two clusters?

A. Single linkage

B. Complete linkage

C. Average linkage

D. Ward linkage

Correct Answer: B. Complete linkage

Question 6

Which characteristic is especially associated with DBSCAN?

A. It must be given the exact number of clusters.

B. It can identify irregular shapes and label noise.

C. It always assumes spherical clusters.

D. It requires every observation to belong to a cluster.

Correct Answer: B. It can identify irregular shapes and label noise.

Question 7

A silhouette coefficient close to 1 generally indicates that an observation is:

A. Well matched to its own cluster and separated from others.

B. Definitely an outlier.

C. Equidistant from every cluster.

D. Assigned to a cluster with no other observations.

Correct Answer: A. Well matched to its own cluster and separated from others.

Question 8

For sets A and B, which formula defines Jaccard similarity?

A. |A ∪ B| / |A ∩ B|

B. |A ∩ B| / |A ∪ B|

C. |A| + |B|

D. |A − B| / |A ∪ B|

Correct Answer: B. |A ∩ B| / |A ∪ B|

Question 9

Why is a low k-means inertia value alone insufficient for selecting k?

A. Inertia is unrelated to clustering.

B. Inertia always increases as k increases.

C. Inertia cannot be calculated for numeric data.

D. Inertia generally decreases as more clusters are added.

Correct Answer: D. Inertia generally decreases as more clusters are added.

Question 10

Which is an appropriate governance practice for segmentation affecting customers?

A. Treat cluster labels as permanent identities.

B. Exclude all domain experts from interpretation.

C. Monitor outcomes and assess possible discriminatory effects.

D. Use identifiers as clustering features.

Correct Answer: C. Monitor outcomes and assess possible discriminatory effects.