Segmentation and Unsupervised Analytics

Introduction and context

Unsupervised analytics examines data when the outcome of interest is not supplied as a labelled target. Instead of learning to predict a known class such as “will churn” or “will not churn,” the analyst searches for structure: groups of similar customers, compact representations of many variables, frequently co-occurring products, or observations that differ markedly from the majority.

These methods are especially useful in business analytics because customer behaviour, transactions and operational records often contain important patterns that were not anticipated when the data was collected. A retailer may not know the types of shoppers that exist before segmentation; a bank may not have labelled every unusual transaction; and a survey may contain dozens of correlated measures whose underlying dimensions are unclear.

However, unsupervised methods do not automatically discover objective truth. Results depend on variable selection, measurement scale, distance or similarity definition, algorithmic assumptions and managerial interpretation. A technically neat cluster can be commercially useless, while a statistically weak pattern may be operationally valuable if it supports a clear decision. The central professional task is therefore to connect mathematical structure with business purpose and evidence.

Learning objectives

By the end of this chapter, you should be able to:

1. distinguish clustering, principal component analysis (PCA), association analysis and anomaly discovery;
2. prepare data appropriately for unsupervised analysis;
3. apply and interpret k-means, hierarchical clustering, PCA and market-basket measures;
4. evaluate the stability, quality and practical usefulness of discovered patterns;
5. communicate segments and associations responsibly, including ethical and governance considerations.

1. The logic of unsupervised analytics

A dataset can be represented as a matrix X with n observations and p variables. Observation i is the vector xᵢ = (xᵢ₁, xᵢ₂, …, xᵢₚ). In supervised learning, a target variable y is available. In unsupervised learning, the algorithm uses X alone to identify a latent structure, summary representation or unusual observation.

Four related tasks are particularly important:

An unsupervised result is descriptive rather than automatically causal. If customers who buy umbrellas also buy waterproof shoes, this association does not prove that one purchase causes the other. Similarly, a cluster is a group defined by the selected variables and algorithm, not necessarily a naturally occurring social category.

Data preparation is part of the model

Before analysis, the analyst should define the unit of observation. A customer-level dataset, a transaction-level dataset and a product-level dataset answer different questions. Duplicate records, impossible values, inconsistent identifiers and leakage from future information can distort results.

Numerical variables measured on different scales often require standardisation. The common z-score transformation is:

where xᵢⱼ is the value of variable j for observation i, μⱼ is the sample mean of variable j, and σⱼ is its sample standard deviation. The transformed variable has mean approximately 0 and standard deviation 1, provided σⱼ > 0. Standardisation is appropriate when variables should contribute comparably and have meaningful means and standard deviations. It can be inappropriate when original units have direct managerial meaning or when a variable’s scale should deliberately receive greater weight.

For skewed monetary variables, a transformation such as x' = log(1+x) can reduce the influence of extreme values; x must be non-negative. Categorical variables require suitable encoding, and missing values should be investigated rather than silently replaced. Imputation choices can create artificial similarity, so sensitivity checks are advisable.

2. Similarity and distance

Clustering depends on how similarity is measured. For two observations x and y with p numerical variables, Euclidean distance is:

It is non-negative, symmetric and measured in the units of the variables when a single scale is used. It is sensitive to scale and to large deviations. Squared Euclidean distance, d²(x,y), is often used computationally because it avoids the square root while preserving the ordering of distances.

Manhattan distance is:

It can be more robust than Euclidean distance when differences accumulate across variables. For binary data, Jaccard similarity is often useful when shared presences matter but shared absences do not:

Here a is the number of attributes present in both records, b is present in A only, and c is present in B only. The denominator excludes joint absences. The chosen measure should reflect the business meaning of similarity, not merely software defaults.

3. Clustering for segmentation

Clustering partitions or organises observations so that members within a group are more similar than members in different groups. A useful segment should be:

• internally coherent;
• meaningfully different from other segments;
• sufficiently large or valuable to act upon;
• stable under reasonable data and modelling changes; and
• reachable through an operational decision such as pricing, communication or service design.

k-means clustering

The k-means algorithm assigns n observations to k clusters and seeks to minimise within-cluster squared error. Its objective is:

where Cᵣ is cluster r, μᵣ is its centroid, and ||xᵢ − μᵣ||² is squared Euclidean distance from observation i to that centroid. The algorithm alternates between assigning each observation to its nearest centroid and recalculating each centroid as the arithmetic mean of its assigned observations. It stops when assignments or the objective change negligibly.

The method assumes that numeric variables are meaningful, clusters are reasonably compact under the selected distance, and the arithmetic mean is an appropriate centre. It requires k to be specified, is sensitive to outliers and may converge to a local rather than global minimum. Different initial centroids can produce different results, so multiple starts are standard practice.

Worked example: one-dimensional k-means

Suppose standardised customer spending values are 1, 2, 3 and 10, and k = 2. Take initial centroids 2 and 10. Values 1, 2 and 3 are assigned to the first centroid; 10 is assigned to the second. The updated centroids are:

The within-cluster sum of squared errors is:

This solution is compact, but it does not establish that two segments are commercially appropriate. The isolated value 10 may represent a genuine premium customer, a data error or an outlier that should be analysed separately.

Choosing the number of clusters

There is no universally correct value of k. Analysts commonly compare:

• Elbow analysis: plot within-cluster sum of squares against k and look for diminishing improvement. The “elbow” is a judgement aid, not a theorem.
• Silhouette coefficient: for observation i, let a(i) be its mean distance to members of its own cluster and b(i) the smallest mean distance to another cluster. Then:

For non-singleton clusters, s(i) lies between −1 and 1. Higher values indicate better separation relative to cohesion; values near zero indicate overlap. The overall silhouette is the mean across observations. The measure is meaningful only relative to the selected distance and should not replace business validation.

• Stability: rerun the analysis on bootstrap samples, time periods or alternative initialisations. A useful segment should not disappear under small perturbations.
• Actionability: assess whether segment membership changes a real decision and whether the organisation can identify members in future.

Hierarchical clustering

Hierarchical clustering creates a nested tree, or dendrogram, rather than requiring one final k at the beginning. In agglomerative clustering, every observation starts as its own cluster; the closest clusters are repeatedly merged. Linkage rules include:

The dendrogram can be cut at different levels, making it useful for exploration. Single linkage can produce chaining; complete linkage is sensitive to extreme points; Ward’s method is designed for compact numerical clusters and is closely related to variance minimisation. Hierarchical methods can be computationally demanding for very large datasets.

Other segmentation approaches

Density-based methods such as DBSCAN identify dense regions separated by sparse regions and can label noise. They are useful for irregularly shaped groups and anomaly discovery, but their results depend on neighbourhood parameters and can struggle when density varies. Model-based clustering treats observations as generated by a mixture of probability distributions, often Gaussian distributions. It can provide probabilities of membership, but distributional assumptions must be checked.

4. Principal component analysis

PCA reduces correlated numerical variables to a smaller number of uncorrelated linear combinations called principal components. It is used for visualisation, noise reduction, feature construction and identifying broad dimensions in a measurement system.

After centring (and often standardising) the data, the covariance matrix is:

Here X is the centred n × p data matrix, Xᵀ is its transpose, n is the number of observations and S is a p × p covariance matrix. If variables are standardised, the corresponding matrix is the correlation matrix.

A principal component is a linear combination:

where wₘ is a p-element loading vector. The first component chooses a unit-length vector w₁ that maximises the variance of Xw₁. Successive components are constrained to be orthogonal to preceding components and maximise the remaining variance. Algebraically, the loading vectors are eigenvectors of S and their eigenvalues λₘ are the variances of the components.

The proportion of variance explained by component m is:

The denominator is total variance when S is used. With correlation-based PCA, each standardised variable contributes variance 1, so total variance equals p. PCA is descriptive: components maximise variance, not predictive power, causal meaning or managerial importance.

Interpreting PCA

Loadings indicate the contribution of original variables to a component. A component with large positive loadings for frequency, recency-adjusted activity and spending may be interpreted as an engagement dimension. Interpretation requires examining signs, magnitudes and the original variables. Sign is arbitrary: multiplying a component and its loadings by −1 gives the same substantive solution.

PCA is most useful when variables are numeric, relationships are approximately linear and correlations justify reduction. It can be distorted by outliers and by arbitrary coding of categorical variables. If variables have different units, analysts must decide whether covariance-based PCA or standardised correlation-based PCA is appropriate. Components retained solely because they explain variance may not be the best predictors or decision variables.

5. Association analysis and market baskets

Association analysis examines item co-occurrence in transactions. Let A and B be itemsets and let N be the number of transactions. The support of itemset A is:

It is the proportion of transactions containing A. A rule A → B has confidence:

provided support(A) > 0. Confidence estimates the proportion of A-containing transactions that also contain B, but it can be high simply because B is common. Lift adjusts for the baseline frequency of B:

Equivalently, lift is support(A∪B) divided by support(A)×support(B), provided both supports are positive. Lift greater than 1 indicates positive association relative to independence; lift approximately 1 is consistent with independence; lift below 1 indicates a negative association. These are descriptive measures and do not establish causation.

Worked example: a retail rule

In 1,000 transactions, 200 contain coffee, 300 contain bread and 120 contain both. For coffee → bread:

Thus 12% of all baskets contain both products, 60% of coffee baskets also contain bread, and the joint occurrence is twice the level expected from bread’s overall prevalence. A retailer still needs to consider margin, stock availability, seasonality, promotions and whether the rule persists in a validation period.

Apriori and rule discovery

The Apriori principle states that if an itemset is infrequent, every larger itemset containing it must also be infrequent. This enables pruning: candidate itemsets whose subsets fail a minimum-support threshold are not expanded. A typical workflow is to encode transactions as item-presence records, select minimum support, generate frequent itemsets, produce rules, and filter using confidence, lift and business constraints.

Association mining can generate thousands of apparently strong rules. Multiple testing and selection bias make validation essential. Split historical transactions into discovery and evaluation periods, or validate rules in a later campaign. Rules should also be assessed using incremental response rather than merely basket co-occurrence.

6. Anomaly discovery

An anomaly is an observation that deviates substantially from a reference pattern. In one variable, a z-score can flag unusually high or low values:

A common descriptive rule flags |zᵢ| > 3, but this threshold is not universal and is unreliable for skewed or heavy-tailed data. A robust alternative uses the median and median absolute deviation (MAD):

For multivariate data, the Mahalanobis distance is:

where μ is the mean vector, S is the covariance matrix and S⁻¹ is its inverse. This distance accounts for scale and correlation, assuming S is suitably estimated and invertible. In high dimensions or with collinear variables, covariance estimation can be unstable; robust covariance, PCA-based methods, isolation forests or local-density methods may be preferable.

Anomaly scores are not proof of fraud. A legitimate high-value customer, a system migration or a seasonal event may look unusual. Investigation workflows should combine scores with context, human review and documented escalation rules.

7. Applied case: illustrative omnichannel retailer

Background

BrightCart, an illustrative national retailer, has 180,000 customers and wants to improve retention and campaign relevance. Its data contains annual orders, monetary value, product-category shares, discount use, mobile activity and customer-service contacts. Transaction data also records products purchased together.

Problem

Marketing currently uses one broad campaign. Managers want defensible segments, a reduced set of behavioural dimensions, useful basket rules and a process for identifying unusual purchasing behaviour.

Analysis and approach

The analytics team defines the customer as the unit of analysis and uses the previous 12 months, preventing future information from entering the segmentation. Monetary values are log-transformed, extreme data errors are investigated, and continuous variables are standardised. Variables are selected based on intended decisions: contact frequency, service design and offer type. Postal code is excluded from the initial model to reduce the risk of reproducing socioeconomic discrimination.

Several k-means solutions from k = 2 to k = 6 are compared. A four-cluster solution is selected because it has acceptable silhouette performance, remains broadly stable across bootstrap samples and supports distinct actions:

1. Routine value shoppers: regular purchases and moderate value;
2. Premium convenience shoppers: high value, high digital activity and low discount dependence;
3. Promotion-responsive shoppers: high discount use and irregular activity;
4. At-risk occasional shoppers: declining activity and recent service contact.

The labels are descriptive summaries, not permanent identities. Segment profiles are validated in a later month and linked to campaign response. PCA is then applied to correlated engagement variables. The first component represents digital engagement, while the second contrasts value with discount dependence. These components are used for visualisation and interpretation, not as automatic substitutes for the original operational variables.

For market-basket analysis, rules are discovered in the first nine months and evaluated in the final three months. A rule involving coffee accessories and premium coffee has reasonable support and lift, but is used only for customers who have not already purchased the recommended item. Margin and inventory constraints are added before deployment.

Anomaly detection flags sudden high-value orders and unusual combinations of delivery location and payment behaviour. The system sends alerts for review; it does not automatically block customers.

Outcome

The retailer gains a manageable segmentation vocabulary and tests differentiated campaigns. The premium segment receives service benefits rather than blanket discounts, while promotion-responsive customers receive carefully bounded offers. The team rejects several unstable clusters and low-support rules. The project is considered successful not because it maximises a single statistical score, but because the patterns are stable, interpretable, measurable and connected to controlled actions.

Learning points

The case demonstrates that segmentation is an iterative decision process. Data preparation, validation, ethics, operational feasibility and measurement of incremental outcomes are as important as algorithm selection. PCA improves understanding but does not automatically create superior business features; association rules require temporal and commercial validation; and anomalies require investigation rather than punishment.

8. Evaluation, ethics and governance

Practical evaluation framework

A professional review should ask:

• Data validity: Are the observation unit, time window and variables appropriate?
• Statistical quality: Are clusters coherent, separated and stable? Are rules supported beyond the discovery sample?
• Interpretability: Can stakeholders explain what distinguishes groups?
• Actionability: Is there a feasible intervention and a measurable outcome?
• Economic value: Do benefits exceed campaign, service and implementation costs?
• Reproducibility: Are preprocessing, random seeds, parameters and versions recorded?

Ethical and regulatory issues

Unsupervised methods can create sensitive inferred attributes even when protected variables are excluded. A cluster may indirectly encode ethnicity, health, income or vulnerability through location and behaviour. Uses involving credit, insurance, employment or essential services require heightened governance and may be subject to privacy, equality and automated-decision regulation.

Organisations should establish a lawful purpose, data minimisation, retention limits, access controls and transparency appropriate to the context. Customers should not be unfairly denied service because an opaque anomaly model flags them. Analysts should test segment representation and outcome differences across relevant groups, document legitimate business justification and provide human review or appeal where decisions materially affect individuals.

Technology and contemporary practice

Modern analytics platforms support distributed computation, automated feature pipelines, interactive embeddings and monitoring dashboards. Automated machine learning can compare algorithms and parameters, but it cannot decide whether a segment is ethical or operationally meaningful. Representation learning and neural embeddings can capture complex similarity, yet their dimensions may be difficult to explain. Generative AI can assist with segment narratives or rule summaries, but all generated interpretations require verification against source data.

Production systems should monitor population drift, segment size, rule support, anomaly rates and campaign outcomes. A model that was appropriate during a promotion or pandemic may become misleading when behaviour changes. Versioned data, reproducible code, approval records and periodic review are essential controls.

Best-practice workflow

1. Define the decision and unit of analysis.
2. Audit quality, missingness, leakage, privacy and representativeness.
3. Select variables and similarity measures linked to the decision.
4. Transform and scale transparently; document every choice.
5. Fit multiple plausible methods and parameter settings.
6. Evaluate statistical quality, stability and interpretability.
7. Validate on later data or a holdout period.
8. Translate patterns into constrained, ethical actions.
9. Measure incremental business and customer outcomes.
10. Monitor drift and retire patterns that no longer remain useful.

Chapter Summary

Unsupervised analytics discovers structure without a labelled outcome. Clustering supports segmentation, but its results depend on data preparation, distance measures and algorithmic assumptions. K-means minimises within-cluster squared distance and is useful for compact numeric groups; hierarchical and density-based methods offer different structures and assumptions. PCA converts correlated variables into orthogonal components that maximise variance, assisting reduction and visualisation but not automatically providing causal or predictive meaning. Association analysis uses support, confidence and lift to describe co-occurring items, while anomaly methods identify observations requiring investigation. Across all methods, statistical quality must be combined with stability, interpretability, economic usefulness, ethical governance and operational validation.

Key Terms

Clustering: Grouping observations according to a chosen similarity or distance measure.

Unsupervised learning: Analysis in which no labelled target variable is supplied.

Segmentation: Dividing a population into groups for differentiated analysis or action.

Euclidean distance: Straight-line distance based on squared differences across variables.

Standardisation: Rescaling a variable to have mean 0 and standard deviation 1.

Centroid: The arithmetic mean location of observations in a cluster.

k-means: An algorithm that minimises within-cluster squared distances for a chosen k.

Hierarchical clustering: Clustering that produces a nested tree of merges or divisions.

Silhouette coefficient: A measure comparing an observation’s within-cluster cohesion with its nearest alternative cluster.

Principal component: A linear combination of variables capturing an ordered amount of variance.

Loading: The coefficient linking an original variable to a principal component.

Eigenvalue: The variance associated with a principal component in PCA.

Dimensionality reduction: Representing data using fewer variables or dimensions.

Itemset: A set of products or events occurring in a transaction.

Support: The proportion of transactions containing an itemset.

Confidence: The proportion of antecedent transactions that also contain the consequent.

Lift: Confidence divided by consequent support, measuring association relative to baseline frequency.

Anomaly: An observation that differs substantially from a reference pattern.

Data leakage: Use of information unavailable at the time a decision would be made.

Actionability: The extent to which an analytical result supports a feasible intervention.

Review Questions

Short Answer Questions

1. Why should variables often be standardised before Euclidean-distance clustering?
2. What does the silhouette coefficient compare?
3. Distinguish support, confidence and lift in association analysis.
4. Why is PCA not automatically a predictive method?
5. Give two reasons why an anomaly flag should not be treated as proof of fraud.

Long Answer Questions

1. Explain the k-means algorithm, its objective function, assumptions and limitations.
2. Compare k-means, hierarchical clustering and density-based clustering for customer segmentation.
3. Describe PCA mathematically and explain how loadings and explained variance are interpreted.
4. Explain how association rules can be discovered, evaluated and validated in a retail setting.
5. Develop a complete governance and evaluation framework for an unsupervised segmentation project.

Analytical / Application Questions

1. A retailer has variables measured in dollars, number of visits and percentage discount use. Propose a preparation and clustering strategy, including how you would assess the chosen number of clusters.
2. In 2,000 baskets, 400 contain A, 500 contain B and 160 contain both. Calculate support for A and B together, confidence for A → B and lift. Interpret the result.
3. A bank’s anomaly model flags a small group concentrated in one neighbourhood. Design an investigation that tests data quality, model bias, legitimate explanations and customer impact before any intervention.

Multiple Choice Questions with Answers

Question 1

Which statement best describes unsupervised analytics?

A. It predicts a labelled target for every observation.

B. It discovers structure without a supplied target label.

C. It estimates only causal effects from experiments.

D. It replaces data preparation with automated modelling.

Correct Answer: B. It discovers structure without a supplied target label.

Question 2

What is the main purpose of standardising variables before distance-based clustering?

A. To make every observation identical.

B. To remove all correlations between variables.

C. To prevent variables with large units from dominating distance.

D. To guarantee that the selected number of clusters is correct.

Correct Answer: C. To prevent variables with large units from dominating distance.

Question 3

In k-means, what does the centroid represent?

A. The mean location of observations assigned to a cluster.

B. The most distant observation in a cluster.

C. The median distance between all clusters.

D. The first principal component of the dataset.

Correct Answer: A. The mean location of observations assigned to a cluster.

Question 4

A silhouette value close to zero generally indicates that an observation is:

A. Perfectly separated from every other cluster.

B. Very close to the centre of its own cluster and far from all others.

C. Likely to be an impossible data value.

D. Near the boundary between its cluster and an alternative cluster.

Correct Answer: D. Near the boundary between its cluster and an alternative cluster.

Question 5

What does PCA primarily maximise for the first component?

A. The number of business rules generated.

B. The variance of the projected observations.

C. The number of clusters in the data.

D. The confidence of an association rule.

Correct Answer: B. The variance of the projected observations.

Question 6

In market-basket analysis, a lift of 2 for A → B means that:

A. A occurs in exactly 2% of transactions.

B. Two products must always be purchased together.

C. B is observed with A at twice the level expected from B’s baseline frequency.

D. The rule proves that A causes B.

Correct Answer: C. B is observed with A at twice the level expected from B’s baseline frequency.

Question 7

Using the case values of 1,000 transactions, 200 containing coffee and 120 containing both coffee and bread, what is confidence for coffee → bread?

A. 0.12

B. 0.20

C. 0.30

D. 0.60

Correct Answer: D. 0.60

Question 8

Which statement about association rules is correct?

A. High confidence alone always proves a useful relationship.

B. Lift compares observed co-occurrence with a baseline based on item frequency.

C. Support is calculated only among transactions containing the antecedent.

D. Association rules establish causation by definition.

Correct Answer: B. Lift compares observed co-occurrence with a baseline based on item frequency.

Question 9

Why can Mahalanobis distance be problematic in high-dimensional data?

A. It cannot use more than one variable.

B. It ignores covariance among variables.

C. Its covariance estimate may be unstable or non-invertible.

D. It is defined only for categorical variables.

Correct Answer: C. Its covariance estimate may be unstable or non-invertible.

Question 10

Which is the most appropriate response when an anomaly model flags a customer?

A. Automatically terminate the customer relationship.

B. Publish the customer’s score for transparency.

C. Ignore the alert because anomalies are never useful.

D. Investigate the alert with context, human review and documented rules.

Correct Answer: D. Investigate the alert with context, human review and documented rules.