Segmentation and Unsupervised Learning

Introduction

Organizations often possess large volumes of data without a predefined target variable. A retailer may know what customers bought but not which customer groups exist; a bank may observe transactions without knowing in advance which patterns indicate fraud; and a manufacturer may record sensor readings without labels for operational states. Unsupervised learning addresses such problems by discovering structure in data rather than predicting a known outcome.

This chapter examines four closely related techniques: clustering, dimensionality reduction, association-rule mining, and anomaly detection. Their common purpose is pattern discovery, but they answer different questions. Clustering identifies similar observations, dimensionality reduction summarizes high-dimensional data, association rules reveal co-occurrence relationships, and anomaly detection identifies observations that differ substantially from expected patterns.

Effective unsupervised analytics requires more than selecting an algorithm. Analysts must define the business problem, prepare variables appropriately, select meaningful similarity measures, assess stability, interpret the resulting patterns, and determine whether the findings support action. A mathematically elegant solution may have little business value if segments cannot be reached, explained, or served profitably.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain the purpose and assumptions of unsupervised learning.
2. Select and apply clustering methods for customer and operational segmentation.
3. Use dimensionality-reduction methods to summarize complex data.
4. Construct and interpret association rules for market-basket analysis.
5. Explain anomaly-detection methods for fraud and operational monitoring.
6. Evaluate models using stability, interpretability, validity, and business usefulness.
7. Identify ethical, governance, and implementation risks.

1. Foundations of Unsupervised Learning

In supervised learning, a dataset contains input variables and a target variable. In unsupervised learning, the analyst observes a data matrix but does not have a target that directly specifies the correct answer. Let X be an n × p matrix, where n is the number of observations and p is the number of variables. The purpose is to discover a useful representation or structure in X.

Unsupervised learning is exploratory rather than automatically objective. Different choices of variables, scaling, distance measures, and algorithms can produce different structures. Consequently, a discovered pattern should be treated as a model of usefulness, not as an unquestionable natural fact.

Important terminology

• Observation: A unit being analyzed, such as a customer, transaction, product, or machine.
• Feature or variable: A measured characteristic of an observation.
• Similarity: The degree to which two observations resemble one another.
• Distance: A numerical measure of dissimilarity.
• Cluster: A group of observations that are more similar to one another than to observations outside the group.
• Latent structure: An unobserved pattern inferred from measured variables.
• Dimensionality: The number of variables used to represent observations.
• Support: The proportion of transactions containing an itemset.
• Anomaly: An observation that is unusually different from expected behavior.

A typical workflow is:

1. Define the decision or business use.
2. Specify the unit of analysis and observation period.
3. Clean data, handle missingness, and prevent leakage.
4. Transform and scale variables where appropriate.
5. Select an algorithm and its assumptions.
6. Evaluate statistical validity and stability.
7. Profile and interpret the discovered patterns.
8. Test operational and financial usefulness.
9. Monitor drift and revise the analysis.

2. Data Preparation and Similarity

Unsupervised methods are particularly sensitive to data preparation because there is no target variable to expose errors. Duplicate customers, inconsistent time windows, extreme values, and inappropriate inclusion of identifiers can create artificial structure.

Variable selection

Variables should represent behavior relevant to the decision. For customer segmentation, useful features might include purchase frequency, monetary value, category breadth, discount dependence, service contacts, and digital engagement. A customer ID is an identifier, not a behavioral feature, and should not be used as a clustering input.

Variables should be measured over a consistent period. If one customer has twelve months of purchase history and another has three months, purchase frequency is not comparable unless exposure time is considered.

Scaling

Suppose one feature is annual spending in currency units and another is number of visits. A distance calculation may be dominated by spending simply because its numerical scale is larger. Standardization commonly transforms a variable x into a z-score:

Here, x_i is the value for observation i, μ_x is the sample mean, and σ_x is the sample standard deviation. The transformed variable has mean approximately 0 and standard deviation approximately 1, provided σ_x > 0. Standardization is appropriate when variables measured on different scales should have comparable influence and when mean and standard deviation are not severely distorted by outliers.

Robust scaling may use the median and interquartile range (IQR):

This is often preferable when extreme values are meaningful but would excessively influence the mean and standard deviation.

Distance measures

For two observations x and y with p numerical features, Euclidean distance is:

It is appropriate when variables are numeric, scaled suitably, and straight-line distance is meaningful. Manhattan distance is:

It can be more robust to individual large coordinate differences. For binary variables, Jaccard similarity is often useful when shared absences are uninformative:

The corresponding Jaccard distance is 1 − J(A,B). Mixed data may require a mixed-type distance measure or a carefully designed transformation. Analysts should not apply Euclidean distance mechanically to categorical variables encoded with arbitrary numerical labels.

3. Clustering Methods

Clustering divides observations into groups according to a similarity criterion. The appropriate method depends on the expected cluster shape, data type, noise level, and operational purpose.

3.1 k-means clustering

The k-means algorithm assigns each observation to one of k clusters and estimates a centroid for each cluster. It seeks to minimize within-cluster sum of squares:

Here, C_r is cluster r, μ_r is its centroid, and ||x_i − μ_r||² is squared Euclidean distance. The algorithm alternates between assigning each observation to its nearest centroid and recomputing each centroid as the mean of its assigned observations.

The method is computationally efficient and easy to explain. However, it requires k to be specified, is sensitive to initialization and scaling, and tends to favor compact, roughly spherical clusters. It is not naturally suited to highly skewed variables, categorical data, or clusters with very different densities.

A practical procedure is to run k-means several times with different random starts, compare solutions, and use a fixed seed for reproducibility. Multiple starts reduce the chance of accepting a poor local solution, but they do not prove that the selected k is correct.

3.2 Hierarchical clustering

Hierarchical clustering creates a nested sequence of groups represented by a dendrogram. Agglomerative clustering begins with each observation as a separate cluster and repeatedly merges the closest clusters. The definition of distance between clusters depends on the linkage rule:

• Single linkage: minimum pairwise distance; can produce chaining.
• Complete linkage: maximum pairwise distance; favors compact groups.
• Average linkage: average pairwise distance.
• Ward’s method: merges groups producing the smallest increase in within-cluster squared error; generally assumes numeric data and squared Euclidean distance.

Hierarchical methods do not require a single value of k at the beginning. The dendrogram can be cut at different levels. Their disadvantages include higher computational cost for large datasets and sensitivity to the distance and linkage choices.

3.3 Density-based clustering

Density-based methods define clusters as dense regions separated by sparse regions. DBSCAN uses two principal parameters: ε, the neighborhood radius, and minPts, the minimum number of nearby observations required to define a dense region. It can identify irregularly shaped clusters and label isolated points as noise. It performs poorly when clusters have very different densities or when the feature space is high-dimensional without suitable preparation.

3.4 Choosing the number of clusters

The elbow method plots within-cluster variation against k and looks for a point beyond which additional clusters yield diminishing improvement. The silhouette coefficient for observation i is:

Here, a(i) is the average distance from i to observations in its own cluster, and b(i) is the smallest average distance from i to observations in another cluster. Values near 1 indicate good separation, values near 0 indicate overlap, and negative values suggest possible misassignment. The overall silhouette score is the mean of s(i) across observations.

These are diagnostic measures, not decision rules. A solution with a slightly lower silhouette score may be more useful if it produces segments that marketing can identify and serve distinctly.

Worked example: customer segmentation

A retailer summarizes six customers using standardized purchase frequency and average order value:

A two-cluster k-means solution might produce a high-value group {A, B} and a lower-activity group {C, D, E, F}. The result is not automatically a valid marketing segmentation. Analysts should inspect whether E and F are genuinely similar to C and D, compare alternative solutions, and add variables such as category preference or discount usage. The segment labels should describe behavior rather than imply personal worth—for example, “high-frequency, high-value customers” rather than “best customers.”

4. Dimensionality Reduction

High-dimensional data can be difficult to visualize, model, and interpret. Dimensionality reduction transforms p variables into a smaller number of components while attempting to retain important information.

Principal component analysis

Principal component analysis (PCA) creates linear combinations of centered variables. The first principal component is the direction that maximizes projected variance; subsequent components are orthogonal to previous components and maximize remaining variance.

If X is a centered n × p data matrix, the covariance matrix is:

The principal directions are eigenvectors of S. If λ_j is the eigenvalue associated with component j, its explained-variance ratio is:

The ratio is dimensionless, and the sum of all ratios is 1 when all p components are retained. PCA is appropriate for numeric variables and is sensitive to scaling; analysts commonly standardize variables when units differ. PCA captures variance, not necessarily business importance or predictive information.

Other methods

• Factor analysis models observed variables as manifestations of fewer latent factors plus error and is often used when a latent-concept interpretation is central.
• t-distributed stochastic neighbor embedding (t-SNE) is useful for visual exploration of local neighborhoods but can distort global distances and should not be treated as a clustering proof.
• Uniform Manifold Approximation and Projection (UMAP) can provide a useful low-dimensional visualization, but results depend on parameters and random initialization.
• Multiple correspondence analysis is designed for categorical data.

Dimensionality reduction can precede clustering, but this should be justified. Removing components may eliminate low-variance features that are commercially important, and clustering in a reduced space can produce different results from clustering in the original feature space.

5. Association-Rule Mining

Association analysis examines which items occur together in transactions. It is widely used in market-basket analysis, recommendation systems, website navigation, and service bundling.

For an itemset A, support is:

For a rule A → B, confidence is:

Confidence estimates the proportion of transactions containing A that also contain B. It is not the probability that A causes B.

Lift compares the observed co-occurrence with the co-occurrence expected if A and B were independent:

A lift greater than 1 indicates positive association, lift equal to 1 is consistent with independence, and lift below 1 indicates negative association. The formulas assume transactions are defined consistently and items are represented accurately.

Worked example

Suppose 1,000 shopping baskets are observed. Coffee appears in 200 baskets, filters in 100 baskets, and both in 60 baskets.

• support(coffee) = 200/1,000 = 0.20, or 20%.
• support(filters) = 100/1,000 = 0.10, or 10%.
• support(coffee ∪ filters) = 60/1,000 = 0.06, or 6%.
• confidence(coffee → filters) = 60/200 = 0.30, or 30%.
• lift = 0.30/0.10 = 3.0.

Thus, 30% of coffee buyers also bought filters, which is three times the rate expected from filters’ overall frequency. The result may support merchandising or recommendation, but it does not establish that buying coffee causes filter purchases.

A common algorithm is Apriori, which uses the downward-closure property: if an itemset is infrequent, every larger itemset containing it must also be infrequent. This permits pruning of candidate itemsets. Analysts should control for extremely common items, multiple testing, seasonality, promotions, and profitability. A rule with high lift may concern a rare combination and generate little revenue.

6. Anomaly Detection

Anomaly detection identifies observations that differ substantially from a reference pattern. Applications include payment fraud, cybersecurity, equipment monitoring, quality control, and unusual employee-access behavior.

Statistical methods

For a single approximately normal variable, a standardized score can flag extreme observations:

A threshold such as |z_i| > 3 is a heuristic, not a universal law. It assumes a reasonably stable distribution and becomes unreliable for skewed or multimodal data.

For multivariate data, the Mahalanobis distance is:

Here, μ is the mean vector and Σ is the covariance matrix, assumed invertible or suitably regularized. The measure accounts for correlations among variables. Under multivariate normal assumptions, D² can be compared approximately with a chi-square reference distribution, but real business data frequently violate normality.

Machine-learning methods

• Isolation Forest: randomly partitions observations; anomalies tend to be isolated in fewer splits.
• Local Outlier Factor: compares the local density around an observation with the density around its neighbors.
• One-class support vector machine: learns a boundary around most observations.
• Autoencoder: learns to reconstruct normal patterns; high reconstruction error may indicate an anomaly.

Anomaly detection is not the same as fraud classification. An unusual transaction may be legitimate, and fraudulent transactions may resemble normal transactions. A strong operational system combines anomaly scores with investigation, rules, customer context, and feedback from confirmed cases.

7. Evaluation: Stability, Interpretability, and Usefulness

Unsupervised results require multi-dimensional evaluation.

Statistical validity

Assess compactness and separation using measures such as within-cluster sum of squares, silhouette score, or adjusted Rand index when a trusted reference partition exists. For association rules, examine support, confidence, lift, statistical significance, and effect size. For anomaly systems, evaluate precision among investigated alerts, false-positive burden, detection delay, and coverage of known incidents.

Stability

A stable solution should not change radically when reasonable perturbations are introduced. Analysts can:

• resample observations and refit the model;
• vary scaling and preprocessing choices;
• test nearby values of k or algorithm parameters;
• compare random initializations;
• examine agreement between partitions.

For two clusterings, the adjusted Rand index (ARI) is one possible comparison measure. It adjusts agreement for the agreement expected by chance. Stability does not guarantee truth, but extreme instability warns that the apparent structure may be weak.

Interpretability

Segments should have distinctive, understandable profiles. Analysts can calculate feature means, medians, proportions, and distributions by cluster. Profiles should be validated against variables not used to create the clusters, such as future retention, channel response, or service cost. Labels must be evidence-based and avoid stigmatizing language.

Business usefulness

A useful segment has a meaningful difference in needs or economics, can be identified in future data, and supports a differentiated action. A segment that cannot be targeted, monitored, or served differently may be analytically interesting but operationally unnecessary. Decision criteria should include incremental revenue, cost reduction, risk reduction, customer experience, and implementation cost.

8. Applied Case: Illustrative Retail Analytics Project

Background

A fictional omnichannel retailer has 2.4 million customer records, transaction histories, website events, promotion exposures, and customer-service interactions. Marketing currently uses a single loyalty campaign for all customers, producing high discount costs and declining response rates.

Problem

The retailer wants to identify actionable customer groups, discover product affinities, and detect potentially fraudulent transactions without excluding legitimate customers unfairly.

Analysis and approach

The analytics team defines the customer as the unit of analysis and uses the previous twelve months for behavioral features. It excludes customer identifiers, standardizes skewed monetary variables using a log transformation after confirming that values are nonnegative, and caps only demonstrably erroneous transaction amounts. It tests k-means, hierarchical clustering, and a density-based method.

A four-segment k-means solution is selected provisionally because it is reasonably stable across bootstrap samples and produces interpretable profiles:

1. Frequent full-price shoppers: high frequency, low discount dependence, broad category usage.
2. Promotion-responsive shoppers: moderate frequency, high coupon usage, strong campaign response.
3. Occasional high-value shoppers: low frequency but high order value.
4. Low-engagement customers: infrequent purchases and limited digital activity.

The team uses association rules separately for baskets, finding a strong relationship between home-office equipment and accessory products. It checks support and profitability before introducing a bundle. For fraud monitoring, an Isolation Forest combines transaction amount, time, device history, geographic distance, and account-change events. Alerts are routed for review rather than automatically rejected.

Outcome

A controlled campaign experiment compares segment-specific offers with the previous universal campaign. The retailer evaluates incremental margin rather than response rate alone. It finds that promotion-responsive customers benefit from targeted coupons, while frequent full-price shoppers respond better to early access and service benefits. High-value occasional shoppers receive replenishment reminders rather than broad discounts. Fraud analysts report fewer low-quality alerts after incorporating customer history.

Learning points

The value came from the full process, not from clustering alone. The team defined a decision first, separated descriptive segmentation from fraud detection, tested stability, used holdout experiments, and maintained human review for high-impact fraud decisions. It also monitored whether segment membership changed because of genuine behavior or changes in data collection.

9. Ethical, Governance, and Implementation Issues

Unsupervised methods can reproduce bias even when protected characteristics are not explicitly included. Proxy variables such as location, language, device type, or purchasing channel may correlate with protected attributes. Segments can therefore lead to unequal pricing, exclusion, or surveillance.

Organizations should document the purpose, data sources, transformations, algorithms, parameters, validation results, and intended actions. Access to detailed customer and transaction data should follow data-minimization and security principles. Retention periods should be justified, and individuals should not be denied important services solely because an opaque anomaly score is high.

Fairness testing should examine whether error rates, service access, or commercial treatment differ systematically across relevant groups, subject to legal and privacy constraints. Human review, appeal procedures, audit logs, and model monitoring are important safeguards. Privacy-preserving aggregation may be appropriate for market-basket analysis, particularly when transactions can reveal sensitive health, financial, or personal information.

10. Contemporary Developments and Best Practices

Modern analytics platforms increasingly combine scalable distributed computation, streaming anomaly detection, graph analytics, and representation learning. Embeddings can represent products, customers, or events in vector spaces, enabling similarity search and recommendation. However, high-dimensional representations may be difficult to explain and can encode historical bias.

Generative AI can assist with cluster summaries or analyst interaction, but it should not be trusted to invent segment meanings. Every generated interpretation must be traced to measured evidence. Automated systems should preserve reproducibility through versioned data, code, parameters, and random seeds.

Best practices include:

• start with a decision and not an algorithm;
• define the unit, time window, and population clearly;
• remove identifiers and leakage variables;
• scale and transform variables deliberately;
• compare several plausible methods;
• assess stability under resampling and parameter changes;
• profile clusters using variables both inside and outside the model;
• validate actions through experiments or carefully designed quasi-experiments;
• monitor drift, alert quality, and business outcomes;
• document limitations and preserve human accountability.

Chapter Summary

Unsupervised learning discovers structure when no target variable is available. Clustering supports segmentation, but its results depend on variable selection, scaling, distance measures, algorithm assumptions, and the chosen number of groups. k-means is efficient for compact numerical clusters; hierarchical methods reveal nested structure; and density-based methods can identify irregular groups and noise.

Dimensionality reduction summarizes complex data. PCA creates orthogonal linear components ordered by explained variance, while other methods serve categorical data or visualization. Association-rule mining identifies co-occurring items using support, confidence, and lift, but association does not imply causation. Anomaly detection identifies unusual observations through statistical distances or machine-learning methods, yet unusual behavior is not automatically fraudulent.

Reliable unsupervised analytics requires evaluation of statistical validity, stability, interpretability, and business usefulness. Ethical practice requires attention to privacy, proxy discrimination, transparency, human review, and governance. The strongest projects connect discovered patterns to controlled actions and measurable outcomes.

Key Terms

Unsupervised learning: Learning that discovers structure without a predefined target variable.

Clustering: Grouping observations according to similarity.

Centroid: The mean position of observations assigned to a cluster.

k-means: A clustering method that minimizes within-cluster squared Euclidean distance.

Hierarchical clustering: A method that creates nested groupings represented by a dendrogram.

Density-based clustering: Clustering based on dense regions separated by sparse regions.

Distance measure: A numerical measure of dissimilarity between observations.

Standardization: Rescaling a variable by subtracting its mean and dividing by its standard deviation.

Principal component analysis: A method that transforms correlated variables into orthogonal components.

Explained-variance ratio: The proportion of total variance represented by a component.

Association rule: A conditional co-occurrence relationship such as A → B.

Support: The proportion of transactions containing an itemset.

Confidence: The proportion of A transactions that also contain B.

Lift: The ratio of observed conditional occurrence to occurrence expected under independence.

Anomaly: An observation that differs substantially from expected behavior.

Mahalanobis distance: A covariance-adjusted multivariate distance.

Stability: The degree to which a discovered pattern persists under reasonable changes.

Interpretability: The extent to which analysts and decision-makers can understand a model’s output.

Data drift: Change in the distribution or meaning of data over time.

Leakage: Use of information that would not be available at the time of the intended decision.

Review Questions

Short Answer Questions

1. Why is scaling important before many clustering algorithms are applied?
2. Distinguish support, confidence, and lift in association-rule mining.
3. What does a silhouette score measure?
4. Why is an anomaly score not equivalent to a fraud label?
5. State one advantage and one limitation of PCA.

Long Answer Questions

1. Compare k-means, hierarchical clustering, and density-based clustering in terms of assumptions, strengths, and limitations.
2. Explain how an organization should evaluate whether a customer segmentation is useful beyond statistical cluster quality.
3. Describe the PCA workflow, including centering, scaling, covariance, eigenvalues, and explained variance.
4. Explain how association rules can support retail decisions while avoiding causal overinterpretation.
5. Discuss ethical and governance risks in customer segmentation and anomaly detection.

Analytical / Application Questions

1. A retailer obtains a high silhouette score after clustering customers using only total annual spending. Analyze why the result may still be commercially weak and propose a better feature strategy.
2. In 10,000 baskets, 800 contain tea, 500 contain biscuits, and 160 contain both. Calculate support for tea and biscuits together, confidence for tea → biscuits, and lift for that rule. Interpret the results.
3. Design an evaluation plan for a fraud anomaly-detection system in which investigators can review only 500 alerts per day. Specify threshold selection, performance measures, human-review procedures, and monitoring requirements.

Multiple Choice Questions with Answers

Question 1

Which statement best describes unsupervised learning?

A. It predicts a predefined target using labeled examples.

B. It discovers structure without a predefined target variable.

C. It guarantees causal explanations for observed patterns.

D. It requires every observation to have a known class label.

Correct Answer: B. It discovers structure without a predefined target variable.

Question 2

Why is standardization often used before Euclidean-distance clustering?

A. It removes all correlations among variables.

B. It guarantees normally distributed features.

C. It prevents variables with larger numerical scales from dominating distance.

D. It automatically determines the correct number of clusters.

Correct Answer: C. It prevents variables with larger numerical scales from dominating distance.

Question 3

A silhouette value close to 1 generally indicates that an observation is:

A. Well matched to its own cluster and separated from other clusters.

B. Equally close to every cluster.

C. Definitely an anomaly.

D. Assigned to a cluster with no centroid.

Correct Answer: A. Well matched to its own cluster and separated from other clusters.

Question 4

Which clustering method can label low-density observations as noise?

A. PCA

B. Apriori

C. DBSCAN

D. Ordinary least squares

Correct Answer: C. DBSCAN

Question 5

If a rule has lift equal to 1, its observed co-occurrence is most consistent with:

A. Perfect positive dependence

B. Independence between the itemsets

C. A guaranteed causal relationship

D. A zero-support itemset

Correct Answer: B. Independence between the itemsets

Question 6

In PCA, the explained-variance ratio of a component is calculated as:

A. Its eigenvalue divided by the sum of all retained eigenvalues.

B. Its mean divided by its standard deviation.

C. Its confidence divided by its support.

D. Its distance from the nearest centroid.

Correct Answer: A. Its eigenvalue divided by the sum of all retained eigenvalues.

Question 7

For the Mahalanobis distance, the covariance matrix is important because it:

A. Converts categorical labels into binary variables.

B. Accounts for scale and correlation among variables.

C. Selects the number of association rules.

D. Removes the need for data cleaning.

Correct Answer: B. Accounts for scale and correlation among variables.

Question 8

A transaction rule has support for A of 0.20 and support for A and B together of 0.06. What is confidence for A → B?

A. 0.03

B. 0.14

C. 0.30

D. 3.33

Correct Answer: C. 0.30

Question 9

Which is the strongest evidence that a segmentation is operationally useful?

A. It has the most colorful visualization.

B. It produces different actions that improve measured business outcomes in a controlled test.

C. It uses the largest possible number of variables.

D. It is generated by the newest algorithm.

Correct Answer: B. It produces different actions that improve measured business outcomes in a controlled test.

Question 10

Why should an anomaly-detection alert usually be reviewed rather than treated as confirmed fraud?

A. Anomalies are always caused by missing data.

B. An unusual observation may be legitimate, and fraud may resemble normal behavior.

C. Anomaly algorithms cannot process numerical data.

D. Fraud has no measurable patterns.

Correct Answer: B. An unusual observation may be legitimate, and fraud may resemble normal behavior.