Unsupervised Learning, Segmentation and Recommendation

Introduction

Many business decisions depend on discovering structure that has not been explicitly labelled. A retailer may know which customers purchased, but not which customer groups exist. A bank may observe transactions, but not which patterns represent unusual behaviour. An online platform may know what users viewed, but not what products should be recommended next. These are typical applications of unsupervised learning: methods that identify patterns, groups, dimensions or relationships in data without a pre-specified target outcome.

This chapter examines five closely related families of methods: clustering, dimensionality reduction, anomaly detection, association analysis and recommender systems. Their common purpose is discovery, but their outputs differ. Clustering assigns observations to groups; dimensionality reduction creates compact representations; anomaly detection identifies unusual observations; association analysis discovers co-occurring events; and recommendation systems rank items for individuals or groups.

Unsupervised methods are especially valuable in business analytics because labels are often expensive, delayed or incomplete. However, the absence of labels does not mean that analysis is assumption-free. Results depend on how variables are measured, how similarity is defined, how many groups are selected, and how discovered patterns are translated into business action.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain the purpose, assumptions and business uses of unsupervised learning.
2. Prepare data appropriately for clustering, anomaly detection and recommendation.
3. Apply and interpret k-means clustering, hierarchical clustering and density-based methods.
4. Explain principal component analysis as a dimensionality-reduction technique.
5. Distinguish point, contextual and collective anomalies.
6. Interpret support, confidence and lift in association analysis.
7. Compare content-based, collaborative and hybrid recommender systems.
8. Evaluate technical quality, business value, fairness, privacy and operational risks.

1. The Logic of Unsupervised Learning

In supervised learning, a model estimates a relationship between predictors and a known target. In unsupervised learning, the data typically consist of observations X = {x₁, x₂, …, x_n} without a labelled response. The model searches for regularities according to an objective function or similarity criterion.

The central question is not simply “What does the algorithm find?” but “What structure is useful and credible for this decision?” A cluster produced from customer age and income may be statistically coherent but commercially weak if it does not lead to different offers or service strategies. Conversely, a modestly separated cluster may be valuable if it identifies customers at risk of leaving.

Important terminology includes:

• Observation or instance: one customer, transaction, product or time period.
• Feature: a measured attribute used by the analysis.
• Similarity or distance: a numerical representation of how alike two observations are.
• Cluster: a group whose members are more similar under the chosen criterion than members of other groups.
• Representation: a transformed description of an observation, often with fewer dimensions.
• Unlabelled data: data without a known target class or outcome.

A reliable workflow is usually iterative:

1. Define the business decision and unit of analysis.
2. Select and validate relevant data.
3. Clean, transform and scale variables.
4. Fit one or more unsupervised methods.
5. Evaluate stability, interpretability and usefulness.
6. Validate findings with domain experts or subsequent outcomes.
7. Deploy actions, monitor drift and reassess regularly.

2. Data Preparation and Similarity

The unit of analysis must match the question. For customer segmentation, one row may represent a customer over a specified period. For fraud analysis, one row may represent a transaction. Mixing transaction-level and customer-level records can produce clusters that reflect purchasing frequency rather than customer characteristics.

2.1 Variable selection and transformation

Variables should be selected because they represent a meaningful behavioural, operational or commercial concept. Redundant variables can give one concept excessive influence. For example, total spend, average basket value and annual spend may jointly dominate a distance calculation.

Highly skewed monetary variables are often transformed using a logarithm. A common transformation is x' = log(1 + x), where x is non-negative. The constant 1 permits zero values. The transformation reduces the influence of very large observations, although it does not remove the need to investigate outliers.

Categorical variables require care. Simple integer coding can falsely imply that categories have an ordered distance. One-hot encoding is appropriate for many nominal variables, but it may produce a high-dimensional representation. Mixed-type data may require specialized distances or algorithms.

2.2 Scaling

Distance-based algorithms are sensitive to measurement units. If annual income is measured in dollars and purchase frequency is measured in counts, income may dominate unless variables are scaled. Standardization uses the sample mean and standard deviation:

Here, x_ij is the value of feature j for observation i, μ_j is the feature mean and σ_j is its standard deviation. The transformed variable is dimensionless. Standardization is most appropriate when a feature is reasonably represented by its mean and standard deviation and when equal variance scaling is substantively defensible. Robust scaling based on medians and interquartile ranges may be preferable when extreme values are common.

The Euclidean distance between two p-dimensional standardized observations is:

It assumes that variables are numeric and that squared deviations are a meaningful measure of dissimilarity. Other choices include Manhattan distance, cosine distance and Gower distance for mixed data. The distance measure is part of the model, not a neutral technical detail.

3. Clustering for Segmentation

Clustering seeks groups with high within-group similarity and lower similarity between groups. It is used for customer segmentation, product portfolios, branch analysis, supplier classification and operational profiling.

3.1 k-means clustering

The k-means algorithm partitions n observations into k clusters. It assigns each observation to the nearest centroid, then recalculates each centroid as the mean of the observations assigned to it. The objective is to minimize the within-cluster sum of squares:

Here, C_r is cluster r, μ_r is its centroid, and the norm is normally Euclidean distance. The algorithm assumes that means are meaningful representatives, that variables are numeric, and that compact, approximately spherical clusters are suitable. It is sensitive to scaling, initialization and outliers.

The practical procedure is:

1. Choose k and initialize k centroids.
2. Assign each observation to its nearest centroid.
3. Recalculate each centroid.
4. Repeat assignment and recalculation until assignments or objective improvement stabilize.

Because initialization can affect the result, multiple initializations are usually used. The result should not be interpreted as proof that exactly k natural groups exist.

Worked example: interpreting a two-variable segmentation

Suppose standardized customers are described by purchase frequency and average order value. A fitted two-cluster model has centroids:

A customer with scores (0.40, 0.20) has squared Euclidean distance to A of (0.40-0.80)² + (0.20-0.70)² = 0.16 + 0.25 = 0.41. Its distance to B is (0.40+0.60)² + (0.20+0.50)² = 1.00 + 0.49 = 1.49. The customer is assigned to A because 0.41 is smaller than 1.49. This assignment does not establish that the customer will respond to a particular promotion; it identifies similarity under the selected variables.

3.2 Choosing the number of clusters

The elbow method plots the within-cluster sum of squares against k. A bend may indicate diminishing improvement as additional clusters are added. The silhouette coefficient compares an observation’s average distance to its own cluster with its average distance to the nearest alternative cluster:

Here, a(i) is the average distance from observation i to members of its own cluster, and b(i) is the smallest average distance to another cluster. Values near 1 indicate good separation, values near 0 indicate overlap, and negative values may indicate possible misassignment. The measure assumes that the chosen distance reflects meaningful similarity.

Statistical criteria should be combined with business interpretability, stability across samples and actionability. A five-cluster solution is not automatically better than a three-cluster solution merely because it reduces within-cluster variation.

3.3 Hierarchical clustering

Hierarchical clustering creates a nested sequence of groups represented by a dendrogram. In agglomerative clustering, each observation begins as its own cluster; the closest clusters are repeatedly merged. The definition of distance between clusters is determined by the linkage rule:

• Single linkage: minimum pairwise distance; can produce chaining.
• Complete linkage: maximum pairwise distance; favours compact groups.
• Average linkage: average pairwise distance.
• Ward’s method: merges clusters with the smallest increase in within-cluster squared error, usually for numeric Euclidean data.

Hierarchical methods are useful when analysts want to inspect solutions at several levels. They can be computationally demanding for very large datasets and are sensitive to distance and linkage choices.

3.4 Density-based clustering

Density-based methods, such as DBSCAN, define clusters as dense regions separated by sparse regions. DBSCAN uses a neighbourhood radius and a minimum number of observations. It can identify irregularly shaped groups and label isolated observations as noise. It is useful for geographic activity, network behaviour and operational events. However, one global density threshold may perform poorly when clusters have very different densities, and high-dimensional distances can become less informative.

4. Dimensionality Reduction and Principal Component Analysis

High-dimensional data may contain correlated variables, noise and difficult-to-visualize structure. Dimensionality reduction creates a lower-dimensional representation while attempting to preserve important information.

4.1 Principal component analysis

Principal component analysis (PCA) transforms p correlated numeric variables into orthogonal components. The first component is the linear combination with maximum variance; each subsequent component maximizes remaining variance subject to being orthogonal to previous components.

For a centred data matrix, PCA is based on the covariance matrix. If Σ is the covariance matrix, a component direction w satisfies:

Here, λ is the eigenvalue associated with eigenvector w. The eigenvalue represents variance in the component direction. Eigenvectors are conventionally normalized so that wᵀw = 1. PCA assumes that linear combinations and variance are meaningful representations of information. It is not designed specifically to preserve class separation or causal structure.

If features have different scales, PCA is commonly performed on standardized variables. The proportion of variance explained by component r is:

A reduced representation may retain the first m components when they provide sufficient interpretability or predictive utility. Analysts should not select components solely by a fixed threshold; the decision should consider reconstruction, visualization, downstream performance and meaning.

PCA can support customer visualization, compression of product attributes and removal of multicollinearity. Its limitations include sensitivity to scaling and outliers, reduced interpretability of components, and the possibility that low-variance business signals are discarded.

5. Anomaly Detection

An anomaly is an observation that differs substantially from an expected pattern. Applications include payment fraud, cyber-security, equipment monitoring, claims review and quality control.

Three useful categories are:

• Point anomaly: one observation is unusual relative to the rest, such as an extremely large transaction.
• Contextual anomaly: an observation is unusual in a particular context, such as high electricity use at a normally closed branch during the night.
• Collective anomaly: a sequence or group is unusual even though individual events may appear normal.

5.1 Methods

Distance-based methods flag observations far from a centre or their neighbours. They are simple but sensitive to scale and high dimensionality. Clustering can identify small or distant groups, though a small legitimate segment may be incorrectly labelled anomalous.

Isolation Forest repeatedly partitions data using random feature thresholds. Observations that are isolated in fewer splits tend to receive higher anomaly scores. The method is useful for large datasets and does not require a distance matrix, but the resulting score is relative to the sample and should not automatically be interpreted as a probability of fraud.

Local Outlier Factor compares the local density of an observation with the density of its neighbours. It can detect local anomalies, although the choice of neighbourhood size matters.

Autoencoders, a type of neural network, learn to reconstruct input data. A high reconstruction error may indicate unusual structure. They require careful training, sufficient data and controls against learning the anomalies themselves.

Anomaly detection is usually a prioritization system rather than an automatic accusation system. A threshold should reflect investigation capacity, the cost of false positives, the cost of missed cases and the consequences of intervention. Precision, recall, alert volume, review time and confirmed-case yield should be monitored when labels become available.

6. Association Analysis

Association analysis discovers items or events that occur together. It is commonly used in market-basket analysis, cross-selling, website navigation and service-bundle design.

Let X and Y be itemsets. Support measures the proportion of transactions containing an itemset:

For a rule X → Y, confidence is:

Confidence estimates the proportion of transactions containing X that also contain Y. It is not a causal probability that purchasing X causes Y.

Lift compares confidence with the baseline prevalence of Y:

Lift greater than 1 indicates positive co-occurrence relative to the baseline; lift equal to 1 indicates independence under the measure; lift below 1 indicates less co-occurrence than expected. These interpretations assume that the transaction definition and item representation are appropriate.

Worked example

Suppose 1,000 baskets are observed. Bread appears in 400 baskets, butter in 250, and both in 150. Then:

• support(bread → butter) = 150/1,000 = 0.15, or 15%.
• confidence(bread → butter) = 150/400 = 0.375, or 37.5%.
• lift = 0.375/0.25 = 1.5.

Thus, baskets containing bread contain butter 1.5 times as often as the overall butter rate. This is an association, not evidence that bread purchases cause butter purchases. Analysts should also consider seasonality, promotions, store layout and product availability.

The Apriori principle states that if an itemset is infrequent, every larger itemset containing it must also be infrequent. This permits pruning of candidate itemsets. Frequent-pattern methods such as FP-Growth can discover frequent sets without generating all candidates in the same manner. Very low support thresholds can produce enormous numbers of rules, so business relevance and statistical validation are essential.

7. Recommender Systems

A recommender system estimates which items may be relevant to a user and ranks them. It may recommend products, media, jobs, learning resources or financial services. Recommendation is a decision-support mechanism, not merely a prediction problem: ranking, exposure, diversity, timing and user experience all matter.

7.1 Content-based recommendation

Content-based systems represent users by item attributes they have previously engaged with and recommend items with similar attributes. They are relatively interpretable and can recommend new items if metadata are available. However, they may over-specialize, repeatedly recommending similar content and limiting discovery.

7.2 Collaborative filtering

Collaborative filtering uses patterns across users and items. In user-based filtering, similar users influence recommendations. In item-based filtering, items receiving similar interaction patterns are related. Similarity may be calculated using cosine similarity:

The vectors must be non-zero; the measure focuses on orientation rather than magnitude. Explicit ratings, purchases, clicks, views and dwell time are possible signals, but implicit feedback is ambiguous: no purchase may mean lack of interest, lack of exposure or lack of need.

Matrix factorization represents an interaction matrix R approximately as:

where P contains latent user factors and Q contains latent item factors. A common regularized objective for observed interactions is:

Here, Ω is the set of observed user-item pairs, r_ui is an observed rating or transformed interaction, p_u and q_i are latent vectors, and λ controls regularization. This formulation assumes that observed interactions are informative and that latent factors can capture useful preference structure. It does not automatically solve exposure bias or popularity bias.

7.3 Hybrid and knowledge-based systems

Hybrid systems combine content and collaborative signals, often improving performance for new users or new items. Knowledge-based recommenders use explicit requirements, constraints or domain rules. They are valuable for expensive purchases, such as insurance, industrial equipment or university programmes, where a user’s needs cannot be inferred from a few clicks.

7.4 Evaluation

Offline evaluation should use time-aware train-test splits when future recommendation is being simulated. Randomly splitting interactions can leak future information into training data. Common ranking measures include precision at k, recall at k, mean reciprocal rank and normalized discounted cumulative gain. For a ranked list, discounted cumulative gain can be written as:

Here, rel_i is the relevance value at rank i. NDCG divides DCG by the ideal DCG for the same user and cutoff, producing a value between 0 and 1 when relevance values are non-negative. Offline metrics should be supplemented by online experiments, business outcomes, coverage, diversity, novelty and fairness.

The cold-start problem arises when a new user or item has insufficient interaction history. Onboarding questions, content metadata, contextual signals and exploration policies can help. A system optimized only for clicks may promote sensational or low-quality items, so objective design must reflect long-term value.

8. Integrated Case: Illustrative Retail Personalization Programme

Background

Consider an illustrative national retailer with stores and a mobile application. Management wants to improve customer retention, reduce irrelevant promotions and increase complementary-product sales.

Problem

The retailer has millions of transaction records, product attributes, campaign responses and app interactions. It has no reliable pre-existing customer segments. It also wants to identify suspicious account activity and recommend products without over-targeting vulnerable customers.

Analysis and approach

The analytics team first defined the customer-month as the principal segmentation unit. It created recency, frequency and monetary-value variables, applied a logarithmic transformation to highly skewed monetary values, and standardized numeric features. A three-cluster solution was compared with four- and five-cluster alternatives using silhouette scores, stability checks and managerial interpretation.

The selected solution identified: occasional low-value shoppers, frequent routine shoppers and high-value multi-category shoppers. These labels were descriptive rather than permanent identities. Membership was recalculated monthly, and campaign teams were instructed not to treat a segment as a proxy for protected characteristics.

PCA was used to visualize purchasing dimensions and investigate redundancy among product-category variables. DBSCAN was used on geolocation and time features to identify unusual account access patterns. Alerts were reviewed by a fraud team; the algorithm did not automatically suspend accounts.

Association analysis found that several rules had high confidence but low incremental value because the consequent item was already popular. The team therefore prioritized rules with adequate support, meaningful lift and inventory availability. Recommendations combined item-based collaborative filtering with product metadata so that new products could enter the system.

Outcome

In an A/B test, the retailer compared the personalized programme with its existing campaign strategy. Evaluation included incremental revenue, margin, unsubscribe rate, recommendation coverage, fraud-review yield and performance across customer groups. The team found that recommendations increased engagement in some categories but reduced diversity when optimized only for clicks. The production design therefore added category-diversity constraints and a frequency cap on promotions.

Learning points

1. A cluster is useful only when it supports differentiated and responsible action.
2. Association rules require baseline comparison; high confidence alone can mislead.
3. Anomaly alerts should support investigation rather than replace judgement.
4. Recommendation evaluation must include long-term and fairness-related outcomes.
5. Deployment requires monitoring because behaviour, inventory and product mix change.

9. Governance, Ethics and Professional Practice

Unsupervised models can reproduce bias even without explicitly using protected attributes. Proxy variables, unequal data coverage and different exposure patterns may create unfair segments or recommendations. A segment associated with lower spending may reflect limited access, not lower potential or motivation.

Privacy principles require purpose limitation, data minimization, appropriate retention and lawful processing. Re-identification risk can arise when multiple seemingly harmless variables are combined. Organisations should document data provenance, feature definitions, model versions, thresholds and decision rights.

Important controls include:

• human review for high-impact actions;
• audit logs for alerts and recommendations;
• testing for disparate error or exposure patterns;
• explanation appropriate to the decision context;
• access controls and encryption;
• monitoring for drift, feedback loops and changing base rates;
• clear procedures for complaints and correction.

Feedback loops are particularly important. If a recommender shows only popular products, those products receive more interactions and appear even more popular. If a fraud model investigates one group more frequently, the organisation may observe more confirmed cases in that group partly because of unequal scrutiny. Analysts must distinguish observed outcomes from underlying behaviour.

10. Contemporary Developments and Best Practices

Modern systems increasingly combine representation learning, embeddings, graph methods and large-scale retrieval with traditional analytics. Embeddings can represent users, products, text or sequences in a vector space, enabling semantic similarity. Graph-based approaches model relationships among customers, accounts, devices and transactions, which can be useful in fraud and recommendation. These methods offer power but may reduce transparency and increase infrastructure, privacy and monitoring requirements.

Best practice is to begin with a transparent baseline, establish a business and ethical objective, and add complexity only when it delivers validated value. Analysts should separate exploratory discovery from automated decision-making, use time-based validation where appropriate, maintain a holdout or control group, and report uncertainty and limitations. Successful deployment is not measured only by a technical score; it requires operational adoption, measurable benefit and responsible treatment of affected people.

Chapter Summary

Unsupervised learning discovers structure without a labelled target. Its main business applications include segmentation, compact representation, anomaly prioritization, product-bundle discovery and personalized ranking. Clustering depends on the chosen variables, scaling, distance and algorithm; k-means is effective for compact numeric groups but requires a selected k and is sensitive to outliers. Hierarchical and density-based methods provide alternative structures and assumptions.

PCA reduces correlated numeric variables into orthogonal components ordered by variance, but components require interpretation and may not preserve business-relevant signals. Anomaly detection identifies unusual points, contexts or collectives and should generally support investigation rather than make unsupported accusations. Association analysis uses support, confidence and lift to describe co-occurrence, not causality. Recommender systems may be content-based, collaborative, hybrid or knowledge-based; they should be evaluated with ranking, business, coverage, diversity and fairness measures.

Across all methods, data preparation, validation, interpretability, governance and monitoring are as important as algorithm selection. The central professional skill is translating discovered patterns into useful, measurable and responsible business action.

Key Terms

Unsupervised learning: Learning from data without a specified target label.

Clustering: Grouping observations according to a similarity or distance criterion.

Segmentation: Dividing a population into meaningful groups for analysis or action.

Centroid: The mean position representing a cluster in k-means.

Euclidean distance: Straight-line distance calculated from squared differences across numeric features.

Silhouette coefficient: A measure comparing within-cluster cohesion with separation from another cluster.

Hierarchical clustering: Clustering that creates nested groups represented by a dendrogram.

Density-based clustering: Clustering based on dense regions separated by sparse regions.

Dimensionality reduction: Representing data with fewer variables or coordinates.

Principal component analysis: A linear transformation that orders orthogonal components by variance.

Anomaly: An observation or pattern that differs from expected behaviour.

Point anomaly: An individual observation that is unusual relative to the dataset.

Association rule: A directional co-occurrence statement such as X → Y.

Support: The proportion of transactions containing an itemset.

Confidence: The proportion of X-containing transactions that also contain Y.

Lift: Confidence divided by the baseline support of the consequent.

Collaborative filtering: Recommendation based on similarities in user-item interactions.

Cold start: Difficulty recommending for new users or items with little history.

Exposure bias: Distortion caused when observed interactions depend on what users were shown.

Feedback loop: A process in which model outputs influence the future data used by the model.

Review Questions

Short Answer Questions

1. Why must variables be scaled before many distance-based clustering methods are applied?
2. What does a silhouette coefficient near zero suggest?
3. Distinguish a point anomaly from a contextual anomaly.
4. Define support, confidence and lift in association analysis.
5. What is the cold-start problem in recommender systems?

Long Answer Questions

1. Explain the k-means algorithm, its objective function, assumptions and major limitations.
2. Compare PCA with clustering as tools for understanding customer data.
3. Discuss how anomaly detection should be integrated into a fraud-investigation process.
4. Compare content-based, collaborative and hybrid recommender systems.
5. Explain why technical performance alone is insufficient for evaluating a business recommender.

Analytical / Application Questions

1. A retailer has income, transaction frequency and annual spend measured in very different units. Design a preprocessing and clustering plan, including transformation, scaling, validation and managerial interpretation.
2. In 2,000 baskets, coffee appears in 600, biscuits in 500 and both in 240. Calculate support for the rule coffee → biscuits, its confidence and its lift. Interpret the result without claiming causation.
3. A fraud model generates many alerts for one customer group, but investigation confirms only a small proportion. Propose an analytical and governance review that considers base rates, exposure, false positives, threshold selection and fairness.

Multiple Choice Questions with Answers

Question 1

Which statement best describes unsupervised learning?

A. It learns from data with a known target label.

B. It discovers structure in data without a specified target label.

C. It guarantees causal explanations for observed patterns.

D. It can only be used with image data.

Correct Answer: B. It discovers structure in data without a specified target label.

Question 2

What is the main purpose of standardizing a numeric feature before Euclidean-distance clustering?

A. To make all observations identical

B. To remove all correlations

C. To prevent measurement units from giving one feature disproportionate influence

D. To guarantee the correct number of clusters

Correct Answer: C. To prevent measurement units from giving one feature disproportionate influence

Question 3

In k-means clustering, what does a centroid represent?

A. The median of every feature in the entire dataset

B. The mean position of observations assigned to a cluster

C. The most anomalous observation

D. The distance between two clusters

Correct Answer: B. The mean position of observations assigned to a cluster

Question 4

A silhouette coefficient close to zero generally indicates that an observation is:

A. Perfectly isolated from every other observation

B. Equally likely to be in any possible cluster by definition

C. Near the boundary between its cluster and another cluster

D. Guaranteed to be an outlier

Correct Answer: C. Near the boundary between its cluster and another cluster

Question 5

PCA orders components primarily according to:

A. Increasing numbers of missing values

B. Explained variance

C. Customer profitability

D. Classification accuracy

Correct Answer: B. Explained variance

Question 6

If support(X ∪ Y) is 0.15 and support(X) is 0.40, what is confidence(X → Y)?

A. 0.0375

B. 0.15

C. 0.375

D. 2.67

Correct Answer: C. 0.375

Question 7

A lift of 1.5 for X → Y indicates that:

A. X causes Y with probability 1.5

B. Y occurs 1.5 times as often among X transactions as its overall rate

C. The rule has 150% support

D. X and Y never occur together

Correct Answer: B. Y occurs 1.5 times as often among X transactions as its overall rate

Question 8

Which recommender approach uses similarities among users or items based on interaction patterns?

A. Collaborative filtering

B. Random sampling

C. Hierarchical scaling

D. Principal component rotation only

Correct Answer: A. Collaborative filtering

Question 9

Which is an example of a contextual anomaly?

A. A transaction larger than every other transaction in all contexts

B. A normal electricity reading recorded at a normally closed branch at midnight

C. A common product appearing in a basket

D. A user receiving a recommendation

Correct Answer: B. A normal electricity reading recorded at a normally closed branch at midnight

Question 10

Why can a random train-test split be problematic for recommendation interactions?

A. It always removes all data from the test set

B. It may allow future interactions to influence training and create temporal leakage

C. It prevents the calculation of precision

D. It guarantees poor recommendations

Correct Answer: B. It may allow future interactions to influence training and create temporal leakage