Multivariate and Pattern Analysis

Introduction and Context

Modern data rarely consist of one outcome explained by one predictor. Customer behaviour, clinical outcomes, industrial performance and public-sector demand are shaped by many variables operating simultaneously. Multivariate analysis provides a set of methods for examining such joint relationships, reducing dimensionality, identifying latent structure, grouping similar observations and diagnosing whether a model is trustworthy.

This chapter focuses on five connected activities: regression diagnostics, principal component analysis (PCA), factor analysis, clustering and segmentation. These methods are complementary rather than interchangeable. Regression is primarily predictive or explanatory; PCA compresses measured variables into orthogonal components; factor analysis models correlations through unobserved constructs; clustering discovers groups; and segmentation converts groups into operationally meaningful profiles.

A sound multivariate analysis is not defined by the sophistication of its algorithm. It depends on appropriate measurement, defensible assumptions, reproducible preprocessing, stability assessment and an interpretation that respects the limits of the evidence.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Diagnose influential observations, nonlinearity, heteroscedasticity and multicollinearity in regression.
2. Explain the mathematics and interpretation of PCA and factor analysis.
3. Select and evaluate clustering methods for different data structures.
4. Design a defensible segmentation workflow.
5. Assess stability, validity, interpretability and ethical risks in multivariate analysis.
6. Translate analytical patterns into actionable recommendations without overstating causality.

1. A Common Analytical Workflow

A robust workflow begins before modelling:

1. Specify the decision problem. Define the unit of analysis, target population, outcome and action that may follow.
2. Audit the data. Examine missingness, duplicates, measurement scales, extreme values, coding errors and possible leakage.
3. Prepare variables. Impute or model missing values appropriately, transform skewed variables when justified, encode categorical variables and standardise variables where scale affects distance or variance.
4. Explore structure. Use correlations, scatterplots, distributions and domain knowledge. Correlation is not evidence of causation.
5. Fit candidate models. Compare methods that reflect the measurement design and decision objective.
6. Diagnose assumptions and sensitivity. Test residual behaviour, influence, dimensionality, cluster stability and alternative preprocessing choices.
7. Validate and communicate. Use holdout or resampling methods, document decisions and present uncertainty.
8. Operationalise cautiously. Define how scores or segments will be monitored, updated and governed.

Standardisation is often important. For variable j, the z-score for observation i is

where x̄_j is the sample mean and s_j is the sample standard deviation. The transformation is dimensionless and gives each variable variance approximately equal to one. It is appropriate when variables are measured on different scales and scale should not determine the result. It is not automatically appropriate when the original units have substantive meaning or when variance itself is analytically important.

2. Regression Diagnostics

Consider the multiple linear regression model

where y is an n × 1 outcome vector, X is an n × p design matrix (usually including a column for the intercept), β is a p × 1 coefficient vector and ε is an error vector. Ordinary least squares estimates β by minimising the sum of squared residuals:

This expression requires XᵀX to be nonsingular. Exact linear dependence among predictors violates that condition; severe near-dependence makes estimates unstable even when an inverse technically exists.

The central assumptions for conventional inference are usually expressed as:

• linearity in parameters and an adequately specified conditional mean;
• independent observations, or a model that accounts for dependence;
• zero conditional mean, E(ε|X) = 0;
• constant conditional variance, Var(ε|X) = σ², for homoscedastic errors;
• no perfect multicollinearity;
• approximately normal errors mainly for small-sample t and F inference, not for unbiasedness itself.

2.1 Residual and variance diagnostics

The fitted value and residual for observation i are

Residuals should be examined against fitted values and important predictors. A curved pattern suggests nonlinearity or omitted structure; a funnel shape suggests heteroscedasticity; clusters may indicate dependence or an omitted categorical variable. A Q–Q plot can reveal heavy tails or skewness. Formal tests such as Breusch–Pagan or White tests can support, but should not replace, graphical and substantive assessment.

If heteroscedasticity is present, coefficient estimates may remain useful under suitable exogeneity, but conventional standard errors can be wrong. Heteroscedasticity-robust standard errors, transformation, weighted least squares or a model with an explicit variance structure may be considered. Robust standard errors do not repair nonlinearity, omitted variables or dependence.

2.2 Leverage, outliers and influence

An unusual outcome is not necessarily an influential observation. Leverage concerns unusual predictor values. The diagonal elements of the hat matrix

are leverage values h_ii. Large leverage means that an observation has unusual predictor values relative to the design. A residual measures outcome unusualness, while influence measures how much the fitted model changes when an observation is removed.

Cook’s distance is commonly defined as

where p is the number of fitted coefficients including the intercept and σ̂² is the residual variance estimate. Larger values indicate greater potential influence; there is no universal cutoff that substitutes for investigation. Analysts should check data entry, population membership, measurement conditions and whether the observation represents a meaningful subgroup. Deleting cases solely to improve fit is poor practice.

2.3 Multicollinearity

Multicollinearity occurs when predictors contain overlapping information. It can produce large standard errors, unstable coefficient signs and sensitivity to small data changes, even when prediction remains accurate. The variance inflation factor for predictor j is

where R_j² is obtained by regressing predictor j on all other predictors. A tolerance is 1/VIF_j. Thresholds such as VIF > 5 or VIF > 10 are heuristics, not laws; acceptable levels depend on sample size, purpose and domain.

Remedies include combining substantively redundant variables, removing one of several proxies, collecting more informative data, using ridge regression or reducing dimensions. Centring can reduce nonessential collinearity between a variable and its interaction term, but it does not remove substantive redundancy between distinct predictors. A coefficient should not be interpreted as a unique causal effect merely because it is statistically significant.

Worked Example: Diagnosing a marketing model

Suppose a retailer models monthly revenue using advertising spend, website visits and number of promotions. The fitted model has strong out-of-sample prediction, but advertising spend and website visits have VIF values of 8.4 and 7.9. Their individual coefficients change sign across bootstrap samples, while a combined advertising-and-digital-activity measure is stable.

The appropriate conclusion is not that one variable is irrelevant. Rather, the data do not reliably separate their unique contributions. The analyst should report the joint predictive contribution, assess whether the variables represent a common underlying activity, and consider regularisation or a pre-specified composite measure. If the business needs budget attribution, an experimental or quasi-experimental design may be more appropriate than interpreting unstable observational coefficients.

3. Principal Component Analysis

PCA transforms correlated variables into a smaller number of uncorrelated linear combinations called principal components. Let X be a centred n × p data matrix. The sample covariance matrix is

PCA finds a unit vector w₁ that maximises the variance of Xw₁. Subsequent vectors are orthogonal to previous vectors and maximise the remaining variance. Equivalently, if S has eigenvectors w_k and eigenvalues λ_k, then the kth component score is

and its variance is λ_k. The eigenvalues satisfy

The explained-variance ratio of component k is λ_k divided by the sum of all eigenvalues. With standardised variables, PCA is based on the correlation matrix and each variable contributes unit total variance. With unstandardised variables, variables with larger units or variance dominate.

Choosing the number of components

Common evidence includes a scree plot, cumulative explained variance, parallel analysis and reconstruction error. No single rule is universally correct. Parallel analysis compares observed eigenvalues with those generated from random data and is often more defensible than retaining components solely because their eigenvalues exceed one.

PCA is a descriptive transformation, not a causal model. Components are mathematically defined directions, and their signs are arbitrary: multiplying a component by −1 changes no substantive result. Loadings are correlations or weights depending on the convention used, so analysts must state how they are computed. Rotating retained components can improve visual interpretation, but rotated components may no longer preserve the original ordered-variance interpretation.

Example of interpretation

Assume five standardised service variables produce first-component loadings of 0.45, 0.48, 0.44, 0.46 and 0.41. The component represents broad service engagement because all variables contribute positively. A second component with positive loadings on digital use and negative loadings on branch use may represent a channel preference contrast. It should not automatically be called “customer quality”; the label must follow the observed variables and context.

PCA is useful for visualisation, noise reduction, regression inputs and clustering. However, using components estimated from the full dataset before cross-validation can leak information into a predictive evaluation. In a predictive workflow, PCA should be fitted only on the training data and then applied to validation or test data.

4. Factor Analysis

Factor analysis is a latent-variable model. It assumes that observed variables are generated partly by common unobserved factors and partly by unique influences:

where x is a p × 1 observed-variable vector, μ is its mean vector, Λ is a p × m loading matrix, f is an m × 1 common-factor vector and u is a p × 1 unique-factor vector. A common model assumes E(f) = 0, E(u) = 0, Cov(f,u) = 0, Cov(f) = Φ and Cov(u) = Ψ, where Ψ is often diagonal. Then

Here Σ is the population covariance matrix. The diagonal elements of ΛΦΛᵀ are communalities: variance explained by common factors. The diagonal elements of Ψ are unique variances, including measurement-specific variance and error.

PCA decomposes total variance and constructs optimal mathematical summaries. Factor analysis models shared covariance and explicitly separates common from unique variance. They may produce similar-looking outputs in some datasets, but they answer different questions.

Factor analysis requires evidence that relationships are sufficiently structured. The Kaiser–Meyer–Olkin measure evaluates sampling adequacy, while Bartlett’s test examines whether the correlation matrix differs from an identity matrix. These are aids, not guarantees of meaningful factors. Adequate sample size depends on communalities, number of variables, loading strength and model complexity.

Extraction methods include principal-axis factoring and maximum likelihood. Maximum likelihood supports formal comparisons and confidence procedures under stronger distributional assumptions. Rotation, such as varimax (orthogonal) or oblimin (oblique), seeks a simpler loading pattern. Oblique rotation is often more realistic because psychological, social and organisational constructs may correlate.

Factor labels are interpretive hypotheses, not discovered truths. Researchers should report the extraction method, rotation, factor-retention evidence, loadings, communalities, reliability and theoretical rationale. Confirmatory factor analysis should be used when a pre-specified measurement model is being tested; exploratory factor analysis and confirmatory factor analysis should not be treated as the same procedure on the same evidence without appropriate validation.

5. Clustering and Pattern Discovery

Clustering groups observations so that members of the same group are more similar according to a chosen representation and distance than members of different groups. It is unsupervised: there is no known target label against which a cluster is automatically correct.

5.1 Distance and representation

For standardised numerical variables, Euclidean distance between observations i and j is

The formula assumes variables are comparable after preprocessing and that a straight-line geometry is meaningful. Manhattan distance uses the sum of absolute differences and may be less sensitive to individual large coordinate differences. Gower distance can combine numerical, ordinal, binary and categorical variables, provided variable treatment and weights are explicitly specified.

Distance is not a neutral fact. Scaling, transformations, variable selection and weighting define what “similar” means. Including ten highly correlated measures of one construct can cause that construct to dominate the clustering objective.

5.2 K-means

K-means partitions n observations into K clusters by minimising within-cluster squared Euclidean distance:

where C_k is cluster k and μ_k is its centroid. The algorithm alternates between assigning each observation to its nearest centroid and recomputing centroids. It converges to a local optimum, so multiple random starts are necessary. K-means assumes numeric variables, spherical-ish clusters in the chosen space and a meaningful mean. It is sensitive to outliers and the selected K.

5.3 Hierarchical and density-based methods

Agglomerative hierarchical clustering begins with each observation as a cluster and repeatedly merges clusters according to a linkage rule. Single linkage can identify elongated chains but is vulnerable to chaining; complete linkage favours compact groups; average linkage balances pairwise distances; Ward’s method merges groups with the smallest increase in within-cluster variance and is closely related to squared-Euclidean objectives.

DBSCAN identifies dense regions separated by lower-density areas. It can find irregular shapes and label noise, but its results depend on neighbourhood radius and minimum-points parameters and may struggle when cluster densities differ substantially. Model-based clustering assumes that data arise from a mixture of probability distributions, such as Gaussian components. It supplies probabilistic membership but depends on distributional and covariance assumptions.

5.4 Evaluating clusters

Internal measures include the silhouette coefficient. For observation i, let a(i) be its mean distance to its own cluster and b(i) the smallest mean distance to another cluster. Then

Values near one indicate separation, values near zero indicate overlap, and negative values suggest possible misassignment. The average silhouette is useful for comparison but does not prove that clusters are real or useful.

External validation may compare clusters with known outcomes, future behaviour or independent variables. Stability asks whether similar clusters arise under resampling, perturbation, alternative initialisations or reasonable preprocessing choices. Practical validity asks whether the groups support different actions and whether those actions produce better outcomes.

6. Segmentation: From Clusters to Decisions

Segmentation is a managerial or operational process that translates heterogeneous populations into actionable groups. It should not end with a dendrogram or a cluster number. A defensible segmentation includes:

• a clearly defined population and unit of analysis;
• variables connected to a decision, not merely variables that are easy to collect;
• a transparent distance and modelling approach;
• profiles using variables not used to form the clusters where possible;
• segment size, stability, accessibility and expected value;
• differentiated interventions and measurable success criteria;
• monitoring for drift and unintended exclusion.

For example, a bank may identify “digital self-service”, “branch-reliant” and “high-contact complex-needs” customers. These labels should be based on observed profiles and validated with future service usage, cost and satisfaction. They should not become fixed assumptions about individuals. A segment is a statistical pattern, not a personal identity.

7. Integrated Illustrative Case: Retail Customer Experience

Background. A fictional omnichannel retailer wants to reduce service costs without lowering customer satisfaction. It has transaction, web, mobile, contact-centre and survey data for 40,000 customers.

Problem. The organisation has 25 highly correlated measures: purchase frequency, digital visits, returns, support contacts, response times, satisfaction items and channel-use indicators. Managers disagree about whether to build a regression model, reduce dimensions or create segments.

Analysis. The team first defines the customer-month as the unit and separates development data from a later validation period. Missing survey responses are investigated rather than automatically coded as dissatisfaction. Continuous variables are winsorised only under a documented policy, categorical variables are encoded appropriately, and usage variables are standardised. Highly redundant survey items are examined through factor analysis. An oblique three-factor solution suggests service responsiveness, digital convenience and perceived resolution quality, with acceptable but not perfect measurement consistency.

PCA is then used separately for a visualisation and as a sensitivity analysis. A regression model predicts next-month support cost. Residual plots reveal heteroscedasticity, and robust standard errors are used for inference. Contact frequency and complaint count show high VIF values; the team reports their joint association and tests a regularised model rather than claiming separate causal effects.

For segmentation, the team compares k-means on selected factor scores with a Gower-distance hierarchical analysis that includes categorical channel preferences. Four segments are stable across bootstrap samples, but a fifth segment is small and unstable. Silhouette values are moderate, and the most useful validation comes from different future retention and service-cost patterns.

Approach and solution. The retailer retains four segments: digitally efficient, high-value omnichannel, support-intensive and low-engagement. It does not use protected characteristics in targeting. Instead, it audits segment composition and treatment outcomes for disparate impact. Digital guidance is offered to customers who choose it, while support-intensive customers receive proactive resolution and human assistance. Segment membership is refreshed quarterly, and a champion–challenger evaluation compares targeted interventions with standard service.

Outcome. In the illustrative scenario, service cost falls while satisfaction improves for the support-intensive segment. The team reports uncertainty, documents preprocessing and cautions that the analysis is observational: the intervention evaluation, not the cluster model, is needed to establish whether the actions caused improvement.

Learning points. Dimensionality reduction can make correlated measures manageable, but it does not replace domain interpretation. Regression diagnostics reveal what the model cannot safely claim. Clusters become valuable only when stable, distinguishable and connected to ethical, measurable decisions.

8. Limitations, Governance and Contemporary Practice

Multivariate results can be unstable when samples are small relative to variables, measurements are noisy, missingness is informative or the population changes. Multiple modelling choices create researcher degrees of freedom. Analysts should preregister key decisions where feasible, preserve a holdout set, use bootstrap or subsample stability analysis, and report sensitivity to scaling, variable selection, distance and cluster number.

Fairness requires attention to both inputs and consequences. Removing a protected attribute does not remove its proxies. Segments can reproduce historical inequities, and automated scores can influence access to credit, health services or employment. Governance should include purpose limitation, data minimisation, access control, documentation, human review, contestability and monitoring for disparate error or impact. Privacy-preserving aggregation does not guarantee anonymity when many variables enable re-identification.

AI-assisted pattern discovery can suggest transformations, clusters or latent representations, but automation does not establish construct validity or causality. Embeddings and nonlinear methods may improve prediction while reducing interpretability. Their use should be justified by the decision context, accompanied by stability tests and compared with simpler baselines. Reproducible pipelines should record data versions, random seeds, software environments, preprocessing parameters and model outputs.

Best-Practice Checklist

Before presenting a multivariate result, ask:

1. Is the unit of analysis and decision purpose explicit?
2. Were measurement scales, missingness and leakage assessed?
3. Was preprocessing fitted without using future or test information?
4. Are regression assumptions and influential cases documented?
5. Is multicollinearity distinguished from lack of predictive value?
6. Was PCA or factor analysis selected for the correct conceptual purpose?
7. Is the distance metric appropriate to the data types?
8. Are cluster number, stability and external usefulness evaluated?
9. Are labels descriptive rather than unsupported causal or moral judgements?
10. Are uncertainty, fairness, privacy and monitoring addressed?

Chapter Summary

Multivariate and pattern analysis studies relationships that cannot be understood reliably one variable at a time. Regression diagnostics assess residual behaviour, influence, heteroscedasticity and multicollinearity so that coefficients and predictions are not overinterpreted. PCA compresses total variance into orthogonal components, whereas factor analysis models shared covariance through latent constructs and unique variance. Clustering depends critically on representation, scaling, distance and algorithmic assumptions; internal metrics alone cannot establish that a cluster is meaningful. Segmentation adds decision relevance, validation and governance to pattern discovery. Across all methods, stability, interpretability, ethical use and out-of-sample validation are as important as computational fit.

Key Terms

Multivariate analysis: Analysis involving multiple variables jointly.

Residual: The observed outcome minus its fitted value.

Leverage: An observation’s unusualness in predictor space.

Influence: The extent to which an observation changes fitted results.

Cook’s distance: A diagnostic measure of an observation’s influence on regression estimates.

Multicollinearity: Strong linear dependence among predictors.

VIF: Variance inflation factor measuring predictor redundancy.

Heteroscedasticity: Nonconstant conditional error variance.

Principal component: A variance-maximising linear combination of variables subject to orthogonality constraints.

Eigenvalue: The variance associated with a principal component in PCA.

Loading: A coefficient or correlation describing a variable’s relation to a component or factor.

Communality: Variance in an observed variable explained by common factors.

Unique variance: Variable-specific variance and measurement error not explained by common factors.

Rotation: A transformation used to improve factor or component interpretability.

Distance metric: A rule for quantifying dissimilarity between observations.

Centroid: The mean location of observations in a cluster.

Silhouette coefficient: An observation-level measure of within-cluster cohesion and between-cluster separation.

Cluster stability: Consistency of cluster assignments under resampling or reasonable analytical changes.

Segmentation: The creation of actionable groups from population heterogeneity.

Concept drift: Change over time in the data-generating process or population structure.

Review Questions

Short Answer Questions

1. What is the difference between leverage and an outlier in the response variable?
2. Why can multicollinearity harm coefficient interpretation even when prediction is accurate?
3. State the main conceptual difference between PCA and factor analysis.
4. Why must scaling be considered before distance-based clustering?
5. What does a negative silhouette coefficient suggest?

Long Answer Questions

1. Explain the assumptions of multiple linear regression and describe graphical and numerical diagnostics for assessing them.
2. Derive the interpretation of PCA eigenvalues and discuss criteria for selecting the number of components.
3. Compare exploratory factor analysis with PCA, including their variance models, assumptions and uses.
4. Compare k-means, hierarchical clustering and DBSCAN with respect to geometry, parameters and limitations.
5. Explain how a statistically derived segmentation can be converted into an ethical and measurable intervention programme.

Analytical / Application Questions

1. A regression model has stable predictions but coefficient signs change across bootstrap samples, and two predictors have VIF values above 8. Diagnose the problem and recommend an analysis plan.
2. Design a clustering study for mixed numerical, ordinal and categorical customer data. Specify preprocessing, distance measure, validation and stability procedures.
3. A healthcare segmentation produces a small cluster with high readmission rates. Explain how you would determine whether it is a real, actionable pattern rather than an artefact, while addressing fairness and privacy.

Multiple Choice Questions with Answers

Question 1

Which diagnostic is most directly designed to assess predictor redundancy in a multiple regression model?

A. Variance inflation factor

B. Q–Q plot

C. Cook’s distance

D. Silhouette coefficient

Correct Answer: A. Variance inflation factor

Question 2

In PCA based on the covariance matrix, what does an eigenvalue represent?

A. The number of observations in a component

B. The variance associated with a component

C. The residual variance of the regression model

D. The proportion of missing values

Correct Answer: B. The variance associated with a component

Question 3

Which statement best distinguishes factor analysis from PCA?

A. Factor analysis models shared covariance and unique variance

B. PCA requires a dependent variable and factor analysis does not

C. PCA is always causal while factor analysis is descriptive

D. Factor analysis cannot use correlated factors

Correct Answer: A. Factor analysis models shared covariance and unique variance

Question 4

A funnel-shaped residual-versus-fitted plot most commonly suggests:

A. Perfect multicollinearity

B. Heteroscedasticity

C. A high silhouette score

D. An orthogonal rotation

Correct Answer: B. Heteroscedasticity

Question 5

Which clustering method is particularly associated with identifying dense regions and labelling noise?

A. K-means

B. DBSCAN

C. Ward regression

D. Principal-axis factoring

Correct Answer: B. DBSCAN

Question 6

For an observation, a silhouette coefficient close to one generally indicates:

A. Strong membership in a well-separated cluster

B. Exact multicollinearity

C. A high regression leverage value

D. A missing factor loading

Correct Answer: A. Strong membership in a well-separated cluster

Question 7

If variables are measured in dollars, counts and percentages, why might standardisation be used before k-means clustering?

A. To make every observation identical

B. To prevent measurement scale alone from dominating distance

C. To guarantee the correct number of clusters

D. To remove all outliers

Correct Answer: B. To prevent measurement scale alone from dominating distance

Question 8

What is the principal purpose of Cook’s distance?

A. To measure an observation’s potential influence on fitted regression results

B. To determine the number of PCA components

C. To estimate a factor’s reliability

D. To calculate categorical distance

Correct Answer: A. To measure an observation’s potential influence on fitted regression results

Question 9

In a regression model, robust standard errors primarily address:

A. Heteroscedasticity-related standard-error distortion

B. Omitted-variable bias in all circumstances

C. Nonlinear relationships automatically

D. Perfect prediction in clustering

Correct Answer: A. Heteroscedasticity-related standard-error distortion

Question 10

Which practice best protects predictive evaluation when PCA is used before regression?

A. Fit PCA on the complete dataset before splitting it

B. Fit PCA on training data and apply the transformation to validation data

C. Choose components using the test outcome

D. Avoid reporting the transformation

Correct Answer: B. Fit PCA on training data and apply the transformation to validation data