Multivariate and Model-Based Sampling Distributions

Introduction

Quality characteristics rarely occur in isolation. A manufactured component may be evaluated simultaneously for diameter, hardness, surface roughness and tensile strength; a service process may be assessed through waiting time, error rate and customer satisfaction. These characteristics are often correlated. Analysing each variable separately can therefore conceal important information, inflate the probability of false alarms, or lead to contradictory improvement decisions.

Multivariate sampling distributions provide the theoretical basis for estimating and monitoring several correlated characteristics jointly. This chapter develops the sampling behaviour of the sample mean vector, covariance matrix, correlation coefficients and regression estimators. It then introduces the multivariate normal model, Hotelling’s T-squared statistic and model-diagnostic methods relevant to quality engineering and total quality management.

The emphasis is not merely on calculation. At postgraduate level, the practitioner must also judge assumptions, distinguish estimation from prediction, understand the effects of correlation, and recognise when a model-based conclusion is not trustworthy.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Explain why covariance and correlation affect multivariate quality decisions.
2. Derive and interpret the sampling distribution of a sample mean vector.
3. Describe the role of the Wishart distribution in covariance estimation.
4. Evaluate sampling distributions of covariance, correlation and regression estimators.
5. Apply multivariate normal theory and Hotelling’s T-squared statistic.
6. Diagnose model inadequacy using residuals, leverage, influence and multivariate distance.
7. Communicate statistical evidence responsibly in quality-improvement settings.

1. Multivariate Quality Data and Notation

Let each inspected unit provide p measurements. The observation for unit i is represented by the p × 1 vector

x_i = (X_i1, X_i2, …, X_ip)^T.

A random quality vector is denoted by X, with mean vector μ and covariance matrix Σ:

The diagonal elements of Σ are variances, while the off-diagonal elements are covariances. If σ_jk denotes the covariance between characteristics j and k, then the correlation is

where σ_j and σ_k are the positive standard deviations. Correlation is dimensionless; covariance has the product of the measurement units of the two variables.

For n independent observations, the sample mean vector is

and the unbiased sample covariance matrix is

The divisor n − 1 makes S unbiased for Σ under independent, identically distributed sampling with finite second moments. The matrix S must be symmetric and positive semidefinite; it is positive definite when the observations contain sufficient independent variation.

2. The Multivariate Normal Distribution

The multivariate normal distribution is central because it provides exact sampling distributions for several important estimators and is often a useful approximation after suitable transformation. A p-dimensional vector X follows a multivariate normal distribution with mean μ and covariance Σ if its density is

where Σ is positive definite and |Σ| is its determinant. The quadratic form

is the squared Mahalanobis distance from x to μ. It accounts for both scale and correlation. A point that appears unusual on one variable may be ordinary after the covariance structure is considered, whereas a point that is not extreme in any single variable may be unusual jointly.

Important properties include:

• Every linear combination a^T X is normally distributed for a fixed vector a.
• Marginal distributions and conditional distributions are normal.
• Under multivariate normality, the sample mean vector and sample covariance matrix are statistically independent.
• Zero covariance implies independence, a property not generally true outside the normal family.

The model is a working assumption, not a guarantee. Strong skewness, heavy tails, mixtures of production sources and time dependence can invalidate normal-theory procedures.

3. Sampling Distribution of the Sample Mean Vector

Assume X_1, …, X_n are independent observations from a p-variate population with mean μ and covariance Σ. If the population is multivariate normal, then

This means that the expected value of x̄ is μ and its covariance matrix is Σ/n. Thus, each component has variance σ_j²/n, while the covariance between two components of x̄ is σ_jk/n.

If Σ is known, the standardized quadratic statistic is chi-squared:

The result is exact under multivariate normal sampling. If normality is questionable but n is sufficiently large and observations are independent, a multivariate central limit theorem may justify an approximate normal distribution. The required sample size depends on tail behaviour, dimension, dependence and the desired accuracy; there is no universal threshold.

Confidence region for the mean vector with known covariance

A 100(1 − α)% confidence region for μ is

where χ²_{p,1−α} is the 1 − α quantile of a chi-squared distribution with p degrees of freedom. This region is an ellipsoid, not a collection of independent intervals. Its orientation reflects correlation.

Unknown covariance: Hotelling’s T-squared statistic

When Σ is unknown and replaced by S, define

where μ₀ is a specified target or reference mean. If the observations are multivariate normal, n > p, and S is nonsingular, then under H₀: μ = μ₀,

where F_{p,n−p} is an F distribution. Consequently, the test rejects at significance level α when

The statistic measures joint departure from the target in covariance-adjusted units. It should not be interpreted as the sum of separate univariate t-tests, because correlation and multiplicity are handled differently.

4. Sampling Distribution of the Covariance Matrix

For independent multivariate normal observations, the scatter matrix

has a Wishart distribution:

The Wishart distribution is the multivariate analogue of the chi-squared distribution. Its first parameter is the scale matrix Σ and its second parameter is the degrees of freedom ν = n − 1. It describes the random variation of the entire covariance matrix, not just one variance.

For an unbiased sample covariance matrix,

provided the observations are independent and have finite second moments. Unbiasedness does not mean that S is close to Σ for every sample. Sampling variability can be substantial when n is small relative to p.

For a normal population, a useful marginal result is

where S_jj is the sample variance of variable j. Hence

The covariance between elements of S has, under normality, the form

This formula shows why covariance estimates are themselves correlated. It also demonstrates why treating several estimated covariances as independent can produce incorrect uncertainty assessments.

For a sample covariance S_jk, the approximate standard error is obtained by taking the square root of its variance. Setting l = j and m = k gives

In practice, unknown population quantities are replaced by sample estimates. Such plug-in standard errors are approximate and are most defensible when n is reasonably large and the normal model is plausible.

5. Sampling Distribution of the Correlation Coefficient

The sample correlation between variables X and Y is

Unlike covariance, r is dimensionless and lies between −1 and 1. Its distribution is bounded and generally skewed, especially when the population correlation ρ is near ±1 or the sample size is small. Therefore, r itself is not usually treated as normally distributed.

For testing H₀: ρ = 0 under bivariate normality, the exact statistic is

under H₀. This result assumes independent observations and a bivariate normal population.

For confidence intervals and comparisons, Fisher’s transformation is usually more effective:

When the data are bivariate normal, z is approximately normal with mean

and standard error 1/√(n − 3). A 100(1 − α)% interval on the transformed scale is

where z_{1−α/2} is a standard-normal quantile. The endpoints are then transformed back using

The transformation is inappropriate without modification for dependence, severe non-normality or very small samples. Correlation also does not establish causation: a common machine setting, batch effect or time trend may produce an association.

6. Regression Estimators as Sampling Distributions

Consider the multiple linear regression model

where y is an n × 1 response vector, X is an n × k design matrix of full column rank, β is a k × 1 vector of unknown coefficients and ε is an n × 1 error vector. Under the classical model,

and, for exact normal-theory inference, ε|X ~ N_n(0, σ²I_n).

The ordinary least-squares estimator is

Its conditional sampling distribution is

Thus each coefficient estimator is unbiased under the zero-conditional-mean assumption, but its precision depends on the design matrix. Strong correlation among predictors makes X^T X nearly singular and increases coefficient variances, a condition called multicollinearity.

The residual variance estimator is

where SSE = (y − Xβ̂)^T(y − Xβ̂) and n − k is the residual degrees of freedom. Under normal errors,

and β̂ and s² are independent. The standard error of coefficient β̂_j is the square root of the jth diagonal element of s²(X^T X)^(−1). The usual t statistic for H₀: β_j = β_{j0} is

under the stated model and null hypothesis.

Prediction and mean-response uncertainty

For a new mean response at predictor vector x₀, the fitted mean is x₀^Tβ̂, with variance

for predicting one new individual response. The 1 term represents new-process-unit error. For estimating the mean response at x₀, the variance is instead

Confusing these two intervals is a common quality-management error: a prediction interval is wider than a confidence interval for the mean.

7. Multivariate Response Models

When several responses are measured on each unit, a multivariate regression model may be written as

where Y is n × q, X is n × k, B is k × q and E contains correlated response errors. A common assumption is that the rows of E are independent and each row follows a q-variate normal distribution with covariance Ω. Correlation among responses is then modelled through Ω rather than ignored.

Separate regressions may provide unbiased coefficient estimates under suitable conditions, but joint modelling can improve efficiency, support simultaneous inference and test whether process factors affect the response vector as a whole. The model is more demanding: Ω must be estimated, responses must be measured consistently, and the interpretation of many parameters requires care.

8. Diagnostics for Multivariate Quality Models

A model should be diagnosed before its inferential output is used for release, process-capability or improvement decisions.

Residual structure

For regression, residuals are e = y − Xβ̂. Residual-versus-fitted plots can reveal nonlinearity or changing variance. A sequence plot can reveal drift, autocorrelation or shifts between operators and batches. Normal probability plots assess whether normal-theory intervals are plausible, although they should not be used mechanically.

Leverage and influence

The hat matrix is

Its diagonal h_ii measures leverage. An observation with unusual predictors can strongly affect the fitted model even if its residual is small. Cook’s distance is a common influence measure:

This expression is for the standard linear model with k fitted parameters. It is a screening measure, not an automatic deletion rule. An influential observation may be a data error, a special-cause event or valid evidence that the model is incomplete.

Multivariate residuals and distances

For a residual vector e_i and estimated residual covariance S_e, the squared standardized distance is

Large values identify observations unusual across responses jointly. A chi-squared reference may be used only approximately and only when the residual model and independence assumptions are credible. In small samples, bootstrap or permutation methods may be preferable.

Assumption review

The analyst should examine independence, measurement-system adequacy, linearity, constant variance, normality, missingness and the stability of the covariance structure. Repeated measurements from the same unit, serial production data and nested batches require models that represent dependence, such as mixed-effects, time-series or generalized estimating-equation approaches.

9. Worked Example: Joint Assessment of Two Quality Characteristics

An illustrative quality team measures thickness and tensile strength on n = 25 independent components. The target mean vector is μ₀ = (10.00 mm, 50.0 MPa)^T. The observed sample mean is x̄ = (10.12 mm, 49.1 MPa)^T, and the sample covariance matrix is

The covariance is 0.30 mm·MPa. The determinant is 0.25×4.00 − 0.30² = 0.91, so S is positive definite. Its inverse is

The difference vector is d = (0.12, −0.9)^T. Multiplication gives d^T S^(−1)d ≈ 0.281. Therefore,

For p = 2 and n = 25, the equivalent F statistic is

At the 5% level, F is compared with F_{2,23;0.95}, approximately 3.42. The result is just below this reference value, so the team would not reject the joint target mean at 5%, although the evidence is close and should not be described as proof that the process is exactly on target.

The result also demonstrates the role of covariance. A separate analysis might judge thickness and strength independently, but the T-squared analysis evaluates their combined, covariance-adjusted departure. Before acting, the team should verify measurement-system capability, independence and the stability of S. Because the decision is near the boundary, a larger sample or prospective confirmation is preferable to a categorical managerial conclusion.

10. Case Study: Correlated Characteristics in a Coating Process

Background

The following is an illustrative case. A coating plant evaluates film thickness and adhesion strength on panels produced across three shifts. Thickness is measured in micrometres and adhesion in newtons. Engineering requirements are joint: excessive thickness increases material cost, while low adhesion creates field-failure risk.

Problem

A univariate dashboard showed thickness within its control limits, but customer complaints about peeling increased. Engineers suspected that the thickness–adhesion relationship differed by curing temperature and shift.

Analysis

The quality team first audited the measurement system and found acceptable repeatability. It then plotted the two characteristics jointly, stratified by shift and curing temperature. The scatterplot showed positive association overall, but one shift formed a separate cluster. A covariance estimate computed after pooling all shifts overstated the apparent process stability because between-shift variation was mixed with within-shift variation.

A multivariate model included curing temperature, shift and their interaction. Residual plots indicated nonconstant variance at the highest temperature. A transformation of adhesion and a temperature-specific variance model improved the residual structure. The team calculated Hotelling’s T-squared statistic for each weekly subgroup, using a baseline mean and covariance estimated only from a stable reference period.

Approach and solution

The plant adopted three controls: separate baseline covariance matrices for materially different operating regimes; a joint T-squared alarm for correlated deviations; and diagnostic review before assigning a special cause. Investigation of the anomalous shift found a partially blocked curing-air filter. Correcting the filter reduced the between-shift separation and restored the expected thickness–adhesion relationship.

Outcome

The revised system detected a developing joint problem earlier than the previous pair of univariate charts. It also reduced false investigations caused by ordinary correlation. Importantly, the alarm did not identify the physical cause by itself; engineering investigation remained necessary.

Learning points

1. Correlation should be modelled, not treated as nuisance information.
2. A pooled covariance matrix can be misleading when several regimes are present.
3. Multivariate alarms require stable baselines and validated measurement systems.
4. Statistical detection and root-cause analysis are complementary activities.

11. Limitations, Governance and Contemporary Practice

The principal limitation of model-based sampling distributions is dependence on assumptions. High dimensionality is especially challenging: when p approaches n, S becomes unstable or singular, and T-squared cannot be computed in the ordinary way. Regularized covariance estimators, shrinkage methods and robust covariance procedures may then be considered, but their sampling behaviour differs from classical formulas.

Modern quality analytics increasingly combines multivariate statistics with process historians, sensor streams and machine learning. Principal component methods can reduce correlated measurements to a smaller set of latent directions, while partial least squares can link many process variables to quality outcomes. These tools should supplement—not replace—measurement validation, engineering knowledge and transparent diagnostics.

Governance is essential when statistical alarms affect product release, worker evaluation or supplier penalties. Data provenance, version-controlled code, documented exclusions and reproducible baselines should be maintained. Automated or AI-assisted anomaly detection must be monitored for drift and subgroup bias. A model trained on one product family should not silently govern another. Human review is particularly important when an unusual observation may represent a genuine safety event rather than an error.

Best practice is to: define the quality decision before analysing data; preserve units and traceability; inspect dependence and stratification; use covariance-aware methods; report uncertainty; validate assumptions; distinguish exploratory from confirmatory analysis; and periodically re-estimate models only through a controlled change process.

Chapter Summary

Multivariate sampling distributions extend classical inference to correlated quality characteristics. The sample mean vector has covariance Σ/n and, under multivariate normality, yields a chi-squared quadratic form when Σ is known. With unknown covariance, Hotelling’s T-squared statistic provides a joint test of a target mean and is converted to an F distribution under normality and nonsingularity conditions.

The sample scatter matrix follows a Wishart distribution under independent multivariate normal sampling. This result explains the joint variability of covariance estimates. Correlation coefficients have bounded, non-normal sampling distributions; Fisher’s transformation provides an effective approximate interval procedure. In regression, least-squares estimators are normally distributed under normal errors, while the residual variance has a scaled chi-squared distribution. Prediction intervals are wider than intervals for mean responses because they include new-unit variation.

Diagnostics—including residual analysis, leverage, influence and multivariate distance—are essential before using model-based results. In quality management, covariance-aware analysis can improve detection of joint process shifts, but it cannot replace measurement-system analysis, engineering investigation or responsible governance.

Key Terms

Multivariate normal distribution: A joint normal distribution for a vector of correlated variables.

Mean vector: The vector of expected values of the measured characteristics.

Covariance matrix: A symmetric matrix containing variances and covariances.

Correlation: A unit-free standardized covariance.

Mahalanobis distance: A covariance-adjusted distance from an observation to a reference mean.

Wishart distribution: The sampling distribution of a scatter matrix under multivariate normal sampling.

Hotelling’s T-squared: A multivariate statistic for testing a mean vector or monitoring joint departures.

Positive definite matrix: A matrix producing a positive quadratic form for every nonzero vector.

Fisher transformation: A logarithmic transformation used to approximate normality for a sample correlation.

Regression estimator: An estimate of a regression coefficient obtained from sample data.

Multicollinearity: Strong dependence among explanatory variables that inflates coefficient uncertainty.

Leverage: The extent to which an observation has unusual predictor values.

Influence: The effect of an observation on fitted estimates or conclusions.

Residual: The difference between an observed and fitted response.

Prediction interval: An interval for a future individual observation.

Confidence region: A multidimensional set of plausible parameter values.

Regularization: A method that stabilizes estimation by constraining or shrinking parameters.

Measurement-system analysis: Evaluation of the adequacy and variability of a measurement process.

Review Questions

Short Answer Questions

1. Why is a covariance matrix needed when several quality characteristics are analysed jointly?
2. State the distribution of (n − 1)S under independent multivariate normal sampling.
3. Why is Fisher’s transformation useful for a sample correlation?
4. Distinguish a confidence interval for a mean response from a prediction interval.
5. What does leverage measure in a regression model?

Long Answer Questions

1. Derive the Hotelling’s T-squared test statistic and explain the conditions under which its F distribution applies.
2. Explain how the Wishart distribution describes sampling variation in a sample covariance matrix.
3. Discuss the assumptions required for classical regression coefficient tests and the consequences of violating them.
4. Compare separate univariate analyses with a multivariate analysis of correlated quality characteristics.
5. Develop a diagnostic plan for a multivariate quality model used in production monitoring.

Analytical / Application Questions

1. A process measures three correlated characteristics and has a sample covariance matrix that is nearly singular. Explain the statistical and managerial risks, and recommend an appropriate analytical response.
2. A weekly T-squared alarm occurs after a supplier change, but no individual quality characteristic exceeds its univariate limit. Explain how you would investigate and communicate the result.
3. A regression model has a significant overall F test but no individually significant coefficients. Analyse how multicollinearity, covariance among estimators and model specification could produce this outcome.

Multiple Choice Questions with Answers

Question 1

Under independent multivariate normal sampling, which distribution describes (n − 1)S?

A. Wishart distribution

B. Poisson distribution

C. Uniform distribution

D. Binomial distribution

Correct Answer: A. Wishart distribution

Question 2

If Σ is known and X̄ is based on n observations, which statistic has a χ²_p distribution under multivariate normality?

A. (X̄ − μ)^TΣ(X̄ − μ)

B. n(X̄ − μ)^TΣ^(−1)(X̄ − μ)

C. (X̄ − μ)^TΣ^(−1)(X̄ − μ)/n

D. n(X̄ − μ)^TΣ(X̄ − μ)

Correct Answer: B. n(X̄ − μ)^TΣ^(−1)(X̄ − μ)

Question 3

What is the principal purpose of Fisher’s transformation of a sample correlation?

A. To remove all measurement error

B. To convert covariance into variance

C. To obtain an approximately normal scale for inference

D. To guarantee causal interpretation

Correct Answer: C. To obtain an approximately normal scale for inference

Question 4

In the classical linear model, what is the conditional covariance matrix of β̂?

A. σ²X^TX

B. σ²(X^TX)^(−1)

C. (X^TX)/σ²

D. σ²I_n

Correct Answer: B. σ²(X^TX)^(−1)

Question 5

Why is a prediction interval wider than a confidence interval for a mean response at the same predictor value?

A. It excludes residual error

B. It includes variation of a new individual response

C. It uses fewer degrees of freedom by definition

D. It assumes zero process variation

Correct Answer: B. It includes variation of a new individual response

Question 6

A covariance of 0.30 between thickness in millimetres and strength in MPa has which units?

A. Dimensionless

B. mm/MPa

C. mm·MPa

D. MPa²

Correct Answer: C. mm·MPa

Question 7

What does a large diagonal element of the regression hat matrix indicate?

A. High leverage

B. Low response variance

C. Perfect normality

D. A zero residual

Correct Answer: A. High leverage

Question 8

For Hotelling’s T-squared statistic with p variables and sample size n, the usual equivalent F statistic is

A. [(n − p)/(p(n − 1))]T²

B. [p(n − 1)/(n − p)]T²

C. T²/(n + p)

D. pT²/(n − 1)

Correct Answer: A. [(n − p)/(p(n − 1))]T²

Question 9

What is a major danger of using a pooled covariance matrix across distinct production regimes?

A. It necessarily makes all variables independent

B. It can obscure regime-specific variation and relationships

C. It forces every variance to zero

D. It prevents calculation of a sample mean

Correct Answer: B. It can obscure regime-specific variation and relationships

Question 10

If p is close to n, what problem commonly affects the sample covariance matrix?

A. It becomes automatically diagonal

B. It becomes unstable or singular

C. It becomes exactly equal to the population covariance

D. It loses all units

Correct Answer: B. It becomes unstable or singular