Applications in Statistical Process Control and Capability

Introduction

Sampling distributions provide the quantitative bridge between observed process data and decisions about process stability, capability, and measurement. In statistical process control (SPC), a sample statistic—such as a subgroup mean, range, standard deviation, or proportion—is compared with limits derived from its sampling distribution. In capability analysis, estimates of process spread and centring are interpreted with recognition that they are uncertain because they come from finite samples. In measurement-system analysis, sampling variation helps distinguish product variation from variation introduced by the measurement process.

These applications are especially important in postgraduate Total Quality Management because control-chart decisions are not purely mechanical. A point outside a limit may indicate a special cause, but it may also be a false alarm caused by incorrect assumptions, autocorrelation, changing process conditions, or an inadequate measurement system. Conversely, a stable-looking chart may conceal a systematic drift or an inappropriate subgrouping strategy.

Learning objectives

After completing this chapter, the reader should be able to:

1. Derive and interpret control limits from sampling distributions.
2. Select rational subgroup sizes and sampling frequencies for practical SPC.
3. Explain how autocorrelation and nonstationarity affect control-chart performance.
4. Quantify uncertainty in capability indices and avoid overconfident capability claims.
5. Use measurement-system evidence when deciding whether capability analysis is credible.
6. Evaluate false-alarm risks and design an appropriate response to out-of-control signals.

1. Sampling distributions as the basis of SPC

A sampling distribution describes the probability behaviour of a statistic calculated from repeated samples. If individual process observations are independent and normally distributed with mean μ and standard deviation σ, then the mean of a subgroup of size n has distribution

Here, X̄ is the subgroup mean, μ is the process mean, σ is the process standard deviation, and n is subgroup size. The standard error of the mean is σ/√n, measured in the same units as the original observations.

This relationship explains why an X̄ chart is more sensitive to shifts in the process mean when n is increased: the sampling distribution becomes narrower. It also explains why an X̄ chart should not be interpreted as if it were an individuals chart. The two charts monitor different statistics and therefore have different sampling distributions.

For a process with unknown σ, σ is estimated from within-subgroup variation. Common estimators include the average range, R̄, and the average subgroup standard deviation, S̄. Control-chart constants incorporate the expected sampling behaviour of these estimators.

1.1 Three-sigma control limits

For a subgroup-mean chart, the general three-sigma limits are

where μ̂ is the estimated centre line and σ̂_X̄ is the estimated standard deviation of subgroup means. If σ is known, σ̂_X̄ = σ/√n. If σ is estimated from the average range, a conventional form is

X̄̄ is the grand mean, R̄ is the average within-subgroup range, and A₂ depends on subgroup size. The constants are based on the expected range of a normally distributed subgroup and are valid only under the assumptions supporting that construction.

For an S chart, commonly used limits are

where B₃ and B₄ depend on subgroup size. For an R chart, the corresponding form is

The lower range limit is often zero when the relevant constant is zero. A negative lower limit is not used because a range cannot be negative.

The constants should be taken from an authoritative SPC reference or software implementation whose convention is known. They should not be guessed or transferred between chart types.

1.2 Control limits are not specification limits

Control limits describe the expected behaviour of a statistic under a stable process. Specification limits, in contrast, express requirements imposed by customers, designers, regulators, or engineering authorities. A process can be statistically stable but incapable of meeting specifications, or unstable while most observations happen to fall within specifications.

This distinction is central to quality governance. Control limits support process diagnosis and prediction under stability; specification limits support conformity decisions. Neither replaces the other.

2. Rational subgrouping and sampling design

A rational subgroup is a set of observations collected close enough in time and under sufficiently similar conditions that within-subgroup variation represents short-term, common-cause variation. Differences between subgroup means then reveal changes over time, shifts between machines, operators, tools, batches, or environmental conditions.

Rational subgrouping is a modelling decision, not merely a convenience. If observations from different machines are mixed in one subgroup, the within-subgroup variation may be inflated and the chart may become insensitive to machine-specific shifts. If observations that are highly correlated are treated as independent, limits may be too narrow and false alarms may increase.

2.1 Choosing subgroup size

The ability of an X̄ chart to detect a mean shift δ depends approximately on the standardized shift

where δ is the change in process mean, n is subgroup size, and σ is the individual-observation standard deviation. Larger n improves sensitivity to small shifts, but it also increases inspection cost and may delay detection if samples are collected infrequently.

For a normally distributed statistic with known parameters, the probability that a subgroup mean exceeds the upper three-sigma limit after a shift δ is approximately

for a positive shift, where Φ is the standard normal cumulative distribution function. The analogous lower-tail probability applies for a negative shift. In practice, estimated parameters, supplementary run rules, and serial dependence alter this probability.

There is no universally optimal subgroup size. A useful design procedure is:

• identify the smallest practically important shift;
• estimate short-term process variation;
• compare candidate values of n using detection probability and cost;
• choose a sampling interval consistent with the time scale of likely assignable causes;
• verify the design using historical or pilot data.

Small subgroups are often appropriate when shifts must be detected quickly and measurements are expensive. Larger subgroups are useful when individual measurements are noisy but a stable estimate of the mean is required.

2.2 Sampling frequency and average run length

The in-control average run length, ARL₀, is the expected number of plotted points before a false signal when the process is stable. With independent normal observations and only the basic three-sigma rule, the two-sided false-signal probability per point is approximately 0.0027, giving

This is an approximation. It assumes known parameters, independent plotted statistics, and no additional run rules. When parameters are estimated, the actual in-control performance can differ, especially during the start-up period.

The out-of-control average run length, ARL₁, is the expected number of plotted points required to signal after a specified shift. ARL₁ is not a constant property of a chart; it depends on shift size, subgroup size, sampling interval, chart type, parameter estimation, and decision rules.

A process owner should therefore select a sampling frequency by balancing the cost of inspection and investigation against the cost of delayed detection. A chart that signals quickly but generates many false alarms can damage trust and encourage operators to ignore signals.

3. Attribute charts and non-normal data

Sampling distributions also determine limits for attribute charts. If each unit is independently classified as defective with probability p, the number of defectives in a sample of fixed size n follows a binomial distribution:

The sample proportion is p̂ = X/n, with mean p and standard deviation √(p(1−p)/n). Approximate three-sigma limits for a p chart are

where p̄ is the estimated average defective proportion. Limits are truncated at 0 and 1 when necessary. For unequal sample sizes, the limits vary with n.

For counts of defects per constant area or opportunity under a Poisson model, a c chart uses

The model assumes a constant opportunity size and a Poisson-like event process. If opportunity sizes vary, a u chart or another suitable model is needed.

Normal-theory limits may be inappropriate for skewed, bounded, count, or strongly discrete data. Transformations, distribution-specific charts, generalized linear models, or nonparametric methods may be preferable. The choice should be based on the data-generating mechanism rather than on a desire to make every chart look normal.

4. Autocorrelation, nonstationarity and false alarms

4.1 Autocorrelation

Autocorrelation exists when observations close in time are statistically related. A common model is the first-order autoregressive process:

where ρ is the lag-one autocorrelation parameter and ε_t is a random innovation with mean zero. For a stationary AR(1) process, |ρ| < 1.

Positive autocorrelation reduces the effective amount of independent information. For the mean of n consecutive observations, an approximate variance is

where σ² is the marginal variance and ρ_k is the autocorrelation at lag k. Under independence, all ρ_k are zero and the expression reduces to σ²/n. Positive autocorrelation makes the variance larger than the independent-sampling calculation, so limits based on independence may be too narrow and generate excessive signals.

Negative autocorrelation can have the opposite effect, although it should not automatically be treated as beneficial: it may reflect control-system dynamics, alternating measurements, or data-processing artefacts.

Possible responses include modelling the serial structure, sampling less frequently, charting residuals from an appropriate time-series model, using a time-weighted chart designed for correlated observations, or changing the process so that the source of dependence is understood. Simply widening limits without investigating the dependence can conceal real problems.

4.2 Nonstationarity

A process is nonstationary when its statistical properties change over time. Causes include tool wear, seasonal conditions, material changes, maintenance cycles, learning effects, recipe changes, and evolving automation. If data from different regimes are pooled, the estimated centre and spread may represent no actual operating condition.

A common error is to calculate limits from a historical period containing known shifts and then use those limits prospectively. The result may be either excessive false alarms or failure to detect a meaningful change. A better practice is to segment the data by regime, remove documented special-cause periods when justified, and record the rationale for every baseline decision.

4.3 False-alarm risks

False alarms arise when a stable process is incorrectly declared unstable. Their sources include random tail observations, multiple run rules, estimated parameters, autocorrelation, non-random sampling, measurement noise, and repeated chart review.

If several independent alarm rules are applied, the overall false-signal probability is greater than that of any single rule. The exact probability depends on the rule definitions and dependence among plotted points. Consequently, a plant should validate its complete decision protocol—not just its three-sigma limits—using simulation, historical replay, or analytical calculations.

A signal is evidence for investigation, not automatic proof of a special cause. The investigation should examine the measurement system, data integrity, timing, materials, equipment, operators, environment, and recent changes.

5. Capability indices and sampling uncertainty

Capability analysis should normally be performed only after the process has demonstrated reasonable statistical stability and the measurement system is adequate. For bilateral specifications, let LSL and USL denote the lower and upper specification limits. The potential capability index is

where σ̂ estimates within-process standard deviation. C_p measures the ratio of the specification width to the process spread but does not assess centring.

The centred capability index is

where μ̂ is the estimated process mean. The two one-sided indices are sometimes written C_pu = (USL−μ̂)/(3σ̂) and C_pl = (μ̂−LSL)/(3σ̂), so that C_pk = min(C_pu, C_pl).

These indices are dimensionless. They assume that σ̂ represents the relevant process variation, that the specification limits are applicable, and that the process distribution and stability assumptions are appropriate. A normal-distribution interpretation of expected nonconformance is not valid merely because a capability index has been calculated.

5.1 Illustrative capability calculation

Suppose a shaft diameter has USL = 10.10 mm and LSL = 9.90 mm. A stable study estimates μ̂ = 10.04 mm and σ̂ = 0.025 mm. Then

The one-sided values are

and

Therefore, C_pk = 0.80. The process has reasonably narrow spread relative to the specification width, but it is poorly centred toward the upper specification. Reporting only C_p = 1.33 would conceal the principal management problem.

5.2 Confidence intervals and capability decisions

Capability indices are estimates, not fixed process properties. Their uncertainty arises from finite sample size, uncertain mean, uncertain standard deviation, non-normality, autocorrelation, and possible baseline instability.

For a normal process with independent observations and a sufficiently large sample, an approximate standard error for C_p is often written

where n is the number of individual observations used to estimate σ. This approximation is mainly a planning or screening aid. It does not fully capture uncertainty in C_pk, because C_pk depends jointly on μ̂ and σ̂ and is affected by which specification side is limiting.

For serious release, supplier, or regulatory decisions, confidence intervals should be obtained using an appropriate method. Options include exact or approximate normal-theory methods under justified assumptions, parametric simulation, and bootstrap procedures. A bootstrap should respect the data structure: ordinary resampling is unsuitable for strongly autocorrelated observations unless blocks or a fitted time-series model are used.

A capability requirement should specify whether it concerns a point estimate, a lower confidence bound, or a long-term performance index. For example, “estimated C_pk ≥ 1.33” is materially weaker than “the lower 95% confidence bound for C_pk is at least 1.33.” The latter demands more evidence and is more defensible for high-risk decisions.

5.3 Non-normal and one-sided capability

For skewed or bounded distributions, conventional indices may misrepresent tail risk. Alternatives include percentile-based capability measures, transformation-based analysis, distribution fitting with goodness-of-fit assessment, and direct estimation of the proportion outside specifications. The selected method should reflect the decision objective: tail probability, expected loss, or conformity rate.

For a one-sided specification, a one-sided index such as C_pu or C_pl is more appropriate than C_p, which requires both specifications.

6. Measurement-system decisions

A capability study is only as credible as the measurement system that produced the data. Measurement error can be represented by

where Y is the observed measurement, X is the true characteristic, and E is measurement error. If X and E are independent and unbiased, then

This relationship shows why measurement error inflates observed variation and can reduce the apparent capability index. If the measurement system also has bias, the measured mean may be displaced relative to the true mean, creating incorrect centring conclusions.

A measurement-system study should evaluate repeatability, reproducibility, stability, bias, resolution, and linearity as relevant. Repeatability is variation when the same appraiser measures the same part repeatedly with the same instrument. Reproducibility concerns variation between appraisers or measurement conditions. Stability concerns change in measurement behaviour over time.

The decision should be risk-based. A measurement system may be adequate for process monitoring but inadequate for final acceptance of a tight specification. Conversely, an expensive high-resolution system may not be necessary for a broad screening decision.

A commonly used screening concept compares measurement variation with total process variation, often expressed as a percentage of study variation or tolerance. Thresholds vary by industry and governing standard; therefore, no universal acceptance percentage should be applied without reference to the applicable customer, regulatory, or organizational requirement. The underlying evidence and decision rationale should be documented.

7. Integrated applied case: precision-pump assembly

Background

The following is an illustrative case. A manufacturer produces precision-pump assemblies whose shaft runout must not exceed 0.080 mm. The production team wants to use an individuals-and-moving-range chart because only one assembly is tested at a time, followed by a capability claim for a new machining cell.

Problem

The initial 120 observations produce an estimated mean of 0.041 mm and an estimated standard deviation of 0.009 mm. A preliminary calculation gives a one-sided capability index of approximately

However, the data are collected every 30 seconds, and the process engineer suspects thermal drift and serial dependence. The gauge study also indicates that measurement variation is approximately 0.004 mm under the current study conditions.

Analysis

First, the team plots the time order and calculates the autocorrelation function. Positive lag-one autocorrelation is evident, with a gradual upward drift during long production runs. The data therefore do not represent 120 independent observations from one stationary distribution.

Second, the team separates observations by warm-up condition and production interval. The apparent standard deviation within stable intervals is lower than the pooled standard deviation, while the interval-to-interval means differ. Pooling the regimes had converted systematic drift into apparent random variation.

Third, the team evaluates the measurement system. The measurement error is not negligible relative to the process spread. If measurement error is independent and its variance estimate is reliable, a rough error-adjusted variance estimate is

This subtraction is meaningful only under the stated independence and variance-component assumptions; it should not be used mechanically when the measurement study is weak or the components are confounded.

Approach and solution

The team adopts the following actions:

1. A warm-up period is defined and recorded before capability data are collected.
2. The measurement system is improved and its calibration and stability are verified.
3. A time-ordered chart is used to monitor drift, while residual analysis is considered for serial dependence.
4. Capability is estimated only from a demonstrably stable interval.
5. A bootstrap confidence interval based on blocks is used because observations within short time periods are correlated.
6. The customer report distinguishes short-term capability from expected long-term performance.

Outcome

The revised study produces a lower point estimate than the original calculation, but its confidence interval is narrower because the process and measurement conditions are better controlled. The organization does not release the process solely because a point estimate exceeds an internal target. Instead, it requires the lower confidence bound and the control-chart evidence to satisfy the documented approval rule.

Learning points

The original index was not mathematically meaningless, but it answered an unclear question using data that violated important assumptions. Capability analysis is a measurement-and-process system decision, not merely a ratio calculated from a spreadsheet. Stable conditions, rational sampling, appropriate dependence treatment, and explicit uncertainty criteria are necessary for a defensible conclusion.

8. Governance, technology and best practice

Modern SPC software can calculate limits, estimate autocorrelation, simulate ARL performance, and generate bootstrap confidence intervals. These capabilities improve analysis but do not remove the need for engineering judgement. Automated alarms should include data-quality checks, time stamps, version-controlled limit revisions, and an auditable record of investigations.

Artificial intelligence may assist with anomaly detection, multivariate monitoring, image-based inspection, and prediction of tool wear. However, an algorithmic alarm is not automatically a statistical-control signal. AI models can drift, learn historical biases, or identify correlations with no causal explanation. Their use should include validation, monitoring of false positives and false negatives, human review for high-consequence decisions, and clear separation between exploratory prediction and formal conformity evidence.

Best practice includes:

• define the process, measurement, sampling unit, and decision purpose before choosing a chart;
• establish rational subgroups and preserve time order;
• estimate limits from a justified, stable baseline;
• distinguish control limits, specification limits, and confidence limits;
• examine autocorrelation and nonstationarity;
• investigate signals rather than merely deleting them;
• assess measurement-system adequacy before capability analysis;
• report capability uncertainty and the distributional assumptions used;
• document software, constants, transformations, exclusions, and limit changes;
• align acceptance rules with customer, regulatory, and organizational requirements.

Challenges and limitations

SPC methods are powerful but conditional. Three-sigma limits do not guarantee a desired economic outcome. A stable process can be incapable, and a capable-looking process can be unstable. Parameter estimates from short baselines may be unreliable. Multiple charts and repeated testing alter false-alarm probabilities. Autocorrelation can invalidate independent-sampling assumptions. Non-normality can make normal-theory tail estimates misleading. Measurement error can distort both spread and centring. Finally, a statistically significant signal may be operationally unimportant, while a small shift may be critical in a safety-related process.

The appropriate response is not to abandon statistical methods but to make assumptions visible, match the method to the data-generating system, and connect statistical evidence with engineering, customer, and risk requirements.

Chapter Summary

Sampling distributions determine how subgroup statistics vary and therefore provide the foundation for SPC control limits. The mean has standard error σ/√n under independent sampling, while range, standard-deviation, proportion, and count charts require their own sampling models and constants. Rational subgrouping must preserve meaningful short-term conditions and expose changes between subgroups.

Subgroup size and sampling frequency should be selected using the practically important shift, detection probability, cost, and time scale of assignable causes. ARL values are useful but depend on independence, parameter estimation, and the complete alarm protocol. Autocorrelation increases or decreases effective information, while nonstationarity makes pooled baselines potentially misleading.

Capability indices such as C_p and C_pk describe estimated spread and centring relative to specifications; they do not prove stability or conformity. Confidence intervals, simulation, or dependence-aware bootstrap methods are needed when capability decisions have significant consequences. Measurement-system variation must be assessed because it contributes to observed variation and can distort capability conclusions.

The central professional principle is that SPC and capability analysis are evidence-based decision systems. Their validity depends on sampling design, assumptions, measurement quality, data integrity, and disciplined interpretation.

Key Terms

Sampling distribution: Probability distribution of a statistic across repeated samples.

Rational subgroup: A subgroup collected under conditions intended to represent common-cause variation.

Control limit: A statistically derived boundary used to assess process stability.

Specification limit: A customer, design, regulatory, or engineering requirement.

Average run length: Expected number of plotted points before a signal occurs.

False alarm: An incorrect signal of instability from a stable process.

Autocorrelation: Statistical dependence between observations separated by time lags.

Nonstationarity: Change in a process’s statistical properties over time.

Capability: The ability of a stable process to meet specification requirements.

Cₚ: Potential capability index based on specification width and process spread.

Cₚₖ: Capability index incorporating both process spread and centring.

Measurement error: Difference between an observed measurement and the true characteristic.

Repeatability: Variation under repeated measurement by the same appraiser and method.

Reproducibility: Variation attributable to different appraisers or measurement conditions.

Bootstrap: Resampling method used to estimate sampling uncertainty.

AR(1) process: First-order autoregressive process in which the current value depends on the preceding value.

Within-subgroup variation: Short-term variation estimated inside rational subgroups.

Long-term variation: Variation across changing conditions, time periods, or production regimes.

Review Questions

Short Answer Questions

1. Why are control limits different from specification limits?
2. What is meant by rational subgrouping?
3. How does increasing subgroup size affect the standard error of the subgroup mean?
4. Why can positive autocorrelation produce excessive control-chart signals?
5. Why is C_pk generally more informative than C_p when process centring is uncertain?

Long Answer Questions

1. Explain how sampling distributions support the design and interpretation of X̄ and R charts. State the assumptions involved.
2. Discuss how subgroup size and sampling frequency should be selected for a process with expensive inspection and rapidly developing assignable causes.
3. Compare the consequences of autocorrelation and nonstationarity for SPC baselines and control limits.
4. Explain why capability indices require uncertainty analysis and describe suitable approaches for obtaining confidence intervals.
5. Discuss how measurement-system repeatability, reproducibility, bias, and stability affect capability decisions.

Analytical / Application Questions

1. A stable process has USL = 50 units, LSL = 40 units, μ̂ = 46 units, and σ̂ = 1 unit. Calculate C_p, C_pu, C_pl, and C_pk. Interpret the result.
2. A production line uses an X̄ chart based on subgroups of four, but observations within each subgroup are positively autocorrelated. Explain how the nominal control limits may be affected and propose an investigation plan.
3. A supplier reports C_pk = 1.50 from 30 observations but provides no control chart, measurement-system study, or confidence interval. Evaluate whether the customer should accept the capability claim and specify the additional evidence required.

Multiple Choice Questions with Answers

Question 1

For independent normal observations with process standard deviation σ, what is the standard error of a subgroup mean based on n observations?

A. σn

B. σ/√n

C. σ²/n

D. √n/σ

Correct Answer: B. σ/√n

Question 2

Which statement correctly distinguishes control limits from specification limits?

A. Control limits are always set by the customer.

B. Specification limits are estimated from subgroup ranges.

C. Control limits describe expected statistical process behaviour, whereas specification limits express requirements.

D. They are interchangeable terms.

Correct Answer: C. Control limits describe expected statistical process behaviour, whereas specification limits express requirements.

Question 3

If USL = 10.10, LSL = 9.90, and σ̂ = 0.025, what is Cp?

A. 0.80

B. 1.00

C. 1.33

D. 1.87

Correct Answer: C. 1.33

Question 4

In the same process, if μ̂ = 10.04, what is the approximate Cpk?

A. 0.80

B. 1.33

C. 1.60

D. 1.87

Correct Answer: A. 0.80

Question 5

What is the principal purpose of rational subgrouping?

A. To maximize the number of observations in every subgroup

B. To ensure within-subgroup variation represents comparable short-term conditions

C. To combine data from all machines into one estimate

D. To eliminate the need for a measurement-system study

Correct Answer: B. To ensure within-subgroup variation represents comparable short-term conditions

Question 6

Under the basic independent three-sigma rule, the approximate in-control ARL is:

A. 3

B. 30

C. 100

D. 370

Correct Answer: D. 370

Question 7

Positive autocorrelation among observations generally has what effect on the variance of a sample mean compared with an independence assumption?

A. It generally increases the variance.

B. It always makes the variance zero.

C. It has no possible effect.

D. It changes the units of the mean.

Correct Answer: A. It generally increases the variance.

Question 8

Which condition is most important before making a conventional capability claim?

A. The process has demonstrated reasonable statistical stability.

B. At least one observation is inside specification.

C. The histogram has exactly six bars.

D. The control limits equal the specification limits.

Correct Answer: A. The process has demonstrated reasonable statistical stability.

Question 9

If observed measurement variation includes independent process and measurement components, which relationship is appropriate?

A. Var(Y) = Var(X) − Var(E) in all cases

B. Var(Y) = Var(X) + Var(E)

C. Var(Y) = Var(X) × Var(E)

D. Var(Y) = Var(X)/Var(E)

Correct Answer: B. Var(Y) = Var(X) + Var(E)

Question 10

Why may an ordinary bootstrap be unsuitable for strongly autocorrelated observations?

A. It always produces a negative standard deviation.

B. It assumes observations can be resampled independently, which can destroy serial dependence.

C. It cannot estimate any confidence interval.

D. It applies only to attribute data.

Correct Answer: B. It assumes observations can be resampled independently, which can destroy serial dependence.