From Sampling Distributions to Statistical Decisions

Introduction and context

Quality improvement rarely depends on knowing the exact value of a population parameter. A manager may instead need to decide whether a process mean exceeds a specification target, whether a supplier’s defect rate has changed, or whether a new method is economically worthwhile. Sampling distributions provide the bridge between observed sample data and such decisions. They quantify sampling variation, support confidence intervals, and provide the reference distributions used in hypothesis tests.

Statistical decision-making is not a mechanical search for a small p-value. It requires an explicit decision problem, an appropriate sampling design, defensible assumptions, control of error rates, attention to practical importance, and consideration of costs and risks. In total quality management (TQM), these requirements are central to evidence-based improvement, acceptance decisions, process validation, and organizational learning.

Learning objectives

By the end of this chapter, the reader should be able to:

• construct and interpret confidence intervals for means, proportions, and variances;
• formulate and conduct one-sample and two-sample hypothesis tests;
• distinguish Type I error, Type II error, power, statistical significance, and practical significance;
• select tests using sampling-distribution assumptions and diagnostic evidence;
• manage multiple comparisons and report uncertainty transparently;
• connect statistical evidence to quality costs, risks, and managerial decisions.

1. From a sample statistic to a sampling distribution

A population is the complete set of units or opportunities of interest; a sample is the subset actually observed. A parameter, such as the population mean μ or defect probability p, describes the population. A statistic, such as the sample mean x̄ or sample proportion p̂, describes the sample and estimates the parameter.

A sampling distribution is the probability distribution of a statistic over repeated samples drawn according to the same sampling design. It is not the distribution of individual observations. This distinction is fundamental: individual measurements may be highly variable even when the sample mean is estimated precisely.

For a simple random sample of independent observations with population mean μ and standard deviation σ, the sample mean satisfies:

Here n is sample size and SE denotes standard error. If σ is unknown, it is commonly estimated by s, giving SE(x̄) = s/√n. The central limit theorem states that, under broad conditions, the sampling distribution of x̄ approaches normality as n increases, even when the population distribution is not normal. The approximation is less reliable for small samples, highly skewed populations, or populations with extreme outliers.

For a sample proportion from independent Bernoulli observations:

In practice p is usually unknown, so p̂ is substituted for p when estimating the standard error. The normal approximation is generally justified when expected successes and failures are both sufficiently large; exact or alternative methods are preferable otherwise.

The finite population correction may be relevant when sampling without replacement from a finite population and the sampling fraction is not small:

N is the population size. The correction should not be applied automatically to a process that continually generates observations or to a sample that is a small fraction of a very large population.

2. Confidence intervals: estimating with stated precision

A confidence interval combines a point estimate with a margin of error. A 95% confidence procedure is designed so that, over repeated samples under its assumptions, approximately 95% of the intervals contain the true parameter. It is not correct to say that there is a 95% probability that a fixed parameter lies in one already calculated interval under the classical interpretation.

2.1 Mean with known or large-sample standard deviation

When σ is known, or when a normal approximation is justified and σ is reliably estimated, a confidence interval for μ is:

where α is the total error probability and z_(1−α/2) is the standard-normal critical value. For a 95% interval, α = 0.05 and the critical value is approximately 1.96.

When σ is unknown and the observations are plausibly normal, use Student’s t distribution:

The degrees of freedom are n − 1. The t critical value exceeds the corresponding normal value for small samples because estimating σ adds uncertainty.

Worked example: fill volume

A random sample of 25 containers has mean fill 502 ml and standard deviation 6 ml. Assuming independent approximately normal measurements, a 95% confidence interval for the process mean uses t_(0.975,24) ≈ 2.064:

Margin of error = 2.064 × 6/√25 = 2.477 ml.

Thus the interval is 502 ± 2.477, or approximately (499.52 ml, 504.48 ml). The interval describes uncertainty about the mean fill, not the range of individual container fills. A prediction interval for an individual future container would be wider.

2.2 Proportions

A simple Wald interval is:

It performs poorly with small samples or proportions near 0 or 1. The Wilson interval is often preferable. With z = z_(1−α/2), its center and half-width are:

Exact binomial intervals may be used when approximation conditions are weak, although their coverage can be conservative.

2.3 Differences between groups

For two independent means, an interval for μ₁ − μ₂ is:

Welch’s method uses an approximate degrees of freedom ν and does not require equal population variances. It is generally safer than the pooled-variance method when variance equality is doubtful. For paired observations, calculate within-pair differences dᵢ and use a one-sample interval for μ_d. Pairing can reduce noise when each observation in one condition has a meaningful counterpart in the other.

2.4 Planning sample size

For a two-sided confidence interval for a mean with desired half-width E and known or planning value σ:

Round upward to the next whole number. For a proportion with anticipated value p:

If p is unknown, p = 0.5 gives the largest required sample size. These formulas assume a probability sample, independence, and the relevant approximation conditions. Nonresponse, clustering, stratification, finite populations, and attrition may require adjustment.

3. Hypothesis testing as a decision procedure

A hypothesis test begins with a null hypothesis H₀ and an alternative hypothesis H₁. H₀ usually represents a reference condition, such as a target mean or no change. The alternative reflects the improvement or deterioration that matters. A test statistic measures how far the sample result is from H₀ in standard-error units.

For a one-sample mean with unknown σ:

μ₀ is the value specified by H₀. Under normality, t follows a t distribution with n − 1 degrees of freedom when H₀ is true. A two-sided test rejects for unusually large absolute values of t; a one-sided test is justified only when the direction was specified before seeing the data and the opposite direction would not support the decision of interest.

For a one-sample proportion, a large-sample statistic is:

The null value p₀, rather than p̂, appears in the standard error because the reference distribution is constructed under H₀. Exact binomial tests are appropriate when the normal approximation is unreliable.

The p-value is the probability, assuming H₀ is true, of obtaining a result at least as inconsistent with H₀ as the observed result, according to the prespecified alternative. It is not the probability that H₀ is true, nor the probability that the result occurred by chance in an informal sense. A significance level α is a decision threshold selected before analysis.

A confidence interval and a two-sided test at level α are closely linked: if the null value lies outside the corresponding 1 − α confidence interval, the null is rejected. This equivalence does not remove the need to report effect size and uncertainty.

4. Type I error, Type II error, and power

A Type I error rejects a true H₀; its long-run probability is α. A Type II error fails to reject a false H₀; its probability is β for a specified alternative. Power is the probability of detecting that alternative:

Power is not a fixed property of a test. It depends on sample size, variability, α, the true effect, and whether the test is one- or two-sided. Larger samples and lower variability usually increase power. Lowering α reduces false alarms but generally reduces power unless sample size increases.

“Fail to reject” is not equivalent to “prove no effect.” A non-significant result may reflect a genuinely negligible effect, insufficient power, poor measurement, or an unsuitable design. Equivalence and non-inferiority tests address questions of acceptable similarity or bounded disadvantage by defining a meaningful margin Δ before data collection.

Statistical versus practical significance

A tiny effect can be statistically significant in a very large sample, while a consequential effect can be non-significant in a small sample. Quality decisions should specify a minimally important difference, such as a reduction of 0.5 percentage points in defects, a 2-minute reduction in cycle time, or a cost saving of $10 per unit. Report the estimated effect, confidence interval, and operational consequences alongside the p-value.

Standardized effect sizes can support comparison across scales. For two independent means, a commonly used standardized difference is:

where s_p is a pooled standard deviation under the equal-variance model. Standardization is not a substitute for domain-specific units: managers must still ask whether the raw difference affects customers, capability, safety, or cost.

5. Multiple comparisons and selective analysis

Testing many hypotheses increases the chance of at least one false positive. If m independent tests each use α = 0.05, the probability of at least one Type I error is:

For m = 20 this is approximately 0.642 under independence. Dependence changes the exact value, but the general problem remains.

The family-wise error rate (FWER) is the probability of one or more false rejections in a family of tests. The Bonferroni procedure tests each hypothesis at α/m and is simple and conservative. Holm’s step-down procedure is usually more powerful while controlling FWER: order p-values from smallest to largest and compare the ith p-value with α/(m − i + 1).

When many discoveries are screened, controlling the false discovery rate (FDR)—the expected proportion of false discoveries among rejected hypotheses—may be more appropriate. The Benjamini–Hochberg procedure is a common FDR method. The choice depends on consequences: safety validation may require strong FWER control, whereas exploratory process screening may tolerate a controlled proportion of false leads.

Unplanned subgroup searches, repeated testing until significance, and reporting only favorable outcomes inflate false-positive risk. A sound protocol identifies primary outcomes, analysis populations, stopping rules, and comparison families in advance. Exploratory analyses remain valuable when clearly labeled and confirmed in subsequent work.

6. Assumptions, diagnostics, and robust practice

Statistical decisions are conditional on the data-generating process. Important assumptions include:

• Independence: observations or appropriately modeled units are not correlated. Time-ordered process data often violate this condition through autocorrelation.
• Representative sampling: the sample supports the target population. Convenience samples can produce systematic bias that no formula corrects.
• Measurement validity: the gauge or inspection method is sufficiently accurate and repeatable. A gauge capability study should precede fine distinctions in process performance.
• Distributional adequacy: t procedures are often robust for moderate, balanced samples, but extreme skewness and outliers require investigation.
• Stable process context: mixing data from different shifts, machines, or improvement phases can make a single parameter meaningless.

Use run charts, control charts, residual plots, stratification, and subject-matter knowledge. Do not use a normality test as the sole basis for deciding whether a method is valid; graphical and process evidence are more informative. If observations are clustered, use cluster-aware standard errors or hierarchical models. If data are autocorrelated, time-series or control-chart methods may be needed rather than a naive independent-sample test.

7. Statistical decisions and economic criteria

A statistical test answers an evidence question; a quality decision answers an action question. The action may be to adjust a process, stop a line, qualify a supplier, launch a redesign, or collect more information. Decision quality improves when the organization specifies the losses associated with false alarms, missed problems, investigation, customer harm, and delay.

A simple expected-loss framework compares actions across possible states of nature. If action a has consequence C(a, θ) when the true state is θ, the expected loss is:

Choose the action with the smallest expected loss, subject to safety, legal, and ethical constraints. This Bayesian expression requires prior probabilities and a model for the data; frequentist confidence intervals and p-values do not themselves supply those probabilities. In practice, organizations may use scenario probabilities, sensitivity analysis, or conservative thresholds when priors are uncertain.

For a process change, a simplified net-benefit model is:

Benefits may include reduced scrap, warranty claims, rework, energy use, and customer dissatisfaction. Residual failure cost should include probability-weighted consequences, not merely average production cost. A statistically significant improvement with negative net benefit should not automatically be implemented; conversely, a valuable improvement may justify additional data even before conventional significance is achieved.

8. Integrated illustrative case: reducing seal defects

Background. A manufacturer’s historical seal-defect rate is 4.0%. An engineering team introduces a revised heating profile. The quality charter defines a practically important reduction as at least 1.0 percentage point, and the team plans a two-sided comparison because unexpected deterioration would also matter.

Problem. In a random sample of 1,000 post-change units, 25 are defective, so p̂ = 0.025. Management must decide whether the evidence supports a change while considering false alarms, missed improvements, and implementation cost.

Analysis. The estimated reduction is 0.040 − 0.025 = 0.015, or 1.5 percentage points. A rough 95% interval for the post-change rate using the Wilson method is preferable to an uncritical Wald interval because the defect rate is small. The interval should be reported with the exact method and its assumptions. The practical threshold is then assessed directly: does the interval for the reduction exclude 0 and plausibly include at least 0.010? A formal test against p₀ = 0.040 may be conducted using an exact binomial or suitable two-sample method, but statistical significance alone is insufficient.

The team also checks whether the sample covers shifts, machines, and raw-material lots; verifies the inspection system; examines a control chart for stability; and compares energy consumption and cycle time. If the 1,000 units were consecutive from one machine on one shift, the nominal sample size would overstate information.

Approach and decision. Suppose the process is stable, sampling is representative, and the estimated annual saving from fewer defects is $180,000. Implementation and validation cost $60,000, while the expected annual cost of an undetected adverse change is estimated at $250,000. The decision team conducts sensitivity analysis over plausible probabilities and requires a staged pilot with a holdout shift. This combines statistical evidence with economic and operational safeguards.

Outcome. If confirmation across shifts supports the reduction and no safety or capability concern emerges, the change can be standardized through controlled documentation and ongoing monitoring. If the interval is wide, the responsible conclusion is not “no improvement,” but “additional representative data are needed.”

Learning points. Define practical importance before testing; protect against biased sampling; distinguish process stability from one-time evidence; use interval estimates; and translate statistical results into risk-adjusted operational choices.

9. Governance, ethics, and contemporary practice

Quality data may influence worker evaluation, supplier penalties, customer safety, and regulatory submissions. Analysts must preserve data integrity, document exclusions, protect confidential information, and avoid manipulating sampling or stopping rules to obtain a desired result. Regulatory and accreditation requirements may impose traceability, validated methods, and predefined acceptance criteria.

Statistical software and AI can calculate intervals, simulate power, detect unusual patterns, and assist with model selection. They do not eliminate the need to define the estimand, assess sampling bias, understand measurement systems, or justify assumptions. Automated subgroup discovery can create a multiple-comparisons problem; opaque predictive models can obscure uncertainty and accountability. Best practice is to retain an auditable analysis plan, version-controlled data and code, independent review for high-risk decisions, and confirmation on new data.

Best-practice checklist

1. Define the population, estimand, decision, and practically important effect.
2. Use a probability-based or otherwise defensible sampling design.
3. Validate measurement and inspect process stability before inference.
4. Choose methods that match dependence, distribution, and scale of measurement.
5. Plan sample size and power before collecting confirmatory data.
6. Report effect sizes, confidence intervals, assumptions, and missing data—not only p-values.
7. Control multiplicity and distinguish confirmatory from exploratory analyses.
8. Combine statistical evidence with safety, cost, capability, and ethical criteria.
9. Confirm important findings on representative future data.
10. Monitor the process after implementation; a decision is not the end of learning.

Chapter Summary

Sampling distributions quantify how statistics vary across repeated samples and provide the foundation for standard errors, confidence intervals, and hypothesis tests. Confidence intervals express estimation uncertainty, while hypothesis tests evaluate compatibility with a prespecified null model. Type I error, Type II error, and power describe different risks; statistical significance must be separated from practical importance. Multiple comparisons require explicit control of FWER or FDR. Valid inference depends on sampling, measurement, independence, distributional, and process-stability assumptions. In TQM, the final decision should integrate statistical evidence with customer impact, safety, implementation cost, and expected loss. Transparent planning, effect-size reporting, confirmation, and monitoring turn statistical analysis into responsible quality improvement.

Key Terms

Sampling distribution: Distribution of a statistic over repeated samples.

Standard error: Estimated standard deviation of a sampling distribution.

Confidence interval: Interval produced by a procedure with specified long-run coverage.

Confidence level: Nominal long-run coverage, such as 95%.

Null hypothesis: Reference claim evaluated by a statistical test.

Alternative hypothesis: Claim representing the effect or departure of interest.

p-value: Probability, under H₀, of a result at least as extreme as observed.

Significance level: Prespecified Type I error threshold.

Type I error: Rejecting a true null hypothesis.

Type II error: Failing to reject a false null hypothesis.

Power: Probability of detecting a specified alternative.

Practical significance: Operational importance of an effect.

Effect size: Quantified magnitude of a difference or relationship.

Multiple comparisons: Simultaneous testing of several hypotheses.

FWER: Probability of at least one false rejection in a test family.

FDR: Expected proportion of false discoveries among rejections.

Welch procedure: Two-sample method that does not assume equal variances.

Expected loss: Probability-weighted consequence of an action.

Estimand: Precisely defined quantity a study seeks to estimate.

Review Questions

Short Answer Questions

1. What is a sampling distribution, and how does it differ from the distribution of individual observations?
2. Interpret a 95% confidence interval using the classical repeated-sampling meaning.
3. Distinguish Type I error from Type II error.
4. Why can a statistically significant result lack practical significance?
5. What problem is addressed by the Bonferroni procedure?

Long Answer Questions

1. Explain the construction, assumptions, and interpretation of a confidence interval for a population mean when σ is unknown.
2. Compare one-sided and two-sided hypothesis tests, including when each is defensible in quality improvement.
3. Discuss how sample size, variability, α, and effect size influence statistical power.
4. Explain why process stability, measurement validity, and independence are essential before inferential analysis.
5. Develop a framework for integrating statistical evidence with expected economic loss in a process-change decision.

Analytical / Application Questions

1. A process sample has n = 16, x̄ = 48.2, and s = 4.0. Construct a 95% t-based confidence interval for μ and explain the assumptions required.
2. A team tests 12 independent quality metrics at α = 0.05. Explain the multiplicity risk and recommend an appropriate adjustment for a confirmatory study.
3. A supplier’s defect rate falls from 3.5% to 2.8%, but the sample is drawn from one production shift. Identify the inferential weaknesses and design a stronger validation study.

Multiple Choice Questions with Answers

Question 1

Which statement best describes a sampling distribution?

A. The distribution of all individual observations in a population

B. The distribution of a statistic across repeated samples

C. The distribution of measurement errors within one sample

D. The distribution of specification limits across products

Correct Answer: B. The distribution of a statistic across repeated samples

Question 2

For a 95% confidence interval for a mean with unknown population standard deviation and a small normal sample, which reference distribution is generally appropriate?

A. Uniform distribution

B. Chi-square distribution for the mean

C. Student’s t distribution

D. Binomial distribution

Correct Answer: C. Student’s t distribution

Question 3

A Type I error occurs when a researcher:

A. Fails to reject a false null hypothesis

B. Rejects a true null hypothesis

C. Accepts a false alternative hypothesis with certainty

D. Uses a sample that is larger than necessary

Correct Answer: B. Rejects a true null hypothesis

Question 4

Statistical power is defined as:

A. The probability of rejecting a true null hypothesis

B. The probability that the p-value is exactly zero

C. The probability of detecting a specified alternative when it is true

D. The probability that the sample is representative

Correct Answer: C. The probability of detecting a specified alternative when it is true

Question 5

If 20 independent tests are each conducted at α = 0.05, which expression gives the probability of at least one Type I error?

A. 0.05 ÷ 20

B. 1 − (1 − 0.05)^20

C. 20 × 0.95

D. 0.95^20 − 1

Correct Answer: B. 1 − (1 − 0.05)^20

Question 6

Which procedure is designed to control the family-wise error rate and is typically less conservative than simple Bonferroni adjustment?

A. Holm’s step-down procedure

B. Ordinary least squares

C. Wilson interval

D. Finite population correction

Correct Answer: A. Holm’s step-down procedure

Question 7

A p-value is best interpreted as:

A. The probability that the null hypothesis is true

B. The probability that the observed result was caused by bias

C. The probability, under the null hypothesis, of a result at least as extreme as observed

D. The probability that the alternative hypothesis is practically important

Correct Answer: C. The probability, under the null hypothesis, of a result at least as extreme as observed

Question 8

Which change generally increases statistical power, all else being equal?

A. Reducing sample size

B. Increasing measurement variability

C. Increasing sample size

D. Replacing a prespecified effect with an unspecified effect

Correct Answer: C. Increasing sample size

Question 9

When comparing two independent means and equal variances are doubtful, which method is generally preferred?

A. Welch’s two-sample procedure

B. A paired t-test regardless of pairing

C. A one-sample proportion test

D. A chi-square goodness-of-fit test only

Correct Answer: A. Welch’s two-sample procedure

Question 10

Why should a quality decision not rely solely on statistical significance?

A. Significance tests never use sampling distributions

B. A small effect may be significant while having little operational value

C. Confidence intervals cannot be calculated for quality data

D. Economic costs are always unrelated to statistical evidence

Correct Answer: B. A small effect may be significant while having little operational value