Measurement Error, Statistics and Uncertainty

Introduction

No measurement is perfectly exact. Every measured value is an estimate of a quantity, obtained through an instrument, a method, an operator, and an environment. In machine technology, the consequences of poor measurement may include rejected components, incorrect assembly, premature failure, excessive scrap, or the acceptance of unsafe products. Metrology therefore requires more than reading an instrument: it requires identifying error, evaluating variation, estimating uncertainty, and reporting results honestly.

This chapter develops the statistical and metrological foundations needed to distinguish systematic, random, and gross errors; summarize repeated measurements; propagate uncertainty through calculations; construct confidence intervals; and prepare an uncertainty budget. The treatment follows the practical logic used in calibration laboratories, dimensional inspection, manufacturing quality control, and conformity assessment.

Learning Objectives

After studying this chapter, you should be able to:

1. Distinguish systematic, random, and gross errors and identify their sources.
2. Calculate the mean, range, variance, standard deviation, and coefficient of variation of measurement data.
3. Explain accuracy, precision, repeatability, reproducibility, and traceability.
4. Estimate uncertainty using Type A and Type B methods.
5. Propagate standard uncertainties through sums, products, powers, and general measurement models.
6. Construct and interpret confidence intervals and expanded uncertainty intervals.
7. Prepare an uncertainty budget and report a measurement result correctly.
8. Diagnose practical measurement problems in a manufacturing environment.

1. Measurement Error and Measurement Result

Let the measured result be x and the conventional or reference value be x₀. The measurement error is

where E and x have the same unit. In practice, x₀ may not be known exactly; it may itself be an estimate with uncertainty. Consequently, error and uncertainty are different concepts. Error is the difference from a value taken as reference, whereas uncertainty expresses incomplete knowledge about the measurand.

A measurement result should normally be written as:

where ŷ is the best estimate, U is the expanded uncertainty, and the coverage probability associated with U should be stated. For example, “shaft diameter = 25.006 mm ± 0.004 mm, coverage factor k=2, approximately 95% coverage” is more informative than “25.006 mm.”

The measurand must be defined before measurement. “Diameter of the shaft” is incomplete unless temperature, location, cross-sectional direction, contact force, and measurement method are specified. A nominal dimension on a drawing is not automatically the true value of the physical part.

2. Classification of Measurement Errors

2.1 Systematic error

A systematic error is a component of error that remains constant or changes predictably under specified conditions. Examples include:

• a micrometer with a +0.008 mm zero offset;
• a scale factor error caused by incorrect calibration;
• thermal expansion of a workpiece;
• cosine error caused by misalignment;
• a consistent stylus force that deforms a soft component.

Systematic effects do not generally disappear by averaging repeated observations. They must be corrected, controlled, or included in the uncertainty evaluation. If a calibrated instrument has a stated correction, the corrected result may be written as

where x_m is the observed value and C is the correction. The sign of the correction must be established carefully. If the instrument reads 25.008 mm when a reference is 25.000 mm, the correction is −0.008 mm.

A correction is not necessarily exact. The uncertainty of the calibration correction remains a Type B uncertainty component.

2.2 Random error

Random error varies unpredictably from one observation to another. Sources include electrical noise, small positioning differences, operator variation, surface irregularity, and environmental fluctuations. Repeated observations may be modeled as

where μ represents the underlying expected value and ε_i is a random deviation with expected value approximately zero under stable conditions.

Random effects can often be reduced by repeated measurements, because positive and negative deviations partially cancel in the mean. They cannot be eliminated completely, and averaging is useful only when the measurement process remains stable and the observations are meaningfully comparable.

2.3 Gross error

A gross error is a mistake or abnormal event, such as recording 52.31 instead of 25.31 mm, using the wrong range, reading the wrong scale, or measuring a different component. Gross errors are not ordinary statistical variation. They should be prevented, investigated, and corrected using documented evidence. An observation should not be deleted merely because it is inconvenient or distant from the mean.

2.4 Accuracy and precision

Accuracy describes closeness to a reference value; precision describes closeness among repeated results. A set of readings may be precise but inaccurate if a systematic offset is present. Conversely, readings may have a mean close to the reference but show poor precision. Calibration primarily addresses systematic effects, whereas controlled conditions and repeated measurement address random variation.

3. Descriptive Statistics for Repeated Measurements

Suppose n measurements are x₁, x₂, ..., x_n.

3.1 Arithmetic mean

The mean is the usual estimate of the central value when observations are independent and drawn from a stable process. It has the same units as the measurements.

3.2 Range

The range is simple and useful for rapid shop-floor checks, but it uses only two observations and is unstable for small samples.

3.3 Deviations and sample standard deviation

The deviation of an observation is d_i = x_i − x̄. The sample variance and sample standard deviation are

The denominator n−1 is used when the sample mean is estimated from the same data. The standard deviation has the same units as the observations and measures dispersion. The variance has squared units.

3.4 Standard deviation of the mean

If measurements are independent and have approximately the same variance, the standard uncertainty of the mean is

Increasing n reduces uncertainty due to random variation, but it does not remove bias, calibration error, thermal effects, or an invalid method. If observations are correlated, the formula may underestimate uncertainty.

3.5 Coefficient of variation

For a non-zero positive mean, relative variation may be expressed as

It is useful when comparing dispersion across quantities with different scales, but it is unsuitable when the mean is near zero or when the quantity can be negative.

Worked Example 1: Repeated diameter measurements

Five measurements of a nominal 20 mm diameter are: 20.012, 20.008, 20.010, 20.014, and 20.006 mm.

The mean is

The deviations are +0.002, −0.002, 0, +0.004, and −0.004 mm. Their squared sum is 0.000040 mm². Therefore,

The standard deviation of the mean is

Thus the repeated observations estimate the mean diameter as 20.010 mm, with a Type A standard uncertainty of approximately 0.0014 mm if the stated assumptions are reasonable.

4. Probability Distributions and Confidence Intervals

A measurement population is often approximated by a normal distribution when many small independent effects contribute. The normal model is not automatic; skewed data, truncation, drift, and mixed populations require investigation.

For a normal variable with mean μ and standard deviation σ, approximately 68.27% of observations lie within μ ± σ, 95.45% within μ ± 2σ, and 99.73% within μ ± 3σ. These percentages describe a distribution, not a guarantee about a small sample.

When σ is estimated from a small sample, the Student’s t distribution is used. A two-sided confidence interval for the population mean is

where α is the significance level, n−1 is the degrees of freedom, and t_(α/2,n−1) is the relevant t critical value. For a 95% interval, α=0.05. This interval describes the uncertainty of the estimated mean under the statistical model; it is not necessarily an interval containing every future individual measurement.

Worked Example 2: A 95% confidence interval

For the five diameters above, x̄=20.010 mm, s=0.00316 mm, and n=5. For 95% confidence and four degrees of freedom, t=2.776. Thus

The confidence interval is therefore approximately

or 20.0061 to 20.0139 mm. This interval concerns the population mean, assuming independent, stable, approximately normal observations.

5. Uncertainty Evaluation: Type A and Type B

The Guide to the Expression of Uncertainty in Measurement (GUM) separates methods of evaluation, not physical causes.

5.1 Type A evaluation

Type A evaluation uses statistical analysis of repeated observations. For the mean of independent repeated measurements, the standard uncertainty is often u_A = s/√n. Type A evaluation can also use regression residuals, repeated weighing, or analysis of variance when appropriate.

5.2 Type B evaluation

Type B evaluation uses information other than repeated statistical observations, including:

• calibration certificates;
• manufacturer specifications;
• previous measurement data;
• resolution limits;
• environmental records;
• scientific knowledge;
• reference material certificates.

Type B does not mean inferior. A well-established calibration certificate may provide stronger information than a small set of repeated readings.

If a digital display is rounded to a resolution r, and all values within ±r/2 are considered equally likely, a rectangular distribution is often used:

If a specification gives limits ±a with no further information and a rectangular distribution is appropriate,

These formulas apply only when the assumed distribution and limits are justified. A specification limit is not automatically a probability distribution.

For a normal calibration uncertainty reported with coverage factor k, the corresponding standard uncertainty is approximately

provided the certificate’s coverage convention and distributional assumptions are understood.

6. Combining and Propagating Uncertainty

Let the measurand be modeled by

where the input quantities x_i have standard uncertainties u(x_i). For small uncertainties and a sufficiently smooth model, the combined standard uncertainty is

The first summation is the independent-input contribution. The covariance term is required when input quantities are correlated. If inputs are independent, covariance terms are zero.

The sensitivity coefficient c_i = ∂f/∂x_i shows how strongly the output responds to input x_i. The contribution of an independent input is

Important special cases

For addition or subtraction, y = x₁ ± x₂, independent uncertainties combine as

For multiplication or division, y = x₁x₂ or y=x₁/x₂, small relative uncertainties combine approximately as

For a power, y=x^p,

These relative formulas assume independent inputs and sufficiently small uncertainties. The general derivative formula should be used when these conditions do not hold.

Worked Example 3: Uncertainty in density

A metal specimen has mass m=125.00 g with standard uncertainty u(m)=0.02 g, and volume V=15.00 cm³ with u(V)=0.03 cm³. Density is

Assuming independent inputs,

Therefore,

The combined standard result is approximately ρ=8.333 ± 0.017 g/cm³, with a coverage factor of one. If approximately 95% coverage is justified by a normal model, an expanded uncertainty with k=2 is U≈0.034 g/cm³.

7. Expanded Uncertainty and Reporting

The expanded uncertainty is

where k is the coverage factor. k≈2 often gives approximately 95% coverage when the effective degrees of freedom are sufficiently large and the distribution is approximately normal. It is not universally correct. For small samples or dominant non-normal components, a t-based factor or another method may be necessary.

The effective degrees of freedom can be estimated using the Welch–Satterthwaite relation when independent components have known degrees of freedom:

where ν_i is the degrees of freedom associated with component u_i(y). This supports selection of a t-based coverage factor.

A technically sound report should include:

• the measurand and method;
• the measured value and correction, if any;
• the result and unit;
• the expanded uncertainty and coverage factor or coverage probability;
• relevant environmental or operating conditions;
• the traceability statement;
• the decision rule when conformity is assessed.

Rounding should be performed at the end. Uncertainty is commonly reported with one or two significant digits, and the result is rounded to the same decimal place. For example, 25.00637 ± 0.00384 mm may be reported as 25.006 ± 0.004 mm, provided the selected rounding policy is documented.

8. Constructing an Uncertainty Budget

An uncertainty budget lists the important input quantities, their estimates, distributions, standard uncertainties, sensitivity coefficients, and output contributions.

The combined standard uncertainty is the square root of the sum of squared contributions when the sources are independent:

With k=2, the expanded uncertainty is approximately 0.0042 mm, reported as 0.004 mm under a suitable rounding policy. The budget also reveals where improvement is worthwhile: reducing the largest contribution produces the greatest benefit.

A budget must not double-count effects. For example, if a calibration result already includes instrument resolution or environmental correction, those components should not be added again without justification. Correlations must also be considered, especially when the same reference, temperature, or calibration standard affects multiple inputs.

9. Detailed Applied Case: Inspection of a Precision Shaft

Background

This illustrative case concerns a shaft specified as 25.000 mm with a bilateral tolerance of ±0.010 mm. A workshop uses a calibrated micrometer to decide whether a batch is acceptable.

Problem

Three readings on one shaft are 25.006, 25.008, and 25.007 mm. The calibration certificate gives a correction of −0.002 mm with expanded uncertainty 0.004 mm at k=2. The micrometer resolution is 0.001 mm. Temperature effects are estimated as ±0.003 mm and treated as rectangular. The operator must report the corrected diameter and decide whether the result is clearly conforming.

Analysis

The mean observed diameter is

The sample standard deviation is 0.001 mm, so the Type A standard uncertainty of the mean is

Applying the calibration correction gives

The calibration standard uncertainty is 0.004/2=0.002 mm. Resolution contributes 0.001/√12=0.00029 mm. Temperature contributes 0.003/√3=0.00173 mm. Assuming independence,

Using k=2,

A suitable report is therefore approximately 25.005 mm ± 0.005 mm, k=2, subject to the stated assumptions.

Decision and outcome

The nominal tolerance interval is 24.990 to 25.010 mm. The reported interval, approximately 24.999 to 25.010 mm, lies within the specification limits except that its upper endpoint reaches the upper limit after rounding. The shaft estimate itself is conforming, but the result is close to the upper boundary and uncertainty is material. A binary pass decision requires a documented decision rule. Under a guard-band rule requiring the entire expanded interval to lie within specification, the part would not be accepted as demonstrably conforming. Under a simple estimate-based rule, it would pass. These are different decisions and must not be confused.

Learning points

1. Repeated readings alone do not describe total measurement uncertainty.
2. Calibration correction changes the result, while calibration uncertainty remains in the budget.
3. Environmental effects can dominate a high-resolution instrument.
4. Conformity decisions require an explicit decision rule.
5. Reporting uncertainty prevents false precision and supports fair supplier and customer decisions.

10. Practical Control of Measurement Error

A robust measurement process begins with a clear procedure. The operator should verify instrument identification, calibration status, zero condition, range, contact force, alignment, cleanliness, and temperature. Workpieces should be stabilized when thermal expansion is significant. Measurement locations and orientations should be specified for features such as roundness, flatness, and taper.

Control charts can distinguish ordinary process variation from unusual events. A stable process should be investigated before data are pooled across operators, instruments, or time periods. Gauge repeatability and reproducibility studies help determine whether observed variation arises mainly from equipment, appraisers, or parts.

Outlier tests may be useful, but they should never replace technical investigation. An observation may be extreme because it reveals a real process fault. Data editing must be documented, reproducible, and authorized.

Digital instruments and automated systems create additional risks: software configuration errors, unit conversion errors, timestamp problems, loss of raw data, and undocumented algorithmic filtering. Artificial intelligence may assist anomaly detection or image-based inspection, but its output remains a measurement result requiring validation, traceability, uncertainty evaluation, and human oversight. A model’s confidence score is not automatically metrological uncertainty.

11. Limitations and Professional Responsibilities

Statistical formulas depend on assumptions. Independence may fail when measurements drift over time. Normal approximations may fail for bounded, skewed, or multimodal data. Resolution may not equal actual measurement capability. Repeating a biased measurement produces a more precise estimate of the wrong value. Uncertainty budgets are models, not substitutes for engineering judgment.

Metrology also has ethical implications. Reporting extra digits, hiding unfavorable observations, changing a decision rule after seeing results, or claiming traceability without an unbroken calibration chain undermines confidence in the measurement system. Laboratories and inspection departments should retain raw observations, calibration records, environmental data, calculation methods, revisions, and authorization for changes. Results affecting safety, regulatory compliance, or contractual acceptance require especially strong document control.

Chapter Summary

Measurement error is the difference between a measured value and a reference value, while uncertainty expresses incomplete knowledge of the measurand. Systematic errors are persistent or predictable and require correction or evaluation; random errors vary unpredictably and may be reduced by replication; gross errors arise from mistakes or abnormal events and require investigation.

The mean summarizes central tendency, the sample standard deviation describes dispersion, and the standard deviation of the mean quantifies random uncertainty in an average under appropriate independence assumptions. Confidence intervals for a mean use the Student’s t distribution when the population standard deviation is estimated from a small sample.

Type A uncertainty is evaluated statistically from repeated data. Type B uncertainty uses calibration certificates, specifications, resolution, environmental information, and other non-repeated evidence. Uncertainty propagation uses sensitivity coefficients and covariance terms. Independent contributions combine by root-sum-of-squares, and expanded uncertainty is obtained by multiplying combined standard uncertainty by a justified coverage factor.

An uncertainty budget makes assumptions visible and identifies dominant sources. Correct reporting includes the measurand, result, unit, correction, uncertainty, coverage information, method, conditions, and decision rule. In manufacturing, reliable measurement depends on both mathematical evaluation and disciplined control of instruments, operators, environment, data, and documentation.

Key Terms

Measurement error: Difference between a measured value and a reference value.

Uncertainty: Parameter describing the dispersion of values reasonably attributable to a measurand.

Systematic error: Error component that remains constant or changes predictably.

Random error: Error component that varies unpredictably between observations.

Gross error: Mistake or abnormal event producing an invalid observation.

Accuracy: Closeness of agreement with an accepted reference value.

Precision: Closeness of agreement among repeated measurements.

Repeatability: Precision under the same measurement conditions.

Reproducibility: Precision under changed specified conditions.

Traceability: Documented relation to a reference through an unbroken calibration chain.

Mean: Arithmetic average of observations.

Variance: Mean squared dispersion measure, with sample correction when appropriate.

Standard deviation: Square root of variance, expressed in the measurement unit.

Confidence interval: Interval procedure for estimating a population parameter at a stated confidence level.

Type A evaluation: Uncertainty evaluation using statistical analysis of repeated observations.

Type B evaluation: Uncertainty evaluation using non-statistical information.

Sensitivity coefficient: Partial derivative showing output response to an input quantity.

Combined standard uncertainty: Standard uncertainty obtained by combining input contributions.

Expanded uncertainty: Combined standard uncertainty multiplied by a coverage factor.

Uncertainty budget: Structured account of uncertainty sources, assumptions, and contributions.

Review Questions

Short Answer Questions

1. Distinguish systematic error from random error and give one example of each.
2. Why does averaging repeated readings not remove calibration bias?
3. Define repeatability and reproducibility.
4. What is the purpose of the denominator n−1 in sample variance?
5. Explain the difference between Type A and Type B uncertainty evaluation.

Long Answer Questions

1. Explain how accuracy, precision, resolution, and traceability differ, and discuss their importance in dimensional inspection.
2. Derive and explain the standard uncertainty of the mean for independent repeated observations.
3. Describe how a calibration correction should be applied and how its uncertainty enters an uncertainty budget.
4. Explain uncertainty propagation for a general measurement model, including the role of sensitivity coefficients and covariance.
5. Describe the contents of a professional measurement report and explain why a coverage factor and decision rule should be stated.

Analytical / Application Questions

1. A bore is measured six times as 40.012, 40.010, 40.011, 40.015, 40.009, and 40.013 mm. Calculate the mean, sample standard deviation, and standard uncertainty of the mean. State the assumptions needed for interpretation.
2. A length is reported as 100.00 mm with a calibration standard uncertainty of 0.03 mm, a resolution of 0.01 mm, and a temperature contribution of 0.04 mm. Assuming independent contributions, calculate the combined standard uncertainty and expanded uncertainty for k=2.
3. A component estimate is within its drawing tolerance, but its expanded uncertainty interval crosses the specification limit. Compare an estimate-based decision with a guard-band decision and recommend what information should be documented.

Multiple Choice Questions with Answers

Question 1

Which type of error is most directly reduced by taking repeated independent measurements and averaging them?

A. Systematic error

B. Random error

C. Gross error

D. Calibration traceability error

Correct Answer: B. Random error

Question 2

For observations 10.02, 10.04, and 10.06 mm, what is the arithmetic mean?

A. 10.02 mm

B. 10.03 mm

C. 10.04 mm

D. 10.06 mm

Correct Answer: C. 10.04 mm

Question 3

Why is n − 1 used in the denominator of the usual sample variance?

A. To convert millimetres to metres

B. To account for estimating the mean from the same sample

C. To eliminate systematic error

D. To make the variance have the same units as the data

Correct Answer: B. To account for estimating the mean from the same sample

Question 4

A digital instrument has a resolution of 0.02 mm, and a rectangular distribution is justified for rounding effects. What is the standard uncertainty?

A. 0.001 mm

B. 0.0058 mm

C. 0.010 mm

D. 0.020 mm

Correct Answer: B. 0.0058 mm

Question 5

Which statement correctly describes Type B uncertainty evaluation?

A. It always requires at least 30 repeated readings

B. It uses information such as calibration certificates and specifications

C. It applies only to gross errors

D. It is identical to calculating the sample mean

Correct Answer: B. It uses information such as calibration certificates and specifications

Question 6

For independent quantities added together, how are standard uncertainties combined?

A. By simple addition in all cases

B. By subtracting the smaller uncertainty

C. By root-sum-of-squares

D. By multiplying the uncertainties

Correct Answer: C. By root-sum-of-squares

Question 7

A calibration certificate reports an expanded uncertainty of 0.006 mm with coverage factor k = 2. What is the corresponding standard uncertainty, assuming the certificate convention is applicable?

A. 0.001 mm

B. 0.003 mm

C. 0.008 mm

D. 0.012 mm

Correct Answer: B. 0.003 mm

Question 8

What does a sensitivity coefficient represent in uncertainty propagation?

A. The number of observations collected

B. The response of the output to a change in an input quantity

C. The instrument’s display resolution only

D. The confidence level of a calibration laboratory

Correct Answer: B. The response of the output to a change in an input quantity

Question 9

Which statement about a 95% confidence interval for a population mean is correct?

A. It contains exactly 95% of all individual future measurements

B. It is based on an estimation procedure with nominal 95% long-run coverage under its assumptions

C. It proves that systematic error is zero

D. It is always narrower than a one-standard-deviation interval

Correct Answer: B. It is based on an estimation procedure with nominal 95% long-run coverage under its assumptions

Question 10

A reported result is 25.005 mm ± 0.005 mm with k = 2. What does the ±0.005 mm term represent?

A. The instrument resolution only

B. The gross error limit

C. The expanded uncertainty under the stated coverage convention

D. The nominal drawing tolerance

Correct Answer: C. The expanded uncertainty under the stated coverage convention