Verification, Validation and Uncertainty Analysis

Introduction

A mechatronic model is useful only when its results are sufficiently trustworthy for the decision being made. A simulation may reproduce equations correctly and still represent the wrong physical system. Conversely, a model may represent the correct system but contain coding errors, incorrect units, or numerical instability. Engineering practice therefore separates three activities: verification, validation, and calibration. These activities are complemented by uncertainty analysis, sensitivity analysis, robustness assessment, and Monte Carlo simulation.

Verification asks whether the model has been implemented correctly. Validation asks whether the model adequately represents the intended physical system for a defined purpose. Calibration estimates model parameters from observations; it does not, by itself, prove that the model is valid. A credible modeling process also documents test conditions, error measures, uncertainty sources, acceptance criteria, and limitations.

This chapter develops these ideas for mechanical, electrical, control, robotic, and embedded mechatronic systems.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Distinguish verification, validation, and calibration.
2. Design tests that provide meaningful evidence about a model.
3. Quantify prediction error using appropriate metrics.
4. Propagate measurement and parameter uncertainty through a model.
5. Perform local sensitivity, robustness, and Monte Carlo analyses.
6. Define defensible acceptance criteria and document evidence.
7. Interpret validation results without overstating model credibility.

1. Verification, Validation, and Calibration

1.1 Verification

Verification determines whether the mathematical model has been translated into software, hardware, or a numerical algorithm correctly. It addresses the question:

Are we solving the equations correctly?

Typical verification activities include:

• checking units and dimensional consistency;
• comparing a numerical solution with an analytical solution;
• testing conservation of energy, mass, charge, or momentum;
• checking limiting cases, such as zero input or zero friction;
• reducing a complex model to a simpler benchmark model;
• refining the time step or mesh and examining convergence;
• testing individual software functions and interfaces.

For a numerical result y_h obtained using discretization size h, convergence may be assessed by comparing results at successively smaller values of h. If the solution approaches a limiting value as h decreases, the implementation may be numerically convergent. Convergence alone, however, does not establish physical validity.

1.2 Validation

Validation determines whether the model is an adequate representation of the real system for a specified use. It addresses the question:

Are we solving the right equations for this purpose?

Validation requires comparison with independent experimental or field data. The comparison must specify the operating range, input excitation, measured outputs, sensors, sampling rate, environmental conditions, and performance criteria.

Validation is never absolute. A model can be valid for predicting motor speed during nominal operation and invalid for predicting thermal failure, saturation, or behavior outside the tested load range. Consequently, the statement “the model is validated” should be replaced by a more precise statement such as “the model predicts shaft speed within the stated tolerance for loads between 0.2 and 1.0 N·m.”

1.3 Calibration

Calibration estimates unknown or poorly known parameters so that model outputs agree with selected observations. Parameters may include friction coefficients, motor constants, damping, sensor bias, or thermal resistance.

Suppose measured outputs are y_i and model outputs are ŷ_i(θ), where θ is a vector of parameters. A common least-squares calibration minimizes:

Here, n is the number of observations, w_i is a non-negative weighting factor, and θ is the parameter vector. The formula assumes that squared deviations are an appropriate loss function and that the observations being combined are comparable after weighting. If measurement variances σ_i² are known, a common choice is w_i = 1/σ_i².

Calibration data must not be confused with validation data. If the same data are used both to estimate parameters and to claim predictive success, the reported performance may be optimistic. A defensible study separates calibration, validation, and, where possible, final test datasets.

2. A Credibility Workflow

A rigorous workflow begins before simulation results are produced.

Step 1: Define the intended use

State the decision supported by the model. Examples include selecting a motor, predicting positioning error, testing a controller, estimating battery life, or assessing safety margins. The intended use determines which outputs, operating regimes, and error tolerances matter.

Step 2: Define quantities and acceptance criteria

Specify input variables, output variables, units, sampling conditions, and acceptable error. A criterion should be measurable. For example:

• maximum absolute position error ≤ 0.5 mm;
• root-mean-square speed error ≤ 2% of rated speed;
• predicted settling time within ±10%;
• probability of violating a temperature limit less than 1%.

Acceptance criteria should be established before examining results whenever practical. Otherwise, criteria may be unconsciously adjusted to obtain a favorable conclusion.

Step 3: Verify the model

Use analytical solutions, limiting cases, independent implementations, conservation laws, and numerical convergence studies.

Step 4: Plan and conduct experiments

The test should excite the behaviors relevant to the intended use. A constant input may be inadequate for identifying dynamics. Step, ramp, chirp, pseudo-random, load-change, and disturbance tests may reveal different properties. Sensor bandwidth, actuator saturation, quantization, backlash, and environmental conditions must be recorded.

Step 5: Calibrate only identifiable parameters

A parameter is identifiable when the available data contain enough information to estimate it uniquely or with acceptable uncertainty. Highly correlated parameters can produce similar outputs, making individual estimates unreliable even when the overall fit appears good.

Step 6: Validate using independent data

Compare predictions with measurements not used for calibration. Examine both numerical metrics and residual plots. A small average error can conceal systematic phase shift, bias, drift, or poor performance during transients.

Step 7: Quantify uncertainty and report limitations

Report assumptions, parameter ranges, measurement uncertainty, test conditions, excluded phenomena, and the region in which the model is supported by evidence.

3. Quantifying Prediction Error

Let y_i be the measured value and ŷ_i the model prediction at sample i. The residual is:

The sign convention must be stated because some disciplines define residuals in the opposite direction.

3.1 Common error metrics

The mean error, or bias estimate, is:

The mean absolute error is:

The root-mean-square error is:

MAE has the same units as the output and weights all absolute deviations linearly. RMSE also has the output units but gives greater influence to large errors. Bias indicates systematic over- or under-prediction, whereas MAE and RMSE measure overall discrepancy.

A normalized RMSE can be defined as:

where y_max and y_min are the maximum and minimum measured values over the specified test. Other normalizations, such as division by the mean or rated value, may be more meaningful, but the choice must be documented.

Percentage error is meaningful only when the reference value is not near zero:

For dynamic systems, pointwise error metrics should be supplemented with features such as rise time, overshoot, settling time, phase lag, steady-state error, and frequency-response deviation.

3.2 Worked example: error metrics

An actuator is tested at four times. Measured positions are 10, 20, 30, and 40 mm; predictions are 11, 19, 32, and 39 mm. Using measured minus predicted, the residuals are −1, 1, −2, and 1 mm.

The bias is:

The MAE is:

The RMSE is:

The negative bias indicates slight overall over-prediction by the model under the stated residual convention. If the acceptance limits were MAE ≤ 1.5 mm and RMSE ≤ 2.0 mm, these criteria would be satisfied. This conclusion applies only to the tested conditions.

3.3 Residual analysis

Residuals should be plotted against time, input, predicted output, and operating condition. Patterns are evidence of model inadequacy or neglected effects. Examples include:

• a curved residual trend: nonlinear behavior may be missing;
• oscillating residuals: frequency or phase dynamics may be incorrect;
• increasing variance: noise may be heteroscedastic;
• residuals correlated with temperature: thermal dependence may be omitted;
• a sudden residual change: saturation, backlash, mode switching, or sensor faults may exist.

Independence assumptions should not be made automatically for time-series data. Autocorrelated residuals can make apparently large sample sizes misleading.

4. Uncertainty in Mechatronic Models

Uncertainty is lack of complete knowledge about a quantity. It differs from an error, which is the difference between a measured or predicted value and a reference or true value. Uncertainty may arise from:

• sensor resolution, noise, bias, and drift;
• uncertain geometry, mass, inertia, resistance, and friction;
• manufacturing tolerances;
• environmental conditions;
• unknown disturbances and loads;
• model-form simplifications;
• numerical approximation;
• variation between units or operating cycles.

4.1 Type A and Type B evaluation

Repeated observations can provide a statistical estimate of uncertainty, often called Type A evaluation. Other information—calibration certificates, manufacturer specifications, resolution, prior studies, or engineering judgment—supports Type B evaluation. Both should be documented, and their assumptions should be explicit.

For n independent repeated measurements x_i, the sample mean is:

The sample standard deviation is:

The standard uncertainty of the mean is s/√n when observations are independent and identically distributed. This does not represent all uncertainty in an instrument; systematic bias must be considered separately.

4.2 Propagation of uncertainty

Let an output be q = f(x₁, x₂, …, x_m). For small uncertainties and approximately linear behavior near the nominal values, the first-order variance propagation formula is:

Here, u_j is the standard uncertainty of x_j and u_{jk} is the covariance between x_j and x_k. If inputs are independent, covariance terms are zero. The approximation is appropriate when uncertainties are relatively small and f is sufficiently smooth near the operating point.

For q = a + b with independent uncertainties u_a and u_b:

For q = ab, a first-order relative form is:

The relative form assumes nonzero nominal values and independent inputs.

4.3 Worked example: a simple electrical estimate

The power dissipated by a resistor is q = VI. Suppose voltage V = 12.0 V with standard uncertainty u_V = 0.2 V, and current I = 2.0 A with u_I = 0.05 A. The nominal power is:

For independent measurements:

Thus u_q/q ≈ 0.0295 and u_q ≈ 24.0 × 0.0295 = 0.71 W. The result may therefore be reported as approximately 24.0 ± 0.7 W, where the meaning of the interval must be stated. This is a standard-uncertainty result, not automatically a 95% confidence interval.

If the model is strongly nonlinear, inputs have large uncertainty, or distributions are bounded or asymmetric, linear propagation may be inadequate. Monte Carlo propagation is then preferable.

5. Sensitivity Analysis

Sensitivity analysis investigates how changes in inputs or parameters affect outputs. It helps identify influential parameters, prioritize measurements, simplify models, and design robust systems.

5.1 Local sensitivity

For output q = f(x), the dimensional local sensitivity to x_j is:

Its units are output units per input unit. A dimensionless relative sensitivity is:

This describes the approximate fractional change in q caused by a fractional change in x_j. It is meaningful when x_j and q are nonzero and the local linear approximation is reasonable.

For a dynamic model, sensitivity may vary with time, frequency, state, or operating point. A parameter that is unimportant at low speed may dominate near resonance or saturation.

5.2 Global sensitivity

Local derivatives examine one neighborhood. Global methods vary inputs across their full plausible ranges and account for interactions. Approaches include one-at-a-time sweeps, factorial designs, Latin hypercube sampling, rank correlation, and variance-based indices.

A first-order variance-based sensitivity index for input X_j is conceptually the fraction of output variance attributable to that input alone:

Q is the output random variable and E denotes conditional expectation. This index assumes a probabilistic input model and is most naturally interpreted when inputs are independent. Interaction effects require higher-order terms or a total-effect index.

6. Robustness and Acceptance Criteria

A design is robust when acceptable performance is maintained despite expected variation and disturbances. Robustness is not the same as nominal accuracy. A model may match one experiment closely but predict unacceptable behavior under component tolerances, temperature changes, payload variation, or sensor noise.

A robustness study should define:

1. uncertain parameters and credible ranges or distributions;
2. disturbances and environmental conditions;
3. performance measures;
4. hard constraints, such as maximum temperature or current;
5. acceptable probability of violation;
6. worst-case or percentile-based decision rules.

For a requirement q ≤ q_max, a probabilistic criterion may be:

where α is the permitted violation probability. The criterion is meaningful only when the probability distributions and dependencies are credible.

Worst-case analysis evaluates combinations at the limits of their specified ranges. It can be conservative, especially when all extreme values are unlikely to occur simultaneously. Probabilistic analysis may be less conservative but depends more strongly on distributional assumptions.

7. Monte Carlo Analysis

Monte Carlo analysis propagates uncertainty by repeatedly sampling uncertain inputs, evaluating the model, and summarizing the output distribution.

A basic algorithm is:

1. Define probability distributions or bounded sampling rules for uncertain inputs.
2. Generate a sample x^(k) for k = 1,…,N.
3. Evaluate q^(k) = f(x^(k)).
4. Store outputs and check constraints.
5. Estimate mean, standard deviation, quantiles, and violation probability.
6. Repeat with larger N or different random seeds to assess sampling stability.

The sample mean and sample standard deviation of N simulated outputs q^(k) are:

If r outputs violate a limit, the estimated violation probability is p̂ = r/N. For rare-event decisions, a very large N may be required; observing no violations does not prove that the true probability is zero.

Random samples can be inefficient when the model is expensive. Latin hypercube sampling improves coverage of input ranges, while surrogate models can reduce computational cost. Reproducibility requires recording the random-number generator, seed, distributions, sample size, model version, and software environment.

8. Illustrative Case: Servo-Positioning Module

Background

The following is an illustrative case involving a belt-driven servo axis intended to position a lightweight inspection camera. The model includes motor torque, belt elasticity, viscous damping, encoder quantization, and a discrete controller.

Problem

The engineering team must determine whether the model can predict positioning performance before selecting a controller update. The intended operating range is 0.1–0.8 m/s, with payloads from 0.5 to 1.5 kg. The acceptance requirements are:

• steady-state position error ≤ 0.20 mm;
• settling time within ±10% of measured settling time;
• no more than 1% probability of exceeding a motor-current limit under specified variation.

Verification

The software team first verifies units, signs, and coordinate definitions. With friction and external load set to zero, the simulated mechanical energy follows the expected conservation relationship apart from numerical integration error. A reduced first-order motor model is also compared with an independently calculated step response. Decreasing the integration step from 1 ms to 0.5 ms changes the peak position by less than 0.1%, supporting numerical convergence for the selected tests.

Calibration

Friction and belt damping are estimated using low-speed motion tests. The calibration data contain several payloads, but only speeds below 0.4 m/s. The fitted parameters reproduce these tests well. Because high-speed data were not used, the team does not treat the high-speed region as calibrated.

Validation

Independent tests use step commands, ramp commands, and payload changes. The model predicts steady-state error within 0.12 mm for all tests, satisfying the 0.20 mm criterion. However, its predicted settling time is 18% shorter than measured during a payload change. Residuals show a delayed oscillation, suggesting that belt compliance or structural flexibility is underrepresented.

The model therefore passes the position-error criterion but fails the settling-time criterion for the tested payload-change condition. It should not be used to approve a high-bandwidth controller without further refinement.

Uncertainty and robustness analysis

The team assigns distributions to payload, belt stiffness, encoder bias, and friction. Ten thousand Monte Carlo simulations are conducted using a fixed, documented seed. The current-limit violation estimate is 0.7%, below the 1% requirement, but the result is sensitive to belt stiffness. A subsequent sensitivity study shows that improving knowledge of belt stiffness would reduce prediction uncertainty more effectively than refining encoder quantization.

Outcome and learning points

The model is verified for its implementation and partially validated for its intended operating range. Calibration improved agreement but did not establish validity outside the calibration data. Validation metrics must be assessed by operating condition, not only as one aggregate number. Uncertainty and sensitivity results identify both design risk and the most valuable next experiment.

9. Practical and Professional Considerations

Documentation and traceability

A credible model should have a version identifier, parameter source, unit system, solver settings, test data, preprocessing procedure, acceptance criteria, and record of changes. Data should be traceable to instruments and test conditions. Figures should distinguish measured data, calibrated predictions, validation predictions, and uncertainty bands.

Ethical and safety responsibilities

Engineers must not present calibrated agreement as proof of general validity. Omitting failed tests, changing acceptance criteria after seeing results, or reporting only favorable operating conditions can create unsafe decisions. Safety-critical models require independent review, conservative assumptions where justified, and validation evidence appropriate to the hazard and applicable standards.

AI-assisted modeling

Machine-learning surrogates and parameter-estimation tools can accelerate simulation, but they introduce additional risks: data leakage between calibration and validation sets, lack of extrapolation reliability, hidden preprocessing, and poor interpretability. AI-generated model components require the same verification, validation, uncertainty assessment, and version control as conventional code. A high predictive score on held-out data is not sufficient evidence for physical correctness or safety.

10. Best-Practice Checklist

Before accepting a mechatronic simulation, ask:

• Is the intended use explicitly stated?
• Are verification and validation treated as separate activities?
• Were units, signs, limiting cases, and numerical convergence checked?
• Were calibration and validation datasets separated?
• Do tests excite the dynamics relevant to the decision?
• Are bias, transient behavior, and residual patterns examined?
• Are parameter, measurement, environmental, and model-form uncertainties considered?
• Are correlations between uncertain inputs documented?
• Are acceptance criteria quantitative and defined in advance?
• Has robustness been assessed over credible operating conditions?
• Are random seeds, sample sizes, software versions, and model revisions recorded?
• Are limitations and unsupported extrapolations clearly reported?

Chapter Summary

Verification checks whether equations and algorithms have been implemented correctly; validation checks whether the model represents the real system adequately for a defined purpose; and calibration estimates uncertain parameters from data. Calibration can improve agreement but cannot replace independent validation.

Prediction error should be quantified using bias, MAE, RMSE, feature-based metrics, and residual analysis. Uncertainty arises from measurements, parameters, disturbances, numerical methods, and model-form assumptions. First-order propagation is useful for small, smooth, approximately linear uncertainty, whereas Monte Carlo methods are better for nonlinear, bounded, correlated, or non-Gaussian problems.

Sensitivity analysis identifies influential variables and guides measurement and redesign. Robustness analysis tests whether requirements remain satisfied under expected variation. Acceptance criteria must be quantitative, traceable, and linked to the intended engineering decision. Professional credibility depends on independent test data, transparent assumptions, reproducible computation, and honest reporting of limitations.

Key Terms

Verification: Checking that a model or algorithm has been implemented correctly.

Validation: Assessing whether a model adequately represents a real system for a specified purpose.

Calibration: Estimating uncertain model parameters from observations.

Residual: The difference between a measured value and a predicted value under a stated convention.

Bias: Systematic average deviation between prediction and measurement.

MAE: Mean absolute error, the average magnitude of residuals.

RMSE: Root-mean-square error, a metric that emphasizes larger deviations.

Uncertainty: Lack of complete knowledge about a quantity.

Standard uncertainty: An uncertainty expressed as a standard deviation or equivalent standard deviation.

Covariance: A measure of joint variation between two uncertain quantities.

Propagation of uncertainty: Determining how input uncertainty affects output uncertainty.

Sensitivity: The change in an output caused by a change in an input or parameter.

Robustness: Ability to satisfy requirements despite expected variation and disturbances.

Monte Carlo analysis: Repeated random sampling and model evaluation to estimate output distributions.

Acceptance criterion: A predefined measurable condition for judging model or system performance.

Identifiability: The extent to which parameters can be uniquely estimated from available data.

Residual analysis: Examination of residual magnitude, structure, correlation, and dependence on conditions.

Model-form uncertainty: Uncertainty caused by simplifications or missing physical phenomena.

Review Questions

Short Answer Questions

1. What is the principal difference between verification and validation?
2. Why should calibration data and validation data normally be separated?
3. What does RMSE measure, and why does it emphasize large errors?
4. State two sources of uncertainty in a mechatronic model.
5. When is first-order uncertainty propagation likely to be inadequate?

Long Answer Questions

1. Explain a complete workflow for establishing credibility in a mechatronic simulation, from intended use through uncertainty reporting.
2. Compare local and global sensitivity analysis, including situations in which each is useful.
3. Explain how residual plots can reveal model deficiencies that are hidden by a single error metric.
4. Discuss worst-case and probabilistic robustness analysis, including advantages and limitations.
5. Describe how calibration, verification, validation, and documentation should be integrated in a safety-relevant engineering project.

Analytical / Application Questions

1. A model predicts positions of 5, 10, and 15 mm as 4, 11, and 14 mm, respectively. Calculate the bias, MAE, and RMSE using measured minus predicted residuals. Interpret the results.
2. A temperature model depends on resistance, current, and ambient temperature. Design a validation test plan that distinguishes sensor bias, parameter error, and omitted thermal dynamics.
3. A Monte Carlo study produces 160 constraint violations in 20,000 trials. Estimate the violation probability and explain what additional information is needed before concluding that a 1% requirement is satisfied.

Multiple Choice Questions with Answers

Question 1

Which activity primarily asks whether the equations and algorithms have been implemented correctly?

A. Validation

B. Verification

C. Calibration

D. Robustness analysis

Correct Answer: B. Verification

Question 2

Why should validation data generally be independent of calibration data?

A. To increase the number of model parameters

B. To eliminate all measurement noise

C. To assess predictive performance without fitting to the same observations

D. To guarantee that the model is physically complete

Correct Answer: C. To assess predictive performance without fitting to the same observations

Question 3

Measured values are 10, 20, 30 mm and predictions are 11, 19, 32 mm. What is the MAE?

A. 0.67 mm

B. 1.00 mm

C. 1.33 mm

D. 2.00 mm

Correct Answer: C. 1.33 mm

Question 4

Which metric gives greater influence to large residuals because residuals are squared?

A. Bias

B. MAE

C. RMSE

D. Median error

Correct Answer: C. RMSE

Question 5

First-order uncertainty propagation is most appropriate when the model is locally smooth and uncertainties are:

A. Small enough for a linear approximation

B. Always uniformly distributed

C. Known to be perfectly correlated

D. Larger than the nominal values

Correct Answer: A. Small enough for a linear approximation

Question 6

What does a residual pattern correlated with temperature most directly suggest?

A. The model may omit temperature dependence

B. The numerical solver is necessarily unstable

C. The sensor has zero uncertainty

D. The validation dataset is automatically invalid

Correct Answer: A. The model may omit temperature dependence

Question 7

A robustness requirement is written as P(q > q_max) ≤ α. What does α represent?

A. The nominal value of q

B. The permitted probability of violating the limit

C. The number of Monte Carlo samples

D. The calibration residual

Correct Answer: B. The permitted probability of violating the limit

Question 8

In Monte Carlo analysis, increasing the sample size generally helps to:

A. Remove model-form uncertainty completely

B. Guarantee correct input distributions

C. Improve stability of estimated output statistics

D. Make verification unnecessary

Correct Answer: C. Improve stability of estimated output statistics

Question 9

Which result is evidence of numerical convergence?

A. The model matches one experiment after calibration

B. The output approaches a stable value as the time step is reduced

C. The model has many adjustable parameters

D. The residual mean is exactly zero

Correct Answer: B. The output approaches a stable value as the time step is reduced

Question 10

A parameter is identifiable when:

A. It has the largest numerical value

B. It can be estimated uniquely or sufficiently precisely from available data

C. It is measured only once

D. It is excluded from sensitivity analysis

Correct Answer: B. It can be estimated uniquely or sufficiently precisely from available data