Statistical Analysis and Evidence-Based Decision Making

Introduction and context

Mechatronics research produces evidence from sensors, experiments, simulations, prototypes, manufacturing systems, and human–machine interactions. Statistical analysis converts observations into defensible conclusions while making uncertainty explicit. It helps an engineer decide whether a controller improves settling time, whether a vibration signature indicates a fault, whether a manufactured part meets tolerance, or whether an autonomous system performs reliably under changing conditions.

Statistics does not replace engineering judgment. Rather, it provides a disciplined language for separating signal from noise, quantifying variation, comparing alternatives, estimating unknown quantities, and communicating the strength of evidence. Evidence-based decision making combines statistical results with physical theory, safety requirements, cost, constraints, and practical significance.

Learning objectives

By the end of this chapter, a student should be able to:

1. Prepare, summarize, visualize, and critically inspect mechatronics data.
2. Select appropriate descriptive and inferential statistical methods.
3. Construct and interpret confidence intervals and hypothesis tests.
4. Apply correlation, simple and multiple regression, and analysis of variance.
5. Recognize when multivariate methods are useful and identify their assumptions.
6. Report uncertainty, effect size, statistical significance, and practical significance.
7. Use a structured evidence-based workflow for engineering decisions.
8. Identify ethical, reproducibility, and governance issues in statistical analysis.

1. Data, variables, and measurement structure

A population is the complete set of units about which a conclusion is intended; a sample is the observed subset. A unit may be a robot cycle, motor, manufactured component, test run, image, or operating episode. A parameter describes a population, whereas a statistic describes a sample.

Variables may be quantitative or qualitative. Quantitative variables include torque, temperature, energy consumption, and positioning error. Qualitative variables include controller type, fault class, and pass/fail status. A variable may also be classified as:

• Continuous: potentially measured on a continuum, such as voltage.
• Discrete: countable, such as the number of defects.
• Nominal: categories without order.
• Ordinal: ordered categories, such as low, medium, and high severity.
• Interval: differences are meaningful but zero is arbitrary, such as temperature in °C.
• Ratio: differences and ratios are meaningful, such as mass or elapsed time.

An experimental design distinguishes explanatory variables or factors from response variables. For example, sampling frequency and controller gain may be factors, while overshoot and energy use are responses. Confounding occurs when the effect of one factor cannot be separated from another. Randomization, blocking, replication, and controlled operating conditions reduce confounding.

Measurement quality

Accuracy describes closeness to the true or accepted value; precision describes repeatability. A sensor can be precise but inaccurate if it has a systematic bias. Random error varies unpredictably between observations, while systematic error shifts observations consistently. Calibration, traceability, resolution, sampling rate, and synchronization should be documented before analysis.

A measurement model can be written as:

where y is an observed measurement, μ is the measurand or true quantity, b is systematic bias, and ε is random error. The model is a conceptual decomposition, not a guarantee that all errors are additive or normally distributed. In dynamic systems, time dependence and autocorrelation may also be important.

2. Preparing and visualizing data

A defensible workflow begins with a data dictionary. It records variable names, meanings, units, permissible ranges, coding, sensor identity, sampling interval, missing-value codes, and collection conditions. Data should be preserved in an immutable raw form, with transformations recorded in reproducible scripts or notebooks.

Data preparation sequence

1. Inspect structure: identify observations, variables, repeated measures, and experimental groups.
2. Check units: convert values to a common, documented system.
3. Check validity: find impossible values, duplicated records, timestamp reversals, and sensor dropouts.
4. Address missingness: distinguish missing completely at random, missing at random, and missing not at random where possible. Do not silently replace missing values with zero.
5. Investigate outliers: verify whether an extreme value is an error, a rare but valid event, or evidence of a changed operating mode.
6. Create analysis variables: document filtering, aggregation, normalization, and transformations.
7. Split data appropriately: for predictive work, separate training, validation, and test data without allowing future information to leak into the past.

Deleting observations solely because they disagree with a desired conclusion is unacceptable. A removal rule should be defined from measurement or design considerations and reported transparently.

Graphical inspection

A histogram or density plot displays distribution shape. A box plot summarizes median, quartiles, and possible extreme observations. A scatter plot reveals association, nonlinearity, clusters, and changing variance. A time-series plot may expose drift, transients, autocorrelation, and regime changes. A control chart is useful for monitoring a stable process, but limits should not be confused with engineering specification limits.

A useful visualization includes units, a meaningful scale, sample size where relevant, and uncertainty or individual observations. Three-dimensional plots and dual axes should be used cautiously because they can obscure rather than clarify evidence.

3. Descriptive statistics

For observations x₁, x₂, …, xₙ, the arithmetic mean is:

Here x̄ is the sample mean and n is the number of observations. The mean is sensitive to extreme values. The sample median is the middle ordered observation, or the average of the two middle observations when n is even. The mode is the most frequent value.

The sample variance and standard deviation are:

The denominator n − 1, rather than n, gives the usual unbiased estimator of population variance when observations are independent and identically distributed. Standard deviation has the same units as x; variance has squared units. The range is maximum minus minimum, while the interquartile range is the third quartile minus the first quartile and is more resistant to outliers.

The coefficient of variation is:

It is meaningful mainly for ratio-scale variables with a positive, non-near-zero mean. It is dimensionless and can compare relative variability, but is inappropriate when the mean is near zero or negative.

Worked example: positioning error

Suppose five absolute positioning errors, in millimetres, are 1.2, 0.8, 1.0, 1.4, and 0.6. Their mean is:

x̄ = (1.2 + 0.8 + 1.0 + 1.4 + 0.6)/5 = 1.00 mm.

The deviations are 0.2, −0.2, 0, 0.4, and −0.4 mm. The squared deviations sum to 0.40 mm², so:

s² = 0.40/(5 − 1) = 0.10 mm²,

and s ≈ 0.316 mm. The sample describes a mean absolute error of 1.00 mm with a sample standard deviation of approximately 0.32 mm. It does not, by itself, establish performance for all future operating conditions.

4. Probability distributions and sampling variability

A probability model describes plausible variation before or across repeated observations. The normal distribution is often useful for measurement errors, but it should not be assumed automatically. Counts may follow a Poisson or binomial model, while lifetimes may be skewed and better represented by lognormal or Weibull models.

The standard error of the sample mean, under independent observations with finite variance, is:

Increasing n reduces the standard error at a rate proportional to 1/√n; it does not eliminate bias, dependence, or poor measurement. The central limit theorem explains why means may be approximately normal for sufficiently large samples under broad conditions, but the required sample size depends on distribution shape and dependence.

For time-series sensor data, adjacent observations may be correlated. Treating thousands of highly correlated samples as independent can produce artificially small standard errors and misleading p-values. Replication at the independent experimental-unit level is essential.

5. Estimation and confidence intervals

Point estimation gives a single estimate; interval estimation communicates precision. If a population mean is estimated from independent approximately normal observations with unknown standard deviation, a two-sided confidence interval is:

Here α is the significance level, t₁₋α/₂,n₋₁ is the relevant Student t critical value, and n − 1 is the degrees of freedom. The interval estimates the population mean under the stated sampling model. A 95% confidence procedure captures the true parameter in 95% of repeated samples in the long run; it does not mean there is a 95% probability that a fixed parameter lies in this particular interval under the frequentist interpretation.

For a proportion p estimated by p̂, large-sample intervals based on the normal approximation can perform poorly for small samples or proportions near 0 or 1. Wilson or exact methods are often preferable. For complex designs, bootstrap intervals may be useful, provided resampling respects independent experimental units and dependence structure.

Worked example: mean response

A robot’s settling time is measured in 16 independent trials, giving x̄ = 2.40 s and s = 0.32 s. The standard error is 0.32/√16 = 0.08 s. With 15 degrees of freedom, t₀.₉₇₅,₁₅ is approximately 2.131. Thus the 95% interval is:

2.40 ± 2.131(0.08) = 2.40 ± 0.170 s,

or approximately (2.23, 2.57) s. This interval describes uncertainty in the mean settling time, not the range of individual future settling times.

6. Hypothesis testing

A hypothesis test compares data with a null model. The null hypothesis H₀ commonly represents no difference or a specified benchmark, while H₁ represents a difference or directional effect. A test statistic measures how far the data are from H₀ relative to expected sampling variation.

A p-value is the probability, assuming H₀ and the model are true, of obtaining a result at least as extreme as the observed result. It is not the probability that H₀ is true, nor the probability that the result occurred by chance in an informal sense.

The significance level α is selected before analysis when possible. Rejecting H₀ when it is true is a Type I error; failing to reject H₀ when a specified alternative is true is a Type II error. Statistical power is the probability of detecting an effect of a specified size under the alternative model. Power depends on effect size, variability, sample size, design, and α.

A two-sample comparison may use a pooled-variance t test when independent groups have approximately normal responses and equal variances. Welch’s t test does not require equal variances and is often a safer default. A paired test is appropriate when each observation under one condition is naturally matched with an observation under another, such as the same robot tested before and after calibration.

A non-significant result does not prove equivalence. If demonstrating that two controllers are sufficiently similar, use an equivalence test with a pre-specified engineering margin Δ. Practical equivalence requires the confidence interval for the difference to lie within (−Δ, Δ).

Multiple comparisons

Testing many hypotheses increases the chance of at least one false positive. Remedies include pre-specifying primary outcomes, adjusting familywise error using methods such as Bonferroni or Holm, or controlling the false discovery rate when screening many signals. Adjustments should reflect the scientific question rather than being applied mechanically.

7. Effect size and practical significance

Statistical significance depends strongly on sample size. An effect can be statistically significant yet too small to matter in operation; a practically important effect can fail to reach significance in a small study.

For two means, a raw difference retains engineering units. A standardized effect such as Cohen’s d is:

where sₚ is a pooled standard deviation under the equal-variance formulation. Standardized effects aid comparison across studies but do not replace domain-specific units. Engineers should report the estimated effect, confidence interval, baseline, cost, safety margin, and a minimally important difference whenever possible.

8. Correlation and regression

Pearson’s correlation coefficient is:

It measures linear association and lies between −1 and 1. Correlation does not establish causation; a third variable, reverse causality, selection, or common time trend may explain an association.

Simple linear regression models a conditional mean:

Here yᵢ is the response, xᵢ is the predictor, β₀ is the intercept, β₁ is the expected change in y per unit increase in x, and εᵢ is an error term with mean zero under the usual model. Ordinary least squares estimates coefficients by minimizing the sum of squared residuals:

Important assumptions include appropriate functional form, independent errors, constant error variance for conventional intervals, and no influential observations that invalidate interpretation. Normality of errors is most important for small-sample inference, not for calculating fitted values alone.

Multiple regression extends the model:

A coefficient represents an adjusted association, conditional on the other included predictors. Multicollinearity can make coefficients unstable even when prediction is adequate. Residual plots, leverage, influence diagnostics, cross-validation, and domain knowledge should accompany model fitting. Extrapolation beyond the observed predictor range is especially risky.

9. Analysis of variance

Analysis of variance (ANOVA) compares means across multiple groups while partitioning variability. For a one-way design:

where SST is total sum of squares, SSB is between-group sum of squares, and SSW is within-group sum of squares. The F statistic is:

Here k is the number of groups and N is the total number of observations. A significant omnibus F test indicates that not all group means are equal; it does not identify which means differ. Planned contrasts or multiplicity-adjusted post-hoc comparisons are then required.

Two-way ANOVA can assess two factors and their interaction. An interaction means the effect of one factor depends on the level of another. For example, a control algorithm may perform differently at low and high payloads. Randomization, independent errors, approximately equal variance, and an appropriate response scale are important assumptions. Transformations or generalized linear models may be preferable for non-normal outcomes such as counts or binary pass/fail results.

10. Multivariate methods

Mechatronics systems often generate many correlated variables. Multivariate methods can reduce dimensionality or classify observations, but they can also create overfitting and interpretation problems.

Principal component analysis (PCA) transforms centered, and sometimes standardized, variables into orthogonal linear combinations called principal components. The first component explains the greatest variance subject to unit-length constraints; subsequent components explain the greatest remaining variance while being orthogonal to previous components. PCA is descriptive and unsupervised: a component explaining high variance is not necessarily the component most relevant to failure.

Cluster analysis groups observations according to a selected distance or similarity measure. Results depend on scaling, distance, algorithm, and the chosen number of clusters. Clusters should be validated against physical knowledge or independent data.

Discriminant analysis, logistic regression, and classification trees can predict classes such as fault/no fault. Performance should be evaluated using held-out data and metrics appropriate to consequences: sensitivity, specificity, precision, recall, calibration, and expected cost. Accuracy alone can be misleading for imbalanced faults.

11. Uncertainty propagation and reporting

Uncertainty should be reported with estimates, units, coverage or confidence level, and the method used. For a function z = f(x₁,…,xₖ), a first-order approximation gives:

Here u(xᵢ) is the standard uncertainty of input xᵢ, and Cov denotes covariance. This approximation assumes small uncertainties and a sufficiently smooth function. Correlations must not be ignored when inputs share a calibration source or measurement process.

For independent variables, z = x + y has variance approximately u²(z) = u²(x) + u²(y). For a product z = xy, relative variance is approximately:

Monte Carlo propagation is useful for nonlinear models, bounded variables, and non-normal inputs. Report whether uncertainty is standard uncertainty, an expanded uncertainty, a confidence interval, or a prediction interval; these are not interchangeable.

A complete result might state: “Mean settling time was 2.40 s (95% confidence interval 2.23–2.57 s; n = 16 independent trials).” Avoid excessive decimal places unsupported by instrument resolution or uncertainty.

12. Evidence-based decision workflow

A robust engineering decision follows these stages:

1. Define the decision: specify the action, population, response, constraints, and practical threshold.
2. Formulate the estimand: decide whether the target is a mean difference, failure probability, quantile, causal effect, or prediction error.
3. Design data collection: select independent units, randomization, blocking, replication, sample size, and measurement procedures.
4. Inspect and clean: preserve raw data and document all exclusions and transformations.
5. Analyze: use methods whose assumptions match the design and outcome.
6. Quantify uncertainty: provide confidence or prediction intervals and sensitivity analyses.
7. Assess practical consequences: compare effect size with tolerance, safety, cost, energy, and maintainability.
8. Decide and monitor: record the rationale, implement the action, and monitor performance after deployment.

A decision matrix can combine evidence with engineering constraints. For safety-critical systems, a statistically favorable average cannot compensate for an unacceptable tail risk or failure mode.

Applied case: selecting a controller for a precision gantry

This is an illustrative case, not a report of an actual experiment.

Background and problem

A laboratory compares Controller A, the current proportional–derivative controller, with Controller B, which adds feedforward compensation. The engineering requirement is a mean positioning error below 0.50 mm, with no unacceptable increase in energy consumption. Sixteen independent motion cycles are assigned randomly to each controller under matched payload and environmental conditions.

Analysis

The primary response is absolute positioning error. The researchers pre-specify the mean difference B − A and a practical improvement margin of 0.10 mm. They record cycle-level error, settling time, energy, payload, temperature, and fault status. Time samples within a cycle are not treated as independent experimental units; each cycle contributes one summarized response.

They inspect dot plots and residuals, verify calibration records, and find no missing cycles. The mean errors are 0.62 mm for A and 0.43 mm for B. The estimated difference is −0.19 mm. A confidence interval for the difference is calculated using an appropriate two-sample method, with variance equality checked rather than assumed. The interval excludes zero, suggesting evidence of a difference, and its magnitude exceeds the pre-specified practical margin. Energy increases by 3%, with a confidence interval that includes the laboratory’s acceptable 5% limit.

Approach and decision

The team does not base the decision on the p-value alone. It considers the confidence interval, requirement threshold, energy cost, repeatability, and possible payload interaction. A follow-up blocked experiment across low, medium, and high payloads is planned because the initial study covers only one payload. Controller B is recommended for a controlled pilot, conditional on confirming performance across payloads and verifying that the fault rate does not increase.

Learning points

The case illustrates estimand definition, independent replication, practical significance, uncertainty reporting, and staged deployment. It also shows why a statistically better average is insufficient when operating conditions are narrow or safety-related outcomes are not examined.

Limitations, ethics, and contemporary practice

Statistical conclusions are conditional on sampling, measurement, model, and context. Common weaknesses include pseudoreplication, selective reporting, p-hacking, underpowered studies, post hoc subgroup analysis, data leakage, unvalidated transformations, and confusing prediction with explanation. Sensitivity analysis should examine how conclusions change under plausible alternative assumptions.

Ethical analysis requires honest reporting of exclusions, failed trials, uncertainty, and adverse results. Data from operators or users may require informed consent, privacy protection, access control, and minimization of personally identifiable information. Automated decisions should be auditable, and model performance should be examined across relevant operating groups when unequal impacts are possible.

AI and machine learning can assist anomaly detection, feature extraction, surrogate modeling, and predictive maintenance. However, a high-performing model may exploit leakage, environmental shortcuts, or unstable correlations. Engineers should use version control, data and model documentation, reproducible pipelines, calibration checks, uncertainty estimates, and independent test sets. Statistical significance cannot validate a physically impossible model, and machine-learning accuracy cannot by itself establish causal improvement.

Best practice is to preregister important hypotheses when feasible, use power or precision planning, distinguish exploratory from confirmatory analysis, report effect sizes and intervals, share code and metadata where permitted, and retain a clear chain from measurement to decision.

Chapter Summary

Statistical analysis provides the framework for turning mechatronics observations into evidence. Sound work begins with a well-defined population, estimand, measurement process, and experimental design. Data must be checked for units, missingness, dependence, outliers, and leakage before descriptive statistics and visualizations are interpreted. Means, medians, variance, standard deviation, and standard errors summarize different aspects of data and must not be confused.

Confidence intervals quantify estimation precision, while hypothesis tests evaluate compatibility with a null model. P-values do not measure effect importance or prove hypotheses. Effect sizes, engineering thresholds, uncertainty, cost, safety, and operating context determine practical significance. Regression models association or prediction under assumptions that require diagnostic checking; ANOVA partitions variation and can reveal factor interactions. Multivariate methods help analyze correlated signals but require careful scaling, validation, and interpretation.

Evidence-based decisions combine statistical results with domain knowledge and explicit consequences. Reproducibility, ethical data handling, transparent exclusions, independent validation, and post-deployment monitoring are essential, especially when AI or safety-critical systems are involved.

Key Terms

Population: Complete set of units about which inference is intended.

Sample: Observed subset of a population.

Parameter: Numerical characteristic of a population.

Statistic: Numerical characteristic calculated from a sample.

Accuracy: Closeness of a measurement to an accepted true value.

Precision: Degree of repeatability among measurements.

Random error: Unpredictable variation between measurements.

Systematic error: Consistent bias in a measurement process.

Standard deviation: Measure of dispersion in the original units of a variable.

Standard error: Estimated sampling variability of a statistic.

Confidence interval: Interval produced by a procedure with specified long-run coverage.

P-value: Probability, under the null model, of a result at least as extreme as observed.

Type I error: Rejection of a true null hypothesis.

Power: Probability of detecting a specified effect under an alternative model.

Effect size: Quantified magnitude of a difference or association.

Regression: Modeling framework relating a response to one or more predictors.

ANOVA: Method partitioning variation to compare group means and factors.

Multicollinearity: Strong dependence among predictors in a regression model.

Pseudoreplication: Treating non-independent observations as independent replicates.

Practical significance: Importance of an effect for engineering performance or decisions.

Review Questions

Short Answer Questions

1. Distinguish accuracy from precision and give one mechatronics example of each.
2. Why should time samples within one robot cycle not automatically be treated as independent observations?
3. What does a 95% confidence interval mean in repeated-sampling terms?
4. State one difference between statistical significance and practical significance.
5. What does an interaction in a two-factor ANOVA indicate?

Long Answer Questions

1. Describe a complete workflow for preparing sensor data for inferential analysis, including missing values, outliers, units, and reproducibility.
2. Explain the assumptions and interpretation of a linear regression model used to predict energy consumption from motor speed and payload.
3. Compare a paired t test, an independent-samples test, and one-way ANOVA, giving a suitable mechatronics application for each.
4. Explain how uncertainty propagation differs from confidence-interval estimation and identify situations where Monte Carlo propagation is useful.
5. Discuss how statistical analysis should be integrated with safety, cost, and maintainability when selecting a controller.

Analytical / Application Questions

1. Two controllers are tested on the same ten motion trajectories, once each. Explain which comparison design is appropriate, identify a suitable estimand, and state how you would report both statistical and practical significance.
2. A classifier detects 98% of normal cycles but only 60% of rare faults. Analyze why accuracy may be misleading and recommend performance measures and a validation strategy.
3. A regression model has a high R² but poor performance on a new payload range. Diagnose possible causes and propose at least three corrective actions.

Multiple Choice Questions with Answers

Question 1

Which statistic is measured in the same units as the original variable?

A. Variance

B. Standard deviation

C. P-value

D. Coefficient of variation

Correct Answer: B. Standard deviation

Question 2

A 95% confidence interval is best interpreted as:

A. There is a 95% probability that the fixed parameter is inside this calculated interval

B. 95% of individual future observations must lie inside the interval

C. A procedure that captures the true parameter in 95% of repeated samples under its assumptions

D. The probability that the null hypothesis is true is 0.95

Correct Answer: C. A procedure that captures the true parameter in 95% of repeated samples under its assumptions

Question 3

Which design is most appropriate when the same robot is tested before and after calibration?

A. Paired comparison

B. One-way ANOVA with unrelated groups

C. Independent Poisson test

D. Unpaired comparison that ignores matching

Correct Answer: A. Paired comparison

Question 4

In a linear regression model, the coefficient of a predictor represents:

A. The unconditional probability that the predictor causes the response

B. The expected response change per unit predictor change, conditional on the model and other predictors

C. The standard deviation of every individual response

D. The percentage of observations that are outliers

Correct Answer: B. The expected response change per unit predictor change, conditional on the model and other predictors

Question 5

If many hypotheses are tested without adjustment, what generally increases?

A. The probability of at least one false positive

B. Measurement accuracy

C. Physical repeatability

D. The engineering tolerance

Correct Answer: A. The probability of at least one false positive

Question 6

A significant omnibus ANOVA F test establishes that:

A. Every group mean differs from every other group mean

B. All group variances are equal

C. At least one group mean differs under the model

D. The largest group has the highest mean

Correct Answer: C. At least one group mean differs under the model

Question 7

For the five positioning errors in the worked example, the sample mean is:

A. 0.32 mm

B. 0.40 mm

C. 1.00 mm

D. 1.40 mm

Correct Answer: C. 1.00 mm

Question 8

Which statement about Pearson correlation is correct?

A. It proves that one variable causes the other

B. It measures linear association and does not by itself establish causation

C. It can only take positive values

D. It is unaffected by outliers and nonlinear patterns

Correct Answer: B. It measures linear association and does not by itself establish causation

Question 9

Why is pseudoreplication problematic?

A. It always increases sensor resolution

B. It treats dependent observations as independent and can make uncertainty appear too small

C. It prevents all visualization

D. It guarantees conservative p-values

Correct Answer: B. It treats dependent observations as independent and can make uncertainty appear too small

Question 10

A practically important effect that is not statistically significant in a small study should generally lead to:

A. Automatic proof that no effect exists

B. Deletion of inconvenient observations

C. Consideration of precision, power, confidence intervals, and further data collection

D. Reporting only the p-value

Correct Answer: C. Consideration of precision, power, confidence intervals, and further data collection