Experimental Design for Mechatronic Systems

Introduction and Context

Mechatronic systems combine mechanical structures, sensors, actuators, electronics, embedded software, control algorithms, and human or environmental interfaces. Their performance is therefore influenced by many interacting variables. A robotic arm, for example, may be affected simultaneously by payload, joint speed, controller gain, friction, temperature, supply voltage, and trajectory. Changing one factor at a time often gives incomplete or misleading conclusions because factors can interact.

Experimental design provides a systematic method for learning from physical or simulated experiments. It specifies what will be varied, what will be measured, how observations will be collected, and how uncertainty will be controlled. In mechatronics, it supports design verification, controller tuning, reliability improvement, energy optimization, calibration, manufacturing development, and research.

A sound experiment is not simply a set of measurements. It is a planned investigation in which changes in a response can be attributed, as far as reasonably possible, to deliberately varied factors rather than uncontrolled influences.

Learning Objectives

After studying this chapter, a student should be able to:

• distinguish experimental factors, levels, responses, treatments, and nuisance variables;
• formulate hypotheses and define controlled experimental conditions;
• explain randomization, replication, blocking, and confounding control;
• construct and interpret full and fractional factorial designs;
• use analysis of variance conceptually and quantitatively;
• apply response-surface methods to optimization;
• reason about sample size, measurement uncertainty, and practical significance;
• design a defensible experiment for a mechatronic system;
• identify ethical, safety, data-integrity, and reproducibility requirements.

1. The Logic of a Controlled Experiment

An experiment begins with a research question that can be answered through intervention. A useful question is specific about the system, the input variables, the response, and the operating range. “How can the robot be improved?” is too broad. “How do payload and velocity affect positioning error of the robot under a specified trajectory?” is experimentally actionable.

The principal elements are:

A controlled experiment normally includes a baseline or reference condition. A control may be an untreated system, a standard controller, a nominal parameter setting, or a benchmark design. The control does not need to be “no treatment”; it must provide a meaningful comparison.

Variables and operational definitions

An abstract construct must be converted into an operational definition. “Accuracy” might mean absolute position error at a specified target, while “repeatability” might mean the spread of positions obtained from repeated commands. A response should include units, sampling rate, calculation method, and time window. For example, root-mean-square error over a 10 s steady-state interval is more reproducible than “tracking quality.”

For measured errors e₁, e₂, …, eₙ, the root-mean-square error is

where n is the number of samples and eᵢ is the error at sample i, in the relevant measurement unit. RMSE is appropriate when larger errors should be penalized more strongly; it is not identical to mean absolute error.

Hypotheses

A null hypothesis states that specified effects are absent, while an alternative hypothesis states that they are present. For example, H₀ may state that mean positioning error is equal for two controller settings. A practical hypothesis may be directional, but statistical testing should not replace engineering reasoning. A small effect can be statistically significant with a large sample and still be irrelevant to system performance.

2. Sources of Variation and Measurement Systems

Observed variation can arise from:

1. Treatment variation: changes caused by deliberately varied factors.
2. Measurement variation: sensor noise, quantization, calibration error, and data-processing choices.
3. System variation: mechanical backlash, battery state, thermal drift, and component-to-component differences.
4. Environmental variation: temperature, vibration, lighting, electromagnetic interference, and humidity.
5. Unmodeled variation: influences not identified during planning.

Before designing a large experiment, assess the measurement system. A measurement system should be sufficiently accurate, precise, stable, and traceable for the engineering decision. Repeating a highly noisy measurement does not automatically create accurate knowledge; replication reduces uncertainty in estimated means, but systematic bias remains.

Calibration should use traceable standards where available. Sensor range, resolution, bandwidth, latency, saturation, and synchronization should be documented. In a multi-sensor mechatronic system, timestamps and coordinate frames must be aligned. Data should be retained in raw form as well as processed form.

3. Randomization, Replication, and Blocking

Randomization

Randomization assigns treatment runs by a chance mechanism. It prevents time trends, operator expectations, warm-up effects, and unknown influences from being systematically associated with a treatment. Randomization does not mean careless operation: safety constraints and physical feasibility remain mandatory.

If a robot is tested at low speed early in the day and high speed late in the day, speed may be confounded with temperature or battery depletion. Randomizing run order distributes such effects more fairly among treatments. When complete randomization is impossible, restricted randomization or randomized blocks should be used.

Replication

Replication means obtaining independent experimental units under the same treatment, not merely recording many samples from one run. One motion trial sampled at 1 kHz does not provide 1000 independent trials. Those samples may be autocorrelated.

Replication estimates random error and improves precision. If independent observations have standard deviation σ, the standard error of their mean is

where ȳ is the sample mean and n is the number of independent observations. This relationship assumes independence and a common variance. Increasing n from 4 to 16 halves the standard error, but increasing n also requires time, energy, and equipment capacity.

Blocking

A block is a group of experimental units that are relatively similar with respect to a nuisance variable. Treatments are then compared within blocks. For example, each test day may be a block, with all controller settings represented on every day. Blocking removes part of the day-to-day variation from the treatment comparison.

Blocking is especially valuable for battery state, operator, machine identity, material batch, ambient temperature, and production lot. A block should be chosen because it explains variation, not merely because it is convenient.

4. One-Factor Experiments and Analysis of Variance

A one-factor experiment compares levels of one factor. Suppose k controller settings are tested, with nᵢ observations at level i. A one-way analysis of variance partitions total variation into variation between treatment means and variation within treatments:

The corresponding mean squares are

and

where N = ∑nᵢ is the total number of observations. The F statistic is

Under the usual fixed-effects assumptions—independent errors, approximately normal errors, and equal error variance under the null hypothesis—large values of F provide evidence that not all treatment means are equal. ANOVA indicates that a difference exists; post-hoc comparisons or planned contrasts are needed to identify which means differ.

For a randomized complete block design with treatment i in block j, a conceptual model is

where yᵢⱼ is the response, μ is the grand mean, τᵢ is the treatment effect, βⱼ is the block effect, and εᵢⱼ is random error. The model assumes that treatment and block effects are additive unless an interaction is deliberately included.

5. Factorial Designs and Interaction

A factorial design varies two or more factors together. In a 2² design, two factors each have two levels, producing four treatment combinations. For factors A and B, the model may be written as

Here β₀ is the intercept, β_A and β_B are main-effect coefficients, β_AB is the interaction coefficient, x_A and x_B are coded factor values, and ε is random error. Coded levels are commonly −1 and +1, although actual physical values must be reported separately.

A main effect is the average change caused by one factor across the levels of the other factor. An interaction occurs when the effect of one factor depends on the level of another. In a robot, increasing velocity may have little effect at low payload but a large effect at high payload. Reporting only average velocity effects would conceal this engineering behavior.

For a balanced 2² design with responses at combinations (−,−), (+,−), (−,+), and (+,+), the interaction contrast is proportional to

The exact estimated coefficient depends on the number of replicates and the coding convention. The important interpretation is the difference of differences.

Worked example: a 2² robotic experiment

A positioning-error study varies speed and payload:

The speed effect at low payload is 1.00 − 0.80 = 0.20 mm. At high payload it is 2.00 − 1.20 = 0.80 mm. Since these effects differ, speed and payload interact. The high-payload condition magnifies the effect of speed; therefore, selecting a speed without considering payload is not justified.

Full and fractional factorial designs

A full factorial design includes every combination of factor levels. It is efficient for detecting interactions, but the number of runs grows rapidly. With k two-level factors, the number of combinations is

Thus, five two-level factors require 32 combinations before replication or centre points. Fractional factorial designs use a carefully selected subset of combinations. They reduce cost but create aliasing: some effects cannot be distinguished from others. A fractional design is defensible only when the alias structure is understood and higher-order interactions are considered less likely or less important.

Screening designs help identify a small number of influential factors from many candidates. Follow-up experiments should then estimate important interactions and optimize the response.

6. Response-Surface Methodology

When the objective is optimization rather than simple comparison, response-surface methodology (RSM) models a response over a region of quantitative factors. A second-order model is commonly written as

The linear terms describe local slopes, squared terms describe curvature, and cross-product terms describe interactions. RSM is useful after screening has identified a manageable number of factors.

A central composite design generally combines factorial points, axial points, and centre points. A Box–Behnken design uses combinations within the interior of the factor region and can be useful when extreme combinations are unsafe. The chosen design must respect physical constraints: a mathematically convenient design is unacceptable if it commands destructive temperatures, unstable controller gains, or unsafe motion.

Optimization should consider multiple responses. For example, minimum energy may conflict with minimum positioning error and minimum cycle time. A desirability function or Pareto analysis can make these trade-offs explicit, but the final choice remains an engineering decision informed by constraints.

7. Sample-Size Reasoning and Uncertainty

Sample size should be justified before data collection. Relevant quantities include significance level α, desired power 1 − β, expected variability, and the smallest practically important difference Δ.

For a simple comparison of two independent means with equal sample sizes, a commonly used planning approximation is

where n is the number of observations per group, σ is the assumed common standard deviation, Δ is the practically important difference, and z_q is the q-quantile of the standard normal distribution. This approximation assumes independent observations, approximately normal errors, equal variance, and a two-sided comparison. The result should be rounded upward and increased for attrition, blocking, multiple comparisons, or model complexity.

Worked sample-size example

Suppose a controller study assumes σ = 0.40 mm, seeks to detect Δ = 0.50 mm, uses α = 0.05 and power 0.80. Using z₀.₉₇₅ ≈ 1.96 and z₀.₈ ≈ 0.84:

Since 0.40² = 0.16 and (2.80)² = 7.84,

Thus, at least 11 independent observations per group would be required by this approximation. In practice, the researcher may select 12 or more per group to accommodate failed runs and model checks. If the observations are repeated measurements from the same physical unit, a repeated-measures or mixed-effects design may be more appropriate than this independent-groups formula.

Confidence intervals are often more informative than p-values. A mean difference reported with a 95% confidence interval communicates both estimated effect and precision. Statistical significance does not establish practical importance, causality beyond the design, or generalizability outside the tested region.

8. Confounding, Bias, and Validity

Two factors are confounded when their effects cannot be separated in the design. If all tests with controller A use one motor and all tests with controller B use another, controller and motor identity are confounded. If a factor always increases as temperature increases, its effect is confounded with temperature.

Common controls include:

• randomizing run order;
• balancing factor combinations across time and equipment;
• blocking on known nuisance variables;
• calibrating sensors before testing;
• using identical mechanical fixtures;
• recording environmental and operational covariates;
• separating exploratory and confirmatory experiments;
• repeating critical findings on independent equipment.

Internal validity concerns whether the treatment caused the observed response under the experiment. External validity concerns whether conclusions transfer to other loads, machines, environments, or operating ranges. Construct validity concerns whether the selected response truly represents the engineering concept of interest.

Data leakage is a related concern in experiments involving machine learning. If test data influence controller selection or model tuning, the apparent performance of the final system is optimistic. Training, validation, and final test data must be separated by time, unit, or operating scenario as appropriate.

9. Planning and Conducting a Mechatronic Experiment

A practical protocol should contain:

1. research question and hypotheses;
2. system boundary and unit of experimentation;
3. factors, levels, constraints, and excluded regions;
4. response definitions and sensor specifications;
5. design matrix and randomization procedure;
6. warm-up, initialization, and reset procedures;
7. calibration and synchronization plan;
8. replication and stopping rules;
9. data-quality checks and failure handling;
10. analysis model specified before examining outcomes;
11. safety controls and emergency procedures;
12. version-controlled code, configuration files, and raw data.

A pilot experiment is not a substitute for the main study. Its purpose is to reveal feasibility problems, estimate variability, check sensor range, identify unsafe conditions, and refine the protocol. Pilot data should not be silently combined with confirmatory data when procedures have changed.

10. Applied Case: Thermal Optimization of an Autonomous Mobile Robot

The following is an illustrative case, not a report of a particular commercial product.

Background and problem

An autonomous mobile robot experiences excessive controller-computation temperature during long missions. The engineering team wants to reduce temperature without increasing navigation error or reducing mission throughput.

Analysis

Candidate factors are processor frequency, cooling-fan duty cycle, route-planning update interval, and payload. Ambient temperature and battery state are recognized nuisance variables. Responses are processor temperature, navigation-position error, energy consumption, and mission completion time.

A preliminary screening experiment uses a fractional factorial design for the four candidate factors, with ambient temperature treated as a block. Run order is randomized within each block. Fan settings that could cause electrical overload are excluded before experimentation.

Approach and solution

Screening indicates that processor frequency, fan duty cycle, and payload are important, while the update interval has a smaller effect in the tested region. A follow-up response-surface design varies the three important factors. Centre points assess curvature and provide a check on process stability. Replicated centre runs are distributed across the test days to detect thermal drift.

The team fits a second-order model for temperature and separate models for error and energy. Diagnostic plots show increasing residual variance at high payload, so a transformed response and weighted analysis are considered. The final operating point is not the temperature minimum alone; it must satisfy maximum navigation error and mission-time constraints.

Outcome

The selected setting reduces temperature while maintaining the required navigation accuracy. Validation runs on a second robot and at a different ambient temperature produce results within the prediction interval of the model. The team documents the operating envelope rather than claiming universal optimality.

Learning points

The case demonstrates why multiple responses, blocking, replication, centre points, safety constraints, validation, and generalization are necessary. It also illustrates that optimization is a constrained decision problem, not merely the search for the smallest value of one response.

11. Ethics, Safety, and Reproducibility

Mechatronics experiments can injure people, damage equipment, or create unsafe autonomous behavior. Risk assessment should precede testing. Emergency stops, guarding, restricted access, safe failure states, electrical protection, and software interlocks are engineering requirements, not optional documentation.

Researchers must not discard inconvenient observations without a predefined rule, alter data to fit a hypothesis, or report only successful trials. Failed runs should be classified using transparent criteria. Any exclusion must be justified and recorded.

Experiments involving human operators require informed consent, privacy protection, and institutional approval where applicable. Video, biometric data, motion traces, and workplace measurements may be personally identifiable. Cybersecurity is also relevant when networked devices, cloud dashboards, or remotely controlled robots are used.

Contemporary laboratories increasingly use digital twins, automated experiment orchestration, and AI-assisted design. These tools can search design spaces efficiently, but automated recommendations require validation against physical reality. AI-generated analyses should be traceable to data, code, model assumptions, and version history. A digital twin should not be treated as an experimental substitute unless its fidelity has been established for the intended decision.

Best-Practice Checklist

• Define the experimental unit and response before collecting data.
• Vary factors deliberately over a justified and safe range.
• Randomize runs and block important nuisance variables.
• Replicate independent units rather than merely oversampling one run.
• Include interactions when engineering reasoning suggests them.
• Preserve raw data, metadata, code, and instrument settings.
• Inspect residuals and measurement quality, not only p-values.
• Report effect sizes, uncertainty, practical limits, and failed runs.
• Validate important conclusions on new runs, units, or environments.
• Treat safety, privacy, cybersecurity, and reproducibility as part of experimental quality.

Chapter Summary

Experimental design gives mechatronics engineers a disciplined way to establish cause-and-effect relationships in complex systems. Its essential elements are clearly defined factors and responses, controlled conditions, randomization, replication, and blocking. Factorial designs reveal main effects and interactions more efficiently than one-factor-at-a-time experimentation. ANOVA partitions variation and supports comparisons, while response-surface methods model curvature and guide constrained optimization. Sample-size planning links variability, desired power, and practical effect size. Confounding, measurement bias, autocorrelation, and data leakage can invalidate otherwise sophisticated analyses. Safe, ethical, reproducible experimentation requires documented protocols, calibrated instruments, transparent data handling, and validation beyond the original test conditions.

Key Terms

Experimental unit: The smallest unit receiving a treatment.

Factor: An input variable deliberately varied by the researcher.

Level: A specified value or category of a factor.

Treatment: A particular combination of factor levels.

Response: A measured outcome of an experiment.

Nuisance variable: An uncontrolled or secondary variable that affects the response.

Randomization: Assignment of treatments by a chance mechanism.

Replication: Independent repetition of an experimental treatment.

Blocking: Grouping similar experimental units to reduce nuisance variation.

Confounding: An inability to separate the effects of two or more influences.

Interaction: A condition in which the effect of one factor depends on another factor.

Factorial design: A design that studies combinations of factor levels.

ANOVA: Analysis of variance used to partition and compare sources of variation.

Residual: The difference between an observed and model-predicted response.

Response-surface methodology: Modeling and optimization over quantitative factor regions.

Power: The probability of detecting a specified effect when it exists.

Practical significance: Importance of an effect for engineering or operational decisions.

Measurement system: Instruments, procedures, software, and people producing measurements.

External validity: The extent to which findings generalize beyond the experiment.

Reproducibility: The ability to obtain consistent results using documented data, methods, and code.

Review Questions

Short Answer Questions

1. Distinguish an experimental unit, factor, level, treatment, and response using a mechatronic example.
2. Why is randomization important in a time-dependent experiment?
3. Explain the difference between replication and repeated sampling within one run.
4. What is an interaction between two factors?
5. State two ways in which confounding can be controlled.

Long Answer Questions

1. Explain how a randomized complete block design could be used to test controller settings across several test days.
2. Compare full factorial, fractional factorial, and response-surface designs in terms of purpose and limitations.
3. Describe the assumptions and interpretation of a one-factor ANOVA.
4. Discuss measurement-system quality and its effect on experimental conclusions.
5. Develop a complete experimental protocol for evaluating a sensor-fusion algorithm in an autonomous vehicle.

Analytical / Application Questions

1. A robot is tested at two speeds and two payloads. The speed effect is 0.2 mm at low payload and 0.8 mm at high payload. Interpret the interaction and explain its design implication.
2. A researcher tests controller A only in the morning and controller B only in the afternoon. Identify possible confounding variables and propose a revised design.
3. An experiment must optimize temperature, energy, and positioning error simultaneously. Propose a response-surface strategy and explain how constraints and validation should be handled.

Multiple Choice Questions with Answers

Question 1

Which feature most directly distinguishes replication from repeated sampling?

A. Replication uses a larger sampling frequency.

B. Replication uses independent experimental units or runs.

C. Replication always uses different factor levels.

D. Replication eliminates systematic measurement bias.

Correct Answer: B. Replication uses independent experimental units or runs.

Question 2

A nuisance variable is controlled by grouping similar experimental units. What is this procedure called?

A. Blocking

B. Aliasing

C. Screening

D. Coding

Correct Answer: A. Blocking

Question 3

In a 2² factorial design, how many treatment combinations are present?

A. 2

B. 3

C. 4

D. 8

Correct Answer: C. 4

Question 4

If the effect of speed is much larger at high payload than at low payload, the factors speed and payload exhibit what?

A. Replication

B. Interaction

C. Calibration

D. Random error only

Correct Answer: B. Interaction

Question 5

In the model y = β₀ + β_Ax_A + β_Bx_B + β_ABx_Ax_B + ε, what does β_AB represent?

A. The intercept

B. The speed main effect

C. The interaction coefficient

D. The measurement unit

Correct Answer: C. The interaction coefficient

Question 6

For independent observations with standard deviation σ, how does the standard error of the mean change when sample size increases from 4 to 16?

A. It doubles.

B. It is unchanged.

C. It is halved.

D. It becomes four times larger.

Correct Answer: C. It is halved.

Question 7

Which design is especially associated with modeling curvature and optimizing quantitative factors?

A. Response-surface design

B. Case-control design

C. Simple convenience sample

D. Post-test-only observation

Correct Answer: A. Response-surface design

Question 8

A controller is always tested with motor 1, while another controller is always tested with motor 2. What is the principal methodological problem?

A. Blocking is excessive.

B. Controller and motor identity are confounded.

C. The response has too many units.

D. The design has automatic replication.

Correct Answer: B. Controller and motor identity are confounded.

Question 9

What is the primary purpose of randomizing experimental run order?

A. To guarantee a normal response distribution

B. To remove the need for calibration

C. To distribute time-related and unknown influences across treatments

D. To increase the number of factor levels

Correct Answer: C. To distribute time-related and unknown influences across treatments

Question 10

In sample-size planning, reducing the practically important difference Δ while holding other quantities fixed generally requires what?

A. Fewer observations

B. More observations

C. No observations

D. Only qualitative observations

Correct Answer: B. More observations