Machine Learning Models and Evaluation

Introduction and Context

Machine learning (ML) enables a mechatronic system to infer relationships from data rather than relying exclusively on manually derived rules. In a robotic arm, for example, a model may estimate joint torque from position, velocity and acceleration; classify whether a bearing is healthy or defective; or group operating conditions into previously unknown regimes. The engineering value of ML depends not only on model accuracy, but also on data quality, physical plausibility, computational cost, uncertainty, safety and maintainability.

This chapter develops the principal supervised and unsupervised learning models used in mechatronics: regression, classification, clustering, dimensionality reduction and ensemble methods. It also explains bias–variance trade-offs, cross-validation, class imbalance, uncertainty and performance metrics.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Distinguish supervised learning from unsupervised learning and select suitable models for mechatronic problems.
2. Explain and apply linear and nonlinear regression, classification, clustering and dimensionality-reduction methods.
3. Analyse model capacity using the bias–variance framework.
4. Design a leakage-resistant train–validation–test workflow using cross-validation.
5. Select performance metrics appropriate to regression, classification and imbalanced fault-detection tasks.
6. Interpret uncertainty, limitations and deployment risks in engineering applications.

1. Learning Tasks and Data Representation

A dataset consists of observations and variables. Let X ∈ ℝ^n×p denote a feature matrix with n observations and p measured features. The target may be a continuous quantity y ∈ ℝⁿ or a class label y ∈ {1,…,K}ⁿ. Features can include motor current, vibration amplitude, temperature, position, velocity, pressure and time-derived quantities.

In supervised learning, each training observation contains an input–target pair (x_i,y_i). The model learns a function f_θ(x) parameterized by θ. In unsupervised learning, target labels are absent; the objective is to discover structure such as clusters, latent coordinates or anomalies.

A sound workflow begins with the engineering question. “Predict remaining useful life” is a regression problem; “is the gearbox defective?” is a classification problem; “what operating modes exist?” is a clustering problem; and “can the sensor set be represented with fewer variables?” is a dimensionality-reduction problem.

Data preparation

Typical preparation includes time alignment, sensor calibration, missing-value treatment, outlier investigation, feature scaling and feature selection. Scaling is essential for distance-based and gradient-based methods. Standardisation transforms a feature using

where μ is the training-set mean and σ is the training-set standard deviation. The transformation is dimensionless. Parameters must be estimated from the training data only and then applied unchanged to validation and test data. Otherwise, information from the future or test set leaks into training.

For time-dependent systems, random shuffling can also leak information. A model trained on samples surrounding a test interval may appear accurate because adjacent measurements are strongly correlated. Chronological or grouped splitting is usually safer.

2. Regression Models

Regression predicts a continuous response such as temperature, position error, energy consumption or tool wear.

2.1 Linear regression

The multiple linear regression model is

where β₀ is the intercept, β_j are coefficients, x_ij are features and ε_i is an error term. In matrix form, the ordinary least-squares estimate minimizes the residual sum of squares:

Under the assumptions of a correctly specified linear mean relationship, independent errors, constant error variance and no exact multicollinearity, the least-squares estimator is especially useful and interpretable. These assumptions need not be perfectly true for prediction, but violations affect inference and reliability.

Ridge regression adds an L2 penalty:

where λ ≥ 0 controls regularisation. Ridge reduces coefficient variance and is useful when features are correlated. Lasso replaces the squared penalty with an L1 penalty, λ∑|β_j|, and can set some coefficients exactly to zero, thereby performing a form of feature selection. Regularisation parameters must be selected using validation data or cross-validation.

2.2 Nonlinear regression

Polynomial regression, splines, support-vector regression, decision trees and neural networks can represent nonlinear behaviour. A decision tree recursively partitions feature space and predicts a local value in each region. Trees are interpretable but can overfit if grown too deeply. Support-vector regression uses a tolerance tube around predictions and kernels to represent nonlinear relationships; it is effective for moderate-sized datasets but can be computationally demanding.

For physical systems, nonlinear ML should not automatically replace a mechanistic model. Hybrid modelling can combine a known dynamic equation with a learned residual:

Here x_t is the state, u_t the input and y_t+1 the next output. This formulation is appropriate only when the residual is learnable and the physical model is correctly defined. It can improve generalisation while preserving physical structure.

3. Classification Models

Classification assigns observations to categories such as normal, misalignment, overload or bearing damage.

3.1 Logistic regression

For binary classification, logistic regression models the probability of class 1 as

The sigmoid function produces a value between 0 and 1. A threshold, commonly 0.5 but not necessarily optimal, converts probability into a class decision. The coefficients describe changes in log-odds, not direct changes in probability. Multiclass logistic regression can use a softmax function.

3.2 Decision trees and nearest neighbours

A decision tree uses feature thresholds to create regions with relatively homogeneous classes. Its advantages include nonlinear decision boundaries, limited preprocessing and interpretability. Its disadvantages include instability and overfitting. Maximum depth, minimum leaf size and pruning control complexity.

The k-nearest-neighbour method assigns a query to the majority class among its k closest training observations. It is simple but sensitive to scaling, irrelevant features, the choice of k and computational cost during prediction. It is most appropriate when local similarity has engineering meaning.

3.3 Support-vector machines and neural networks

A support-vector classifier seeks a separating boundary with a large margin. The kernel version can create nonlinear boundaries. It is often effective in high-dimensional sensor applications with modest sample sizes, but probability calibration and scaling require attention.

Neural networks learn layered nonlinear transformations. They are powerful for images, acoustic signals and complex multivariate time series, but require sufficient data, careful regularisation and stronger validation. A high-capacity network is not automatically superior to a well-designed tree ensemble or a physically informed feature model.

4. Clustering and Dimensionality Reduction

4.1 Clustering

Clustering groups observations without labelled targets. In k-means, observations are assigned to one of K clusters by minimising within-cluster squared Euclidean distance:

where C_k is cluster k and μ_k its centroid. The algorithm alternates between assigning points to their nearest centroid and recomputing centroids. It assumes that compact, approximately spherical clusters are meaningful under the selected scaling and distance metric. It is sensitive to K, initialisation and outliers.

Hierarchical clustering builds a tree of nested groups and can be visualised with a dendrogram. Density-based methods such as DBSCAN identify dense regions and label isolated points as noise. DBSCAN is useful when clusters have irregular shapes, but its distance and density parameters can be difficult to select in varying-density data.

Clusters are not automatically real physical states. They must be interpreted using operating conditions, domain expertise and, where possible, independent measurements.

4.2 Principal component analysis

Principal component analysis (PCA) represents correlated features using orthogonal directions of maximum variance. After centring the data matrix, the covariance matrix is

where X is the centred feature matrix. PCA obtains eigenvectors of S. The first principal component is the direction of greatest variance; subsequent components are orthogonal and explain successively less variance. If eigenvalues are λ₁,…,λ_p, the proportion explained by the first r components is

PCA is useful for visualisation, compression, denoising and reducing collinearity. However, maximum variance is not equivalent to maximum fault-detection information. A low-variance feature may be diagnostically critical. PCA components are also linear combinations and may be difficult to interpret physically.

5. Bias, Variance and Generalisation

Training error alone is not a reliable measure of engineering performance. A flexible model can memorise noise and perform poorly on unseen machines or operating conditions.

For squared-error prediction, expected test error can be conceptually decomposed into

where bias is systematic error from restrictive assumptions, variance is sensitivity to the training sample, and σ² represents irreducible noise under the assumed data-generating process. The decomposition is an idealised expectation, not a direct measurement from one dataset.

Underfitting corresponds to high bias and low variance; overfitting corresponds to low training bias but high variance. Increasing model complexity often reduces bias and increases variance. Regularisation, more representative data, early stopping, pruning, bagging and feature engineering can improve the balance.

6. Validation and Model Selection

A typical split contains training, validation and test sets. The training set estimates model parameters, the validation set supports model and hyperparameter selection, and the untouched test set provides a final estimate of generalisation. The test set must not be repeatedly consulted during development.

k-fold cross-validation divides training data into k folds. Each fold is used once for validation while the remaining folds are used for fitting. The mean validation score estimates expected performance, while variation across folds indicates sensitivity to the sample. Stratified folds preserve class proportions for classification. Grouped folds keep observations from the same machine, production batch or experiment in one fold. Time-series validation trains on earlier data and validates on later data.

A complete pipeline must fit imputation, scaling, feature selection and resampling within each training fold. Performing these operations before cross-validation causes optimistic estimates. Nested cross-validation is appropriate when an unbiased performance estimate is needed while hyperparameters are extensively tuned.

7. Performance Metrics

Regression metrics

Mean absolute error is

It has the same units as the target and is relatively interpretable. Mean squared error is

and root mean squared error is RMSE = √MSE. MSE and RMSE penalise large errors more strongly than MAE. Mean absolute percentage error can be misleading when actual values are zero or close to zero. Metrics should be related to engineering consequences: a 2 °C error may be acceptable for monitoring but unacceptable for thermal protection.

Classification metrics

A confusion matrix contains true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN). Important metrics are

Precision measures the proportion of alarms that are correct; recall measures the proportion of actual positive cases detected. The F1 score is the harmonic mean:

Specificity is TN/(TN+FP). Receiver operating characteristic (ROC) curves show true-positive rate against false-positive rate across thresholds. Precision–recall curves are often more informative when positive faults are rare. A threshold should be selected using the cost of missed faults, unnecessary maintenance and unsafe operation—not by convention alone.

Unsupervised evaluation

Silhouette analysis compares within-cluster cohesion with separation from other clusters, but no unsupervised metric proves that clusters correspond to physical operating modes. Stability under resampling, engineering interpretability and external validation are also important.

8. Class Imbalance and Uncertainty

Fault data are commonly imbalanced: healthy examples may vastly outnumber failures. Accuracy can then be high even when the model detects no faults. Remedies include class-weighted losses, carefully designed sampling, anomaly detection, threshold adjustment and collection of more representative fault data. Synthetic oversampling must be applied inside training folds; otherwise synthetic information can leak into validation data.

A model’s predicted probability is not automatically a reliable probability. Calibration compares predicted probabilities with observed frequencies. Reliability diagrams, calibration curves and proper scoring rules can assess this property. For safety-related decisions, uncertainty should be communicated rather than hidden behind a single label.

Aleatoric uncertainty arises from measurement noise or inherent variability; epistemic uncertainty arises from limited knowledge or data. Ensembles, bootstrap methods and Bayesian approaches can estimate aspects of uncertainty, but no method eliminates distribution shift. A deployment monitor should detect unfamiliar sensor ranges, missing signals and changing process conditions.

9. Ensemble Learning

Bagging trains models on resampled datasets and averages or votes across them, reducing variance. Random forests combine bagging with random feature selection and are strong general-purpose models for tabular sensor data. Boosting builds models sequentially, with later models emphasising previous errors. Gradient boosting can be highly accurate but is sensitive to hyperparameters and noisy labels. Stacking combines predictions from different model families using a meta-model; the meta-model must be trained without leakage, typically using out-of-fold predictions.

Ensembles improve predictive performance but can reduce interpretability and increase computational and maintenance requirements. Feature importance is not necessarily causal evidence. Permutation importance and partial-dependence analyses can assist interpretation, but correlated features make interpretation difficult.

10. Illustrative Case: Predictive Maintenance of an Industrial Drive

Background

Consider an illustrative three-phase motor driving a conveyor. The controller records vibration, motor current, speed, winding temperature and load every 100 ms. Maintenance engineers want to detect incipient bearing faults early enough to schedule a shutdown.

Problem and analysis

A random sample split would place adjacent time windows from the same motor in both training and test sets, producing overly optimistic results. The plant also has many more healthy windows than fault windows. The engineering objective prioritises high recall, but excessive false alarms impose maintenance costs.

Approach

Signals are synchronised and converted into window-level features: vibration RMS, selected spectral-band energy, temperature slope, current imbalance and speed statistics. Windows from a particular motor and operating campaign are kept together. A chronological training–validation–test split is used. Scaling and feature selection are fitted only on training data.

A regularised logistic regression provides an interpretable baseline. A random forest and gradient-boosted classifier are then compared. Class weights are used, and the decision threshold is selected on validation data using a cost matrix. Performance is reported with recall, precision, PR-AUC, confusion matrices and false alarms per operating hour. Calibration is checked before probabilities are supplied to maintenance software.

Outcome

Suppose the selected model achieves high recall on the held-out campaign but lower performance on a later campaign with a different load distribution. This does not necessarily mean the algorithm is defective; it indicates distribution shift. The deployment design therefore includes operating-range monitoring, periodic review, a human confirmation step and retraining only after new labels have been verified.

Learning points

The case demonstrates that a model is part of a measurement and decision system. Correct splitting, cost-sensitive threshold selection, temporal validation, uncertainty awareness and operational monitoring are as important as the classifier itself.

11. Professional, Ethical and Governance Considerations

Engineers should document data provenance, sensor calibration, feature definitions, model versions, training ranges and known failure modes. Safety-critical control should not depend on an unvalidated statistical model without an independent protection layer. A model that silently fails outside its training distribution can create greater risk than a conservative rule-based system.

Privacy may arise when equipment data are linked to operator identity or production performance. Cybersecurity is also relevant: adversarial or corrupted sensor data can manipulate predictions. Governance should define who approves deployment, who responds to alarms, how changes are validated and when a model is withdrawn. Explainability should be matched to the decision: maintenance personnel need actionable evidence, while a regulator may require traceability and validation records.

12. Best-Practice Workflow

1. Define the engineering decision, costs and acceptable risk.
2. Inspect measurement quality, timing, labels and operating coverage.
3. Split data according to time, machine or experiment to prevent leakage.
4. Establish a simple baseline before using complex models.
5. Build preprocessing and modelling as one reproducible pipeline.
6. Select metrics that reflect engineering consequences and class prevalence.
7. Use cross-validation appropriate to the data-generating process.
8. Tune thresholds and assess calibration, uncertainty and robustness.
9. Test on an untouched, representative dataset.
10. Monitor performance, drift, missing data and human response after deployment.

Chapter Summary

Machine learning in mechatronics involves selecting a model and constructing a trustworthy measurement-to-decision process. Regression predicts continuous quantities, classification assigns operating or fault states, clustering discovers unlabeled structure, and PCA provides a lower-dimensional representation. Linear models offer transparency; trees, kernels, neural networks and ensembles capture increasingly complex relationships.

Generalisation depends on controlling the bias–variance trade-off and preventing leakage. Cross-validation must respect time, machine and group structure. Accuracy alone is inadequate for rare-fault detection; precision, recall, F1, PR curves, calibration and engineering costs are often more useful. Uncertainty, distribution shift, cybersecurity, safety and governance must be considered before deployment. The best model is therefore not simply the one with the highest test score, but the one that performs reliably, transparently and safely in its intended operating environment.

Key Terms

Supervised learning: Learning from input–target pairs.

Unsupervised learning: Learning structure without target labels.

Regression: Prediction of a continuous response.

Classification: Assignment of observations to discrete classes.

Feature: An input variable used by a model.

Regularisation: A constraint or penalty used to control model complexity.

Bias: Systematic prediction error from restrictive assumptions.

Variance: Sensitivity of a fitted model to its training sample.

Cross-validation: Repeated train–validation splitting for model assessment or selection.

Data leakage: Unintended use of information unavailable at prediction time.

Clustering: Grouping observations according to similarity.

k-means: Centroid-based clustering method.

Principal component analysis: Orthogonal linear dimensionality reduction based on variance.

Confusion matrix: Table of correct and incorrect classification outcomes.

Precision: Fraction of predicted positives that are true positives.

Recall: Fraction of actual positives that are detected.

Calibration: Agreement between predicted probabilities and observed frequencies.

Class imbalance: Unequal prevalence of classes.

Ensemble: Combination of multiple predictive models.

Distribution shift: Change between training and deployment data distributions.

Review Questions

Short Answer Questions

1. Distinguish supervised learning from unsupervised learning and give one mechatronic example of each.
2. Why must scaling parameters be estimated from training data only?
3. Define bias and variance in the context of prediction.
4. Why can accuracy be misleading for rare-fault detection?
5. State two reasons for using grouped or chronological cross-validation.

Long Answer Questions

1. Compare linear regression, decision trees and neural networks for predicting motor temperature. Discuss interpretability, data requirements and overfitting.
2. Explain the k-means objective, its assumptions and its limitations for identifying machine operating modes.
3. Describe a leakage-resistant model-development workflow for a predictive-maintenance classifier.
4. Explain how regularisation changes the bias–variance trade-off, comparing ridge and lasso regression.
5. Discuss why calibration and uncertainty matter when a classifier’s output is used to schedule maintenance.

Analytical / Application Questions

1. A fault classifier produces many false alarms but misses few genuine faults. Which metrics and design changes would you investigate before changing the model?
2. A PCA model explains 95% of sensor variance, yet fault detection becomes worse after dimensionality reduction. Give technically plausible explanations and propose tests.
3. A model performs well on randomly split data but poorly on a new production line. Analyse possible causes and design a validation and monitoring strategy.

Multiple Choice Questions with Answers

Question 1

Which task is a supervised regression problem?

A. Grouping motors by operating mode without labels

B. Estimating motor temperature from current and load

C. Compressing sensor data with PCA

D. Detecting dense regions with DBSCAN

Correct Answer: B. Estimating motor temperature from current and load

Question 2

In standardisation, what does σ represent in z=(x−μ)/σ?

A. The target value

B. The model intercept

C. The training-set standard deviation

D. The number of classes

Correct Answer: C. The training-set standard deviation

Question 3

Which metric is especially useful when missing a rare fault is costly?

A. Recall

B. Accuracy only

C. Mean squared error

D. R² only

Correct Answer: A. Recall

Question 4

What is a principal purpose of ridge regression?

A. To remove all observations with outliers

B. To reduce coefficient variance when features are correlated

C. To guarantee zero training error

D. To create class labels without targets

Correct Answer: B. To reduce coefficient variance when features are correlated

Question 5

Which method is most directly based on minimising within-cluster squared distance to centroids?

A. Logistic regression

B. PCA

C. k-means

D. Random forest

Correct Answer: C. k-means

Question 6

Why is a random train–test split risky for strongly time-correlated sensor data?

A. It always increases bias

B. It may place near-duplicate future information in training and test sets

C. It prevents feature scaling

D. It makes regression impossible

Correct Answer: B. It may place near-duplicate future information in training and test sets

Question 7

If TP=18 and FN=2, what is recall?

A. 10%

B. 20%

C. 80%

D. 90%

Correct Answer: D. 90%

Question 8

Which statement about PCA is correct?

A. It guarantees maximum fault-detection accuracy

B. It creates orthogonal directions ordered by explained variance

C. It requires labelled classes

D. It is unaffected by feature scaling

Correct Answer: B. It creates orthogonal directions ordered by explained variance

Question 9

What is a principal advantage of bagging?

A. It generally reduces variance by averaging models

B. It guarantees causal explanations

C. It eliminates the need for validation

D. It always improves recall to 100%

Correct Answer: A. It generally reduces variance by averaging models

Question 10

Which practice best prevents preprocessing leakage during cross-validation?

A. Fit scaling on the complete dataset before splitting

B. Fit preprocessing separately within each training fold

C. Use the test set to select hyperparameters

D. Delete the validation folds

Correct Answer: B. Fit preprocessing separately within each training fold