Supervised Machine Learning

Introduction and Context

Supervised machine learning (ML) learns a mapping from measured inputs to known outputs using labelled examples. In electrical engineering, the inputs may be voltage, current, temperature, vibration, frequency, irradiance, load, or phasor measurements; outputs may be power, energy demand, equipment condition, fault class, remaining useful life, or control quality. The central engineering purpose is not merely to fit historical data, but to produce predictions that remain accurate, physically plausible, safe, and useful under future operating conditions.

A supervised learning dataset is written as

where xᵢ ∈ ℝᵖ is a vector of p measured features, yᵢ is the target, and N is the number of labelled observations. Regression uses a numerical target; classification uses a categorical target. A model f̂ is estimated from training data and then evaluated on data not used for fitting.

Electrical engineering applications are especially demanding because measurements can be noisy, correlated, nonstationary, imbalanced, and constrained by physical laws. A model that achieves a good average score can still be unacceptable if it misses rare faults, extrapolates dangerously, or cannot provide a trustworthy uncertainty estimate.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Distinguish regression from classification and formulate an engineering prediction problem.
2. Develop and interpret linear models, decision trees, ensembles, support vector machines, and nearest-neighbor models.
3. Select suitable loss functions and performance metrics.
4. Apply train–validation–test splitting, cross-validation, and hyperparameter tuning without data leakage.
5. Assess uncertainty, diagnose errors, and evaluate models under realistic electrical operating conditions.
6. Compare models using technical, safety, computational, and maintainability criteria.

1. Formulating an Engineering Prediction Problem

A reliable workflow begins with the target definition. For example, “fault prediction” may mean identifying a fault type, estimating the probability of a fault within 24 hours, or estimating time to failure. These are different supervised tasks and require different labels, metrics, and decision thresholds.

Regression and classification

In regression, y ∈ ℝ. Examples include predicting transformer top-oil temperature, photovoltaic power, feeder voltage, or hourly demand. A common error measure is mean squared error (MSE):

Here yᵢ is the observed value, ŷᵢ is the prediction, and n is the number of test observations. If y is measured in kW, MSE has units kW². Root mean squared error is

and has the same units as the target. Mean absolute error is

MAE is less sensitive to isolated large errors than MSE. These metrics assume that the test observations represent the conditions in which the model will be deployed.

In classification, the target belongs to one of K classes. A binary classifier may predict normal versus faulty operation. Its confusion matrix contains true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Important metrics are

Recall is also called sensitivity or true-positive rate. Precision answers: among predicted faults, how many were actual faults? Recall answers: among actual faults, how many were detected? Accuracy may be misleading when faults are rare. In protection and condition monitoring, a false negative can be much more costly than a false positive, so the metric and decision threshold must reflect engineering consequences.

Feature engineering and data quality

Features should be available at the time a prediction is made. Leakage occurs when information from the future, from the test set, or from the target itself enters training. Examples include using a post-fault alarm to predict the fault that caused it, scaling all data before splitting, or randomly splitting consecutive samples from the same event.

Typical preparation includes timestamp alignment, unit checking, missing-value treatment, outlier investigation, categorical encoding, and scaling. Standardization uses

where μⱼ and σⱼ are calculated from the training set only. The resulting feature is dimensionless. Scaling is important for distance-based and margin-based methods, but generally unnecessary for ordinary decision trees.

Electrical features should also respect domain knowledge. Three-phase measurements may be converted into symmetrical components; periodic signals may be represented using amplitude and phase; and lagged load values may capture temporal dependence. Feature transformations should be documented because they affect interpretability and reproducibility.

2. Linear Models

Linear regression

The multiple linear regression model is

where β₀ is the intercept and βⱼ is the change in predicted target per unit change in xⱼ while other features remain fixed. In matrix form, ŷ = Xβ. Ordinary least squares estimates β by minimizing the residual sum of squares:

When XᵀX is invertible, the least-squares solution is β̂ = (XᵀX)⁻¹Xᵀy. The formula is not used blindly: multicollinearity can make coefficients unstable, and the linear relationship may be inadequate. The model also assumes independent errors for standard inferential interpretations; prediction can still be useful when this assumption is imperfect.

Regularization controls coefficient magnitude. Ridge regression minimizes

and lasso regression minimizes

where λ ≥ 0 is selected using validation. Ridge usually shrinks correlated coefficients together; lasso can set some coefficients exactly to zero. The intercept is commonly excluded from the penalty. Regularization reduces variance but may introduce bias.

Logistic regression

For binary classification, logistic regression models the probability of class 1 as

The log-odds are linear in the features:

A threshold, often 0.5 but not necessarily, converts probability to a class decision. Logistic regression is interpretable and provides a useful baseline, but its decision boundary is linear unless features are transformed. Its probabilities may require calibration, especially under class imbalance or regularization.

3. Decision Trees and Ensembles

A decision tree recursively partitions feature space using rules such as “temperature > 85 °C.” A leaf produces a constant prediction: commonly the mean target for regression or the majority class/proportion for classification.

For classification, a node impurity measure is the Gini impurity:

where pₖ is the proportion of class k in the node. Entropy is

A split is selected to reduce weighted impurity. Deep trees can memorize noise; controlling maximum depth, minimum leaf size, or pruning improves generalization. Trees handle nonlinear interactions and mixed feature scales, but individual trees are unstable: small data changes may produce very different structures.

Random forests reduce variance by fitting many trees to bootstrap samples and random feature subsets. For regression, predictions are averaged; for classification, class votes or averaged class probabilities are used. Extra-trees methods introduce additional randomization in split selection. These models are strong general-purpose baselines, but they may be large, less transparent than a single tree, and poor at extrapolating beyond the training target range.

Gradient boosting builds trees sequentially, with each new tree correcting errors made by the current ensemble. A generic additive form is

where hₘ is the new weak learner and η is the learning rate. Small η with more trees often improves generalization, but excessive boosting causes overfitting. Important hyperparameters include tree depth, number of trees, learning rate, subsampling fraction, and minimum leaf size.

4. Support Vector Machines

A support vector machine (SVM) seeks a decision boundary with a large margin. For linearly separable data, the separating hyperplane is wᵀx + b = 0, and the margin width is 2/||w||. The soft-margin optimization problem is

subject to yᵢ(wᵀxᵢ + b) ≥ 1 − ξᵢ and ξᵢ ≥ 0, where yᵢ ∈ {−1,+1}, ξᵢ are slack variables, and C controls the penalty for violations. Large C emphasizes fitting training examples; smaller C allows a wider margin with more violations.

Nonlinear SVMs use a kernel, such as the radial basis function (RBF):

where γ controls the locality of influence. Large γ creates highly local boundaries and can overfit; small γ produces smoother boundaries. SVMs require feature scaling because distances determine the margin and kernel values. They can be effective in moderate-sized, high-dimensional datasets, but training and prediction may become expensive for very large datasets. Probability outputs are not intrinsic to the basic SVM and may require calibration.

5. Nearest-Neighbor Methods

The k-nearest-neighbor (k-NN) method predicts from the k training observations closest to a query point. With Euclidean distance,

Regression commonly averages the neighbors; classification uses majority voting or distance-weighted voting. Small k gives low bias and high variance; large k gives smoother, more biased predictions. k-NN is simple and nonparametric, but it is sensitive to feature scaling, irrelevant features, missing values, and the curse of dimensionality. In high dimensions, distances become less discriminative, making feature selection or dimensionality reduction important.

6. Model Evaluation and Validation

A basic split consists of training, validation, and test sets. The training set fits parameters, the validation set selects models and hyperparameters, and the untouched test set estimates final generalization. If data are limited, k-fold cross-validation divides the training data into k folds, fits on k−1 folds, and evaluates on the remaining fold, repeating until every fold has served as validation. The mean score is

where Sᵣ is the score in fold r. The variation among fold scores indicates sensitivity to the sample.

Random k-fold splitting is inappropriate when observations are time ordered, spatially correlated, or grouped by equipment. Time-series validation trains on earlier observations and validates on later observations. Grouped validation ensures that measurements from the same transformer, motor, or experimental run do not appear in both training and validation. Stratification preserves class proportions when feasible.

Hyperparameter tuning must occur inside the training process. In nested cross-validation, an inner loop selects hyperparameters and an outer loop estimates performance. This reduces optimistic bias caused by repeatedly using the same validation results to make selections. A preprocessing pipeline should fit scaling, imputation, and feature selection separately within each training fold.

Useful hyperparameters include λ for regularized linear models, tree depth and leaf size, forest size and feature sampling, boosting learning rate and depth, SVM C and γ, and k for k-NN. Grid search is systematic but expensive; random search often explores high-dimensional spaces more efficiently. Bayesian optimization can be useful when each model evaluation is costly.

7. Uncertainty and Error Analysis

A point prediction alone is incomplete for engineering decisions. Sources of uncertainty include measurement noise, finite training data, model approximation, distribution shift, and label uncertainty. A prediction interval aims to contain a future numerical outcome with a stated coverage under specified assumptions; it is not the same as a confidence interval for a mean.

Practical approaches include bootstrap refitting, ensembles, quantile regression, and conformal prediction. Bootstrap uncertainty repeatedly resamples training cases, refits the model, and examines the distribution of predictions. It captures sampling variation but may fail under strong temporal dependence. Conformal methods can produce prediction sets or intervals with finite-sample marginal coverage under exchangeability; coverage may deteriorate when the deployment distribution changes.

Classification uncertainty should include probability calibration. A calibrated classifier’s predicted probability of 0.8 should correspond approximately to an 80% event frequency among comparable cases. Reliability diagrams and calibration metrics can reveal overconfident models. The classification threshold should be chosen using costs, safety requirements, and available response capacity rather than automatically using 0.5.

Error analysis should be systematic. Plot residuals eᵢ = yᵢ − ŷᵢ against predicted value, time, temperature, load, and operating regime. Examine errors by equipment, fault type, season, and sensor quality. A model with low average RMSE may fail during peak demand, exactly when accuracy matters most. For classification, inspect the confusion matrix and representative false positives and false negatives. Investigate whether errors arise from poor labels, missing features, sensor drift, rare regimes, or model limitations.

8. Worked Example: Transformer Temperature Regression

Consider an illustrative dataset for estimating transformer top-oil temperature from load current I, ambient temperature Tₐ, and power factor pf. Suppose a fitted linear model is

where T̂ₒ and Tₐ are in °C, I is in A, and pf is dimensionless. For I = 100 A, Tₐ = 30 °C, and pf = 0.90,

If the measured temperature is 45.0 °C, the residual defined as observed minus predicted is 3.1 °C. The model underpredicts this observation. The coefficients are associations within the fitted model, not automatically causal physical constants. The model should be checked for unit consistency, temperature-range validity, residual patterns, and performance on later operating periods.

Suppose five test residual magnitudes are 2, 3, 1, 4, and 0 °C. Then MAE is (2+3+1+4+0)/5 = 2 °C. If the signed residuals are 2, −3, 1, 4, and 0 °C, RMSE is √[(4+9+1+16+0)/5] = √6 ≈ 2.45 °C. Reporting both metrics shows typical error and sensitivity to larger errors.

9. Applied Case: Early Fault Detection in a Distribution Transformer

This illustrative case concerns a utility seeking early warning of incipient transformer faults. Sensors provide oil temperature, load current, dissolved-gas indicators, ambient temperature, vibration, and voltage quality. Historical maintenance records label windows as normal, incipient fault, or confirmed fault.

Problem. Confirmed faults are rare, labels are delayed, and measurements from the same transformer are strongly correlated over time. A random row-wise split would allow nearly identical observations from one event into both training and test data, overstating performance.

Analysis. Engineers first define the prediction horizon: whether the model must identify a fault 24 hours before intervention. They remove features unavailable at prediction time, align sensors, document missingness, and use grouped time-based validation. Class frequencies and the costs of missed faults and unnecessary inspections are quantified.

Approach. A regularized logistic regression provides an interpretable baseline. A random forest captures nonlinear interactions, while an RBF SVM is evaluated after standardization. Class weights or carefully designed sampling address imbalance. Hyperparameters are tuned only within training folds. Performance is reported using recall, precision, PR-AUC, calibration, and false alarms per transformer-month rather than accuracy alone. Thresholds are selected with operations personnel.

Outcome. Suppose the forest has the best cross-validated recall but produces too many alarms, while logistic regression has slightly lower recall and substantially better interpretability and calibration. The utility may deploy logistic regression for automatic advisory alarms and reserve the forest for analyst support, subject to a pilot and independent temporal test. The result is a deployment decision, not simply a contest for the highest score.

Learning points. Label definition, temporal validation, cost-sensitive evaluation, calibration, and human response procedures are as important as algorithm selection. A model must be monitored after deployment for sensor drift, changes in maintenance policy, and new operating regimes.

10. Limitations, Governance, and Professional Practice

Supervised models learn historical relationships and can fail under distribution shift caused by new equipment, renewable penetration, climate conditions, topology changes, or altered protection settings. They do not replace circuit laws, protection coordination, engineering judgment, or mandatory safety procedures. Physics-informed features, hard operational limits, and model-based residual checks can reduce implausible predictions, but they must be validated rather than assumed effective.

Data governance includes access control, provenance, versioning, label auditability, and retention. Safety-critical applications require traceable testing, change control, cybersecurity protection, fail-safe behavior, and clear responsibility for decisions. Operators should know whether a prediction is advisory or automatically actuating equipment. Personal data may arise in building-load or workforce datasets and must be handled under applicable privacy requirements.

Best practice is to maintain a reproducible pipeline: freeze data definitions, split before preprocessing, record random seeds where relevant, preserve model and feature versions, report uncertainty and subgroup performance, and retain an untouched final test set. Monitor both predictive performance and operational outcomes after deployment. Retraining should be triggered by evidence of drift, not by an arbitrary schedule alone.

Contemporary developments include gradient-boosted decision trees, automated hyperparameter optimization, explainability methods, conformal prediction, and hybrid physics–ML models. Explainability tools can identify influential features, but association is not causation and explanations can be unstable. Their results should support engineering investigation, not substitute for validation.

Chapter Summary

Supervised machine learning estimates relationships between labelled electrical-engineering measurements and numerical or categorical targets. Linear and logistic models provide interpretable baselines; trees capture nonlinear interactions; ensembles improve predictive strength; SVMs provide margin-based classification; and nearest-neighbor methods offer simple local predictions. Each method has assumptions, computational costs, and failure modes.

Sound evaluation requires leakage-free preprocessing, appropriate temporal or grouped validation, cross-validation, hyperparameter tuning, and an untouched test set. Metrics must reflect engineering consequences, especially class imbalance and the relative costs of false alarms and missed faults. Uncertainty, calibration, residual inspection, subgroup analysis, and post-deployment monitoring are essential for trustworthy use. In electrical engineering, model performance must be combined with physical plausibility, safety, governance, cybersecurity, and human responsibility.

Key Terms

Supervised learning: Learning a predictive mapping from labelled examples.

Regression: Prediction of a numerical target.

Classification: Prediction of a categorical target.

Feature: An input variable used by a model.

Target: The output variable to be predicted.

Data leakage: Use of information unavailable at deployment or improperly shared across splits.

Regularization: A penalty or constraint that controls model complexity.

Decision tree: A model that predicts through recursive feature-based splits.

Random forest: An ensemble of randomized decision trees.

Gradient boosting: Sequential construction of learners that reduce previous errors.

Support vector machine: A margin-based model for classification or regression.

Kernel: A function enabling nonlinear similarity calculations.

Nearest neighbor: A method predicting from nearby training observations.

Cross-validation: Repeated training and validation over data folds.

Hyperparameter: A model-setting selected outside parameter fitting.

Calibration: Agreement between predicted probabilities and observed frequencies.

Data drift: Change in the distribution of inputs or relationships over time.

Residual: The observed target minus the model prediction.

Precision: The fraction of positive predictions that are true positives.

Recall: The fraction of actual positives detected by the model.

Review Questions

Short Answer Questions

1. Distinguish regression from classification and give one electrical-engineering example of each.
2. Why must feature scaling be fitted using training data only?
3. Explain the difference between precision and recall.
4. What is the purpose of regularization in a linear model?
5. Why can random row-wise splitting be invalid for time-series sensor data?

Long Answer Questions

1. Compare linear models, decision trees, ensembles, SVMs, and k-NN in terms of assumptions, scaling, interpretability, and computational requirements.
2. Explain how to design a leakage-free cross-validation and hyperparameter-tuning workflow for transformer condition monitoring.
3. Discuss why accuracy is insufficient for rare-fault classification and propose a more appropriate evaluation framework.
4. Explain sources of prediction uncertainty and compare bootstrap, ensemble, and conformal approaches.
5. Describe a complete supervised-learning lifecycle from problem formulation through post-deployment monitoring.

Analytical / Application Questions

1. A fault detector has TP = 36, TN = 900, FP = 24, and FN = 40. Calculate accuracy, precision, recall, and F₁, and explain which metric may be most important if missed faults are costly.
2. A regression model produces residuals of 1, −2, 3, and −4 kW on four test cases. Calculate MAE and RMSE, and interpret the difference between them.
3. Design a validation strategy for data from 20 transformers measured hourly for two years. Explain how you would prevent equipment-specific and temporal leakage, select metrics, and assess uncertainty.

Multiple Choice Questions with Answers

Question 1

Which task is a regression problem?

A. Classifying a relay event as normal or faulty

B. Predicting transformer top-oil temperature in °C

C. Identifying the type of cable fault

D. Deciding whether an alarm should be issued

Correct Answer: B. Predicting transformer top-oil temperature in °C

Question 2

What does recall measure in binary classification?

A. The fraction of predicted positives that are correct

B. The fraction of all predictions that are correct

C. The fraction of actual positives detected

D. The fraction of actual negatives rejected

Correct Answer: C. The fraction of actual positives detected

Question 3

Which method is most directly affected by feature scale because it uses distances?

A. Ordinary decision tree

B. k-nearest neighbors

C. Unpruned random forest

D. Bagged tree classifier

Correct Answer: B. k-nearest neighbors

Question 4

In ridge regression, the regularization penalty is based on:

A. The sum of absolute coefficient values

B. The sum of squared coefficient values

C. The number of decision-tree leaves

D. The number of nearest neighbors

Correct Answer: B. The sum of squared coefficient values

Question 5

Why is an untouched test set used?

A. To fit preprocessing parameters

B. To select every hyperparameter repeatedly

C. To estimate final performance after model choices are complete

D. To increase the number of training labels

Correct Answer: C. To estimate final performance after model choices are complete

Question 6

What is a major danger of random splitting consecutive sensor observations?

A. It always removes all missing values

B. It can place highly related observations from the same event in training and test sets

C. It prevents the use of linear models

D. It guarantees underfitting

Correct Answer: B. It can place highly related observations from the same event in training and test sets

Question 7

In an RBF SVM, increasing γ generally makes the kernel influence:

A. More local

B. Independent of distance

C. More linear

D. Dependent only on the target mean

Correct Answer: A. More local

Question 8

For residuals 2, −2, and 4 kW, what is the MAE?

A. 2 kW

B. 2.67 kW

C. 3.33 kW

D. 4 kW

Correct Answer: B. 2.67 kW

Question 9

What is the principal purpose of probability calibration?

A. To guarantee zero classification error

B. To make predicted probabilities correspond to observed event frequencies

C. To remove the need for a test set

D. To convert regression into clustering

Correct Answer: B. To make predicted probabilities correspond to observed event frequencies

Question 10

Which approach is appropriate when observations belong to multiple individual transformers?

A. Ensure measurements from each transformer can appear in every fold

B. Use grouped validation so a transformer is kept together within a fold

C. Shuffle labels before cross-validation

D. Fit scaling using the complete dataset before splitting

Correct Answer: B. Use grouped validation so a transformer is kept together within a fold