Neural Networks and Deep Learning

Introduction

Neural networks are computational models that learn relationships between measured inputs and desired outputs. In mechatronics, these relationships may connect motor currents to torque, vibration signals to bearing condition, camera images to object location, or machine-state histories to future faults. Unlike manually programmed rules, a neural network can infer complex nonlinear mappings from examples.

Deep learning refers to neural-network models with multiple trainable layers. These layers learn progressively more abstract representations: a vision model may learn edges, shapes, and objects, while a vibration model may learn local frequency patterns and their temporal evolution. The engineering value of deep learning lies not merely in prediction accuracy, but in reliable deployment under noise, changing operating conditions, limited data, computational constraints, and safety requirements.

This chapter develops multilayer perceptrons, convolutional neural networks, recurrent networks, and attention-based models. It also explains regularization, transfer learning, hyperparameter optimization, interpretability, and responsible deployment.

Learning Objectives

After studying this chapter, a student should be able to:

1. Explain the structure and learning mechanism of feedforward neural networks.
2. Design suitable multilayer, convolutional, recurrent, and attention-based architectures for mechatronic data.
3. Formulate loss functions and use backpropagation and gradient-based optimization.
4. Diagnose overfitting, underfitting, data leakage, and distribution shift.
5. Apply regularization, transfer learning, and hyperparameter optimization.
6. Evaluate models using technically appropriate metrics and validation procedures.
7. Interpret predictions and assess safety, reliability, and deployment constraints.

1. Neural-Network Foundations

1.1 The artificial neuron

A neuron receives an input vector and computes a weighted sum followed by an activation function. For input vector x = [x₁, x₂, …, x_d]^T, weights w, bias b, and activation function φ, its output is

The weights and bias are learned parameters. The bias permits the activation threshold to shift. Inputs should generally be scaled, especially when physical variables have different units and magnitudes. Standardization commonly uses

where μ and σ are calculated from the training set only. This transformation is dimensionless, although the original variable may have units such as volts, amperes, metres per second, or degrees Celsius.

Common activation functions include the sigmoid, hyperbolic tangent, and rectified linear unit (ReLU):

Sigmoid outputs lie between 0 and 1 and are useful for binary probabilities, but it can saturate and produce small gradients. Tanh is zero-centred but also saturates. ReLU is computationally simple and often effective in hidden layers, although units can become permanently inactive when their inputs remain negative. Variants such as leaky ReLU reduce this risk.

1.2 Multilayer perceptrons

A multilayer perceptron (MLP) connects layers of neurons. For layer l,

where W^(l) is a weight matrix, b^(l) is a bias vector, h^(l−1) is the previous layer output, and h^(l) is the current output. The input is h^(0) = x. The final layer is selected according to the task: one linear output for regression, sigmoid output for binary classification, and softmax outputs for mutually exclusive multiclass classification.

For logits s₁,…,s_K, softmax gives

The probabilities satisfy 0 ≤ p_k ≤ 1 and ∑_k p_k = 1. Softmax is inappropriate when classes are not mutually exclusive; independent sigmoid outputs are then more suitable.

1.3 Universal approximation and its limits

A sufficiently wide feedforward network with a nonconstant nonlinear activation can approximate broad classes of continuous functions on compact domains. This result does not imply that a network will learn the function efficiently, generalize outside the training distribution, or satisfy physical constraints. Approximation capacity must therefore be distinguished from data efficiency, optimization quality, robustness, and interpretability.

2. Training and Backpropagation

Training minimizes a loss over examples. For supervised data {(x_i,y_i)}, the empirical objective is

where N is the number of training examples, f_θ is the network with parameters θ, and L is a loss function.

For scalar regression, mean squared error is

It has squared output units and penalizes large errors strongly. Mean absolute error,

has the same units as the target and is less sensitive to isolated large errors. For binary classification with target y ∈ {0,1} and predicted probability p, binary cross-entropy is

For multiclass classification, categorical cross-entropy is L = −∑_k y_k ln(p_k), where y_k is one for the correct class and zero otherwise.

Backpropagation applies the chain rule to calculate derivatives of the loss with respect to every parameter. A basic gradient-descent update is

where η is the learning rate. In practice, mini-batches estimate the gradient and optimizers such as momentum, RMSProp, or Adam adapt the update. The learning rate is often more influential than fine adjustments to network width.

A training epoch is one complete pass through the training data. The validation set is used for model selection and tuning; the test set is used once, after decisions are complete, for an unbiased final estimate. Randomly splitting strongly correlated time-series samples can cause leakage. For machines, splitting by time period, production batch, machine, or operating condition is often more defensible.

3. Architectures for Mechatronic Data

3.1 Choosing an architecture

The representation of the input should reflect the physical structure of the data.

3.2 Convolutional neural networks

A convolutional layer applies a small kernel across local positions. In one dimension, for input sequence x and kernel w of length K, a valid cross-correlation operation is

Deep-learning libraries commonly call this operation convolution even though the kernel is not reversed as in the strict mathematical definition. With stride greater than one, outputs are spaced farther apart; padding adds boundary values. For input length L, kernel size K, padding P on each side, and stride S, the output length is

The formula assumes a one-dimensional discrete layer and integer-compatible dimensions. Pooling or strided convolution reduces resolution and increases the receptive field. CNNs share kernel weights across positions, giving parameter efficiency and useful translation-related inductive bias. In mechatronics, one-dimensional CNNs can detect local vibration or current signatures, while two-dimensional CNNs process images, thermal maps, or spectrograms.

3.3 Recurrent neural networks

An RNN maintains a hidden state that summarizes prior inputs:

Here x_t is the input at time t, h_t is the hidden state, and y_t is the output. Training through time can produce vanishing or exploding gradients because derivatives are repeatedly multiplied across time steps. Gradient clipping limits an update when its norm exceeds a selected threshold, but does not solve all long-term dependency problems.

Long short-term memory (LSTM) networks use gates to regulate information. A representative LSTM is

The symbols i_t, f_t, and o_t are input, forget, and output gates; c_t is the cell state; and ⊙ denotes elementwise multiplication. Gated recurrent units (GRUs) provide a simpler alternative with fewer gates.

3.4 Attention and Transformers

Attention allows a representation at one position to use information from other positions. Given queries Q, keys K, and values V, scaled dot-product attention is

where d_k is the key dimension. The scaling reduces excessively large dot products. Self-attention uses the same sequence to produce queries, keys, and values. Multi-head attention computes several attention operations with different learned projections, allowing different relationships to be represented.

Transformers generally combine attention, position information, feedforward layers, residual connections, and normalization. They can model long-range dependencies and parallelize training better than sequential RNNs. Their disadvantages include high memory use for long sequences, substantial data requirements, and attention weights that should not automatically be interpreted as causal explanations.

For irregular sensor streams, timestamps, masking, resampling, or continuous-time representations may be necessary. A model should not infer temporal order from arbitrary row order.

Figure: A mechatronic neural-network pipeline from sensing to decision-making.

4. Generalization and Regularization

Overfitting occurs when a model fits training-specific noise and performs poorly on new data. Underfitting occurs when the model cannot represent the relevant relationship. A useful diagnostic compares training and validation curves, but the diagnosis must consider data leakage and changing operating conditions.

Common regularization methods include:

• Weight decay: add λ||W||₂² to the objective. The coefficient λ ≥ 0 controls the penalty; the factor convention may differ between software implementations.
• Dropout: randomly set hidden activations to zero during training, then use the corresponding expectation-preserving scaling during inference.
• Early stopping: stop when validation performance ceases to improve, using a patience interval.
• Data augmentation: create physically plausible variations, such as modest sensor noise, shifts, or amplitude changes. Augmentation must not alter the class meaning.
• Batch normalization or related normalization: stabilize intermediate activations, although its benefit depends on batch size and deployment conditions.
• Simpler architectures: reduce parameters when data are scarce.

Regularization cannot compensate for unrepresentative data. If a defect occurs only at a temperature absent from training, more dropout does not create the missing knowledge.

Transfer learning

Transfer learning begins with a model trained on a source task and adapts it to a target task. For visual inspection, a model pretrained on a large image dataset may provide useful low-level features. The engineer may initially freeze early layers and train a new output head, then gradually fine-tune selected layers at a low learning rate. Transfer is beneficial when source and target structures are related; it can be harmful when the source domain encourages misleading features.

Hyperparameter optimization

Trainable parameters are learned from data; hyperparameters are chosen by the engineer. Examples include learning rate, batch size, number of layers, hidden width, kernel size, dropout rate, and sequence length. Grid search is simple but expensive. Random search often explores high-impact dimensions more efficiently. Bayesian optimization models validation performance and selects promising trials. Successive-halving methods allocate more resources to promising configurations.

A fair optimization procedure uses a validation protocol fixed before experimentation. Repeatedly testing the final test set turns it into a hidden training resource and produces optimistic performance estimates.

5. Evaluation and Interpretability

Accuracy can be misleading for imbalanced fault data. For binary classification, with true positives TP, true negatives TN, false positives FP, and false negatives FN,

Precision measures the fraction of alarms that are correct; recall measures the fraction of actual faults detected. The F1 score is

when both denominators are nonzero. In safety-critical maintenance, a false negative may cost more than a false positive, so threshold selection should reflect consequences, inspection capacity, and downtime costs. Regression evaluation should include engineering units, such as root mean squared error, which is RMSE = √MSE, and maximum or percentile error where limits matter.

Interpretability methods include permutation importance, partial-dependence analysis, local surrogate explanations, gradient-based saliency, integrated gradients, and SHAP-like additive attribution methods. Each has assumptions. Correlated sensors can make feature importance unstable; saliency can highlight irrelevant high-frequency patterns; and an explanation is not proof of causality. Interpretations should be checked against physics, sensor locations, known failure mechanisms, and controlled tests.

Uncertainty can be represented by calibrated probabilities, ensembles, Bayesian approximations, or prediction intervals. A high-confidence wrong prediction is more dangerous than a low-confidence prediction that triggers human review. Calibration should be tested on data representing intended operation.

6. Applied Case: Illustrative Predictive Maintenance System

Background

Consider an illustrative manufacturing cell containing an induction motor driving a conveyor. Three-phase current, accelerometer signals, rotational speed, and temperature are collected. The objective is to detect early bearing degradation while avoiding unnecessary shutdowns.

Problem and analysis

The raw accelerometer operates at a high sampling rate, whereas temperature changes slowly. Fault examples are rare, and consecutive windows from the same run are strongly correlated. A random row-wise split would allow near-duplicate windows from one run into both training and test sets, producing an unrealistic result.

Approach

First, records are grouped by production run and split chronologically. Sensor channels are synchronized and standardized using training-run statistics. Vibration is divided into fixed windows, with overlap selected according to latency and computational limits. A one-dimensional CNN extracts local waveform patterns. Its output is combined with slower temperature and speed features and passed to an MLP classification head. An alternative LSTM is evaluated because fault signatures may evolve over several windows.

The training loss is weighted to reflect the greater cost of missed faults. Augmentation adds small, physically plausible noise and speed variation, but does not synthesize unrealistic fault signatures. Early stopping is based on run-level validation recall at an acceptable false-alarm rate. The threshold is selected on validation data, not the test data.

Outcome

Suppose the final evaluation reports recall, precision, calibration, and detection delay separately for each operating-speed range. A model with slightly lower overall accuracy but better recall at high load may be preferable. Before deployment, explanations are reviewed by maintenance engineers. If the model relies almost entirely on a temperature channel known to lag the fault, engineers investigate whether the apparent performance is caused by leakage or an unintended operating-condition proxy.

The model is deployed as a decision-support system with a confidence threshold and a fallback rule: uncertain cases are sent for inspection rather than directly stopping the machine. Performance is monitored for changes in sensor calibration, product mix, speed distribution, and fault prevalence.

Learning points

The principal engineering challenge is not selecting the deepest network. It is constructing representative labels, leakage-resistant validation, cost-aware decisions, interpretable evidence, and a safe operational response.

7. Deployment, Ethics, and Professional Practice

Neural networks used in mechatronic systems can affect worker safety, product quality, energy consumption, and maintenance decisions. Engineers should document data sources, sensor limitations, preprocessing, model versions, training dates, validation groups, thresholds, and known failure modes. Access controls and cybersecurity are important because manipulated sensor data can induce unsafe decisions.

A model should not directly override a safety-rated control system unless it has been designed, verified, and certified within the applicable safety framework. In many applications, the network should provide diagnosis or advisory output while deterministic interlocks remain responsible for emergency protection. Human operators require understandable alerts, actionable recommendations, and a way to challenge or override noncritical predictions.

Contemporary developments include self-supervised learning for unlabelled sensor data, multimodal models combining images and time series, neural operators for physical fields, physics-informed learning, edge inference, quantization, pruning, and federated learning. These methods offer benefits but introduce new concerns: self-supervised representations may capture nuisance variables, physics-informed losses may be incorrectly specified, compressed models may lose rare-fault sensitivity, and federated systems still require privacy and security analysis.

Best practice is therefore to treat deep learning as part of a complete engineering system: define the decision and its cost, understand the data-generating process, choose an architecture that matches structure, validate under realistic deployment conditions, quantify uncertainty, monitor drift, and retain a safe fallback.

Chapter Summary

Neural networks learn nonlinear mappings through weighted transformations, activation functions, loss minimization, and gradient-based parameter updates. MLPs suit fixed feature vectors; CNNs exploit local spatial or temporal structure; RNNs represent sequential state; and attention-based models capture relationships across long sequences. Architecture selection should follow the structure and physics of the mechatronic data.

Generalization depends on representative data and leakage-resistant validation as well as regularization. Weight decay, dropout, augmentation, early stopping, transfer learning, and hyperparameter optimization are useful tools, but none replaces sound experimental design. Evaluation must reflect class imbalance, physical units, operating costs, detection delay, and uncertainty. Interpretability methods support engineering review but do not establish causality automatically. Safe deployment requires monitoring, documentation, cybersecurity, human oversight, and separation between predictive software and safety-rated protection.

Key Terms

Neuron: A weighted-sum computational unit followed by an activation function.

Activation function: A nonlinear function applied to a neuron’s preactivation.

Bias: A trainable offset that shifts a neuron’s activation threshold.

Multilayer perceptron: A feedforward network composed of fully connected layers.

Backpropagation: Chain-rule calculation of loss gradients through a network.

Epoch: One complete pass through the training dataset.

Convolution: A local sliding operation using shared kernel parameters.

Receptive field: The input region that can influence a particular output.

Pooling: A resolution-reduction operation that aggregates local values.

RNN: A recurrent neural network that maintains a hidden state over time.

LSTM: A gated recurrent network designed to preserve useful long-term information.

Attention: A weighted information-selection mechanism based on queries and keys.

Transformer: An architecture built primarily from attention and feedforward blocks.

Overfitting: Learning training-specific patterns that fail to generalize.

Regularization: A method that discourages undesirable complexity or sensitivity.

Transfer learning: Adapting knowledge learned on one task or domain to another.

Data leakage: Unintended use of information unavailable at the intended prediction time.

Calibration: Agreement between predicted probabilities and observed frequencies.

Distribution shift: A change between training conditions and deployment conditions.

Interpretability: The extent to which model behaviour can be understood or explained.

Review Questions

Short Answer Questions

1. Why is feature scaling important when training a neural network with physical sensor variables?
2. Distinguish trainable parameters from hyperparameters.
3. What problem does a convolutional kernel solve in a vibration signal?
4. Why can a random split be inappropriate for sequential machine data?
5. State one advantage and one limitation of attention-based models.

Long Answer Questions

1. Explain the forward pass and backpropagation process in a multilayer perceptron, including the role of the loss function and learning rate.
2. Compare MLP, CNN, RNN, and Transformer architectures for mechatronic applications.
3. Discuss overfitting and explain how weight decay, dropout, augmentation, and early stopping address it.
4. Explain how precision, recall, F1 score, and decision thresholds should be selected for predictive maintenance.
5. Describe a responsible workflow for deploying a neural diagnostic system in an industrial environment.

Analytical / Application Questions

1. A vibration classifier shows 99% accuracy but misses many rare bearing faults. Which metrics, validation strategy, and threshold procedure should be investigated, and why?
2. Design a model for predicting the next 2 seconds of robot joint torque from multichannel sensor histories. Justify the input representation, architecture, loss, and deployment safeguards.
3. A visual-inspection model performs well in the laboratory but poorly on a new production line. Analyse at least four possible causes and propose an investigation plan.

Multiple Choice Questions with Answers

Question 1

Which architecture most directly exploits local spatial patterns in an image or vibration sequence?

A. Convolutional neural network

B. Linear regression

C. Decision tree

D. Naive Bayes classifier

Correct Answer: A. Convolutional neural network

Question 2

In the update θ_(t+1) = θ_t − η∇J, what does η represent?

A. The batch size

B. The learning rate

C. The validation error

D. The number of classes

Correct Answer: B. The learning rate

Question 3

Which metric is usually most directly concerned with the fraction of actual faults that are detected?

A. Precision

B. Recall

C. Mean squared error

D. Specificity only

Correct Answer: B. Recall

Question 4

For a binary classifier, which output activation commonly represents the probability of the positive class?

A. ReLU

B. Softmax over ten classes

C. Sigmoid

D. Linear activation without calibration

Correct Answer: C. Sigmoid

Question 5

What is a major reason for splitting correlated machine data by production run rather than by individual window?

A. It increases the number of sensors

B. It prevents near-duplicate information leaking across splits

C. It guarantees perfect accuracy

D. It removes the need for labels

Correct Answer: B. It prevents near-duplicate information leaking across splits

Question 6

In scaled dot-product attention, why is QK^T divided by √d_k?

A. To convert probabilities into labels

B. To reduce excessively large dot products before softmax

C. To remove all positional information

D. To force every key to zero

Correct Answer: B. To reduce excessively large dot products before softmax

Question 7

Which method adds a penalty related to the squared magnitude of weights?

A. Weight decay or L2 regularization

B. Max pooling

C. One-hot encoding

D. Early fusion

Correct Answer: A. Weight decay or L2 regularization

Question 8

A model has TP = 80 and FN = 20. What is its recall?

A. 20%

B. 50%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 9

Which statement about transfer learning is correct?

A. It always eliminates the need for target-domain validation

B. It adapts knowledge from a source task to a target task

C. It prevents distribution shift

D. It can only be used for regression

Correct Answer: B. It adapts knowledge from a source task to a target task

Question 10

Why should attention weights not automatically be treated as causal explanations?

A. Attention mechanisms cannot process sequences

B. A correlation in weighting does not establish causal influence

C. Attention weights are always exactly uniform

D. Attention is used only during data preprocessing

Correct Answer: B. A correlation in weighting does not establish causal influence