13. Machine Learning for Signal Processing

Introduction and Context

Digital signal processing (DSP) traditionally relies on mathematical models, transforms, filters, statistical estimation, and optimization. These methods remain indispensable, particularly when the signal structure is known and computational behavior must be predictable. However, modern sensing systems generate signals that are high-dimensional, nonstationary, multimodal, and affected by unknown disturbances. Examples include wireless waveforms, biomedical recordings, radar echoes, vibration measurements, speech, images, and industrial sensor streams. In such settings, machine learning (ML) complements classical DSP by learning representations, decision rules, or system parameters from data.

Machine learning for signal processing should not be understood as a replacement for DSP. Rather, it is an integration of signal models, domain knowledge, statistical learning, and computational architectures. Signal processing supplies operations such as sampling, denoising, spectral analysis, synchronization, and feature extraction. Machine learning supplies flexible models for classification, regression, detection, prediction, source separation, and inverse problems. The central engineering question is therefore not whether to use ML, but which parts of a signal-processing pipeline should be modeled analytically and which should be learned from data.

This chapter examines feature engineering, kernel methods, deep neural networks, convolutional and recurrent architectures, transformers, self-supervised learning, physics-informed models, and hybrid model-based/data-driven DSP. It also addresses generalization, explainability, deployment, and responsible use.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Explain how machine learning extends conventional DSP workflows.

2. Design signal features using time, frequency, time-frequency, statistical, and domain-informed representations.

3. Compare kernel methods, deep neural networks, CNNs, RNNs, and transformers for signal tasks.

4. Explain the principles of self-supervised and physics-informed learning.

5. Construct hybrid model-based/data-driven solutions for inverse and sequential problems.

6. Evaluate generalization, robustness, interpretability, and deployment risks.

7. Select suitable methods for practical communication, biomedical, audio, radar, and industrial applications.

From Signal Models to Learned Representations

A supervised signal-learning problem is often written as

\[ \hat{y}=f_\theta(x), \]

where \(x\) is a signal or signal-derived representation, \(y\) is a target, and \(f_\theta\) is a parameterized model learned by minimizing a loss over training data. For classification, the loss may be cross-entropy; for regression, mean-squared or robust losses are common. A more complete formulation includes nuisance variables such as channel conditions, sensor position, subject identity, operating load, and noise. A model that performs well only for the training nuisance distribution has memorized a condition rather than learned a transferable signal relationship.

The DSP pipeline usually contains acquisition, calibration, preprocessing, representation, inference, and decision stages. Sampling rate, anti-alias filtering, window length, overlap, normalization, and synchronization can strongly affect the learning problem. A model cannot recover information eliminated by inadequate sampling or severe clipping. Consequently, data preparation is part of system design, not merely a preliminary programming task.

Feature Engineering

Feature engineering transforms raw samples into informative and stable variables. Time-domain features include mean, variance, root-mean-square value, crest factor, zero-crossing rate, peak-to-peak amplitude, autocorrelation, and higher-order statistics. Frequency-domain features include spectral centroid, bandwidth, band power, spectral entropy, harmonic ratios, and dominant frequencies. Time-frequency features are obtained using the short-time Fourier transform (STFT), wavelets, constant-Q transforms, or filter banks.

A useful feature should have several properties: it should respond to the phenomenon of interest, be relatively insensitive to nuisance variation, preserve relevant temporal resolution, and be computationally feasible. Feature selection can remove redundant variables using correlation analysis, mutual information, regularization, or embedded importance measures. Dimensionality reduction methods such as principal component analysis may improve visualization and conditioning, but they can also discard task-relevant information if applied without care.

Feature engineering incorporates prior knowledge. For vibration diagnosis, envelope spectra may expose bearing faults; for digital communications, constellation statistics, cyclostationary features, and higher-order cumulants may distinguish modulation types; for ECG analysis, intervals, wave morphology, and heart-rate variability may be informative. Nevertheless, hand-designed features may fail when the operating domain changes. Deep models therefore often learn representations directly from samples or from minimally processed spectrograms.

Kernel Methods: Flexible Nonlinear Learning with Controlled Complexity

Kernel methods map data implicitly into a high-dimensional feature space. Given a kernel \(K(x_i,x_j)=\langle \phi(x_i),\phi(x_j)\rangle\), a nonlinear decision function can be learned without explicitly constructing \(\phi(x)\). The support vector machine (SVM) is particularly useful for moderate-sized signal datasets. Its margin-based objective seeks a separating function with good resistance to overfitting, while the kernel controls the geometry of similarity.

Common kernels include the linear, polynomial, and Gaussian radial basis function kernels. Kernel ridge regression supports nonlinear estimation with a regularized least-squares objective. One-class SVMs and related methods are useful for novelty or anomaly detection. Kernel selection is a modeling decision: a kernel expresses which pairs of signals should be considered similar. Poor scaling, irrelevant features, and an inappropriate kernel width can substantially reduce performance.

Kernel methods are attractive when labeled data are limited and feature vectors are carefully designed. They become less convenient for very large datasets because the kernel matrix grows with the number of training examples. Approximate feature maps, random Fourier features, or online variants can reduce this burden. In comparison with deep networks, kernel models are often easier to train and require fewer design choices, but they generally learn less hierarchical structure from raw long-duration signals.

Deep Architectures for Signals

Convolutional Neural Networks

A one-dimensional CNN applies learned filters across time. Convolution provides local connectivity and weight sharing, making the architecture suitable for patterns that can occur at different positions. Pooling or strided convolution increases receptive field and reduces resolution. A CNN can process raw waveforms, multichannel sensor sequences, or transformed representations such as spectrograms.

For spectrograms, two-dimensional CNNs exploit localized time-frequency patterns. The choice between raw waveform and spectrogram input depends on data volume, sampling characteristics, latency, and whether frequency structure is central to the task. Dilated convolutions enlarge the receptive field without proportionally increasing depth, which is useful for long temporal contexts.

Recurrent Neural Networks

RNNs model sequential dependence through a hidden state, commonly represented as

\[ h_t=g_\theta(x_t,h_{t-1}). \]

Vanishing and exploding gradients make simple RNNs difficult to train over long sequences. Long short-term memory (LSTM) and gated recurrent unit (GRU) architectures use gates to regulate information retention and forgetting. RNNs are useful for streaming prediction, speech, physiological monitoring, and equipment state estimation, especially when causal operation is required. Their sequential computation, however, can limit training parallelism.

Transformers and Attention

Transformers use attention to compute relationships among sequence elements. For queries, keys, and values, scaled dot-product attention is written as

\[ \mathrm{Attention}(Q,K,V)=\mathrm{softmax}(QK^T/\sqrt{d_k})V. \]

Attention can connect distant events more directly than a conventional recurrent state. Positional encodings or learned position representations preserve order. Transformers are powerful for long-range dependencies and multimodal fusion, but their memory and computation may grow substantially with sequence length. Efficient attention, patching, downsampling, and hierarchical architectures are therefore important for high-rate signals.

Architecture selection should reflect the signal and operating constraints. CNNs are strong for local, translation-tolerant patterns; RNNs suit causal sequential processing; transformers suit long-range interactions and large-scale pretraining. Hybrid CNN-RNN or CNN-transformer designs can first extract local structure and then model broader context.

Learning Paradigms Beyond Ordinary Supervision

Supervised learning requires labeled examples, but labels in signal applications may be expensive, subjective, or available only at coarse time scales. Self-supervised learning addresses this limitation by constructing a learning task from the signal itself. In masked modeling, portions of a sequence or spectrogram are hidden and reconstructed. Contrastive learning encourages augmented views of the same underlying event to have similar representations while separating unrelated examples. Predictive coding, temporal ordering, and future-segment prediction are other possibilities.

The quality of self-supervised learning depends on the augmentations and pretext task. Time shifts, amplitude scaling, additive noise, frequency masking, and channel dropout may be appropriate in one application and destructive in another. A representation learned from large unlabeled data is subsequently adapted to a smaller labeled task. This can improve sample efficiency, but pretraining data must reflect the deployment domain sufficiently to avoid learning irrelevant or biased structure.

Transfer learning is especially valuable when a general representation is available. Fine-tuning may update all parameters or only a task-specific head. Domain adaptation addresses differences between source and target distributions, but it must not conceal a genuine change in the physical phenomenon or label meaning.

Physics-Informed and Hybrid Model-Based Learning

Many signal-processing problems are inverse problems. A measurement may be described by

\[ y=\mathcal{H}(x;\eta)+n, \]

where \(x\) is the latent signal, \(\mathcal{H}\) is a physical measurement operator, \(\eta\) contains system parameters, and \(n\) is noise. Purely data-driven models may produce plausible but physically invalid reconstructions. Physics-informed learning introduces known equations, conservation laws, boundary conditions, measurement operators, or symmetry constraints into the loss or architecture.

A physics-informed objective may take the form

\[ \mathcal{L}=\mathcal{L}_{data}+\lambda_p\mathcal{L}_{physics}+\lambda_c\mathcal{L}_{constraint}. \]

The physics term can penalize residuals of a differential equation, inconsistency with a forward sensor model, violation of energy constraints, or noncausal behavior. The weight \(\lambda_p\) must be selected carefully: excessive enforcement of an inaccurate model can bias the result.

Unrolled networks provide a particularly influential hybrid strategy. An iterative algorithm such as gradient descent, proximal optimization, or approximate message passing is expressed as a finite sequence of layers. The structure of the algorithm is retained, while step sizes, thresholds, transforms, or regularizers are learned. Examples include learned iterative reconstruction for compressed sensing, denoising, channel estimation, and inverse imaging. Such models often offer improved interpretability and data efficiency compared with unconstrained black-box networks.

Generalization, Evaluation, and Explainability

Generalization means maintaining performance on new observations, devices, subjects, environments, or operating conditions. Randomly splitting highly correlated windows from the same recording can produce overly optimistic results because nearly identical segments appear in both training and test sets. Evaluation should instead reflect deployment: subject-wise, device-wise, time-wise, or environment-wise separation may be necessary.

Important evaluation dimensions include accuracy, precision, recall, F1 score, calibration, receiver operating characteristic or precision-recall curves, mean absolute error, latency, memory, energy consumption, and robustness to noise or distribution shift. For imbalanced detection, accuracy alone is inadequate. Confidence calibration is important when decisions trigger alarms or control actions.

Explainability is not a single technique. Feature attribution, saliency maps, occlusion tests, prototype examples, counterfactual analysis, and concept-based explanations answer different questions. A spectrogram saliency map may show where a classifier attended, but it does not prove causal dependence. Explanations should be tested for stability, faithfulness, and agreement with domain knowledge. In safety-critical applications, interpretable intermediate variables, uncertainty estimates, and human review may be more valuable than attractive visualizations.

Uncertainty can be represented through probabilistic models, ensembles, Monte Carlo dropout, or conformal prediction. A system should be able to abstain or request expert review when the input is outside its training distribution. Robustness testing should include sensor failure, adversarial perturbation, calibration drift, missing channels, and realistic interference.

Applications Across Engineering Practice

In wireless communications, ML assists modulation recognition, channel estimation, beam selection, spectrum sensing, equalization, and resource allocation. The design must account for signal-to-noise ratio, synchronization errors, protocol variation, and regulatory constraints. In biomedical DSP, models analyze ECG, EEG, EMG, ultrasound, and wearable signals for detection or monitoring. Patient-specific variation and privacy make subject-independent evaluation essential.

In audio and speech, neural models support enhancement, source separation, keyword detection, and acoustic scene analysis. In radar and sonar, learning supports target classification, tracking, waveform adaptation, and clutter suppression. Industrial systems use vibration, current, acoustic emission, and temperature signals for predictive maintenance. In each domain, the highest-value solution often combines a conventional front end with a learned decision or estimation stage.

Illustrative Case Study: Hybrid Bearing-Fault Diagnosis

Background. An industrial motor is monitored using tri-axial accelerometers. Bearing defects generate periodic impulses, but speed variation, structural resonances, and changing loads make a fixed threshold unreliable.

Problem. The engineering team has limited labeled fault recordings and abundant unlabeled normal-operation data. It must detect early faults while limiting false alarms and provide evidence that maintenance engineers can interpret.

Analysis. Raw acceleration is calibrated and segmented synchronously with shaft-speed measurements. Order tracking compensates for speed variation. Envelope analysis and an STFT produce representations emphasizing impulsive and resonant behavior. Random window-level splitting is rejected because neighboring windows are highly correlated; entire operating runs are reserved for testing.

Approach. A self-supervised CNN is first trained to reconstruct masked time-frequency regions using normal and mixed-condition recordings. Its encoder is then fine-tuned for fault classification with a small labeled set. A model-based constraint compares predicted fault periodicity with shaft-speed-related characteristic frequencies. An uncertainty threshold routes ambiguous cases to an engineer. Explanations include the relevant spectral bands, time locations of impulses, and comparison with healthy reference prototypes.

Outcome. The hybrid design is expected to be more robust than a classifier trained on randomly sampled windows because it uses speed information, physically meaningful representations, and run-level evaluation. Its operational success is assessed not only by F1 score but also by false alarms per operating hour, detection lead time, calibration, and maintenance usefulness.

Learning points. The case demonstrates that labeling strategy, evaluation design, physical constraints, and human-facing evidence are as important as network architecture. It also illustrates how self-supervised learning can exploit unlabeled data without abandoning established vibration-analysis methods.

Risks, Governance, and Professional Practice

Common mistakes include data leakage through preprocessing or window splitting, selecting a model before defining the deployment task, ignoring class imbalance, optimizing only average accuracy, and treating saliency as proof of causation. Excessive hyperparameter search on a fixed test set converts the test set into training information. Reproducibility requires documenting sampling, filtering, segmentation, normalization, labels, data exclusions, random seeds, and hardware.

Signal data may contain health information, voice identity, location, or industrial secrets. Collection and deployment therefore require lawful processing, access control, retention limits, and appropriate anonymization. Biomedical systems must be validated according to applicable medical-device procedures; communications and industrial systems may have sector-specific safety, cybersecurity, and electromagnetic requirements. A learned controller or detector should not be deployed without fail-safe behavior, monitoring, rollback procedures, and lifecycle maintenance.

Fairness is relevant when performance varies across demographic groups, body types, devices, environments, or operating regimes. Dataset documentation, subgroup evaluation, and continuous monitoring help expose such differences. Models should be updated cautiously because changing data distributions can alter both performance and the meaning of learned features.

Emerging Directions and Best Practices

Current research is moving toward foundation models for time series and multimodal sensor data, efficient edge inference, neural operators, diffusion-based signal reconstruction, federated learning, continual learning, and hardware-aware neural architecture design. These developments are promising but do not remove the need for validation. Larger models can amplify data bias and may be impractical under strict latency or power limits.

A disciplined workflow is as follows:

• Define the operational decision, cost of errors, latency, and allowable uncertainty.
• Establish a classical DSP baseline before adding ML.
• Build leakage-resistant train, validation, and test partitions.
• Inspect signal quality, label reliability, and domain coverage.
• Choose representations and architectures according to signal structure and deployment constraints.
• Use augmentation and self-supervision only when their invariances are physically justified.
• Compare pure data-driven, model-based, and hybrid alternatives.
• Evaluate robustness, calibration, subgroup behavior, computational cost, and failure modes.
• Preserve human oversight and monitor performance after deployment.

The most reliable systems are rarely the most purely data-driven. They are systems in which sampling, filtering, physical structure, learned representation, uncertainty, and operational decision-making are designed together.

Chapter Summary

Machine learning extends DSP by learning nonlinear representations, estimators, and decision rules from signal data. Feature engineering remains valuable when domain structure is known, while kernel methods offer effective nonlinear learning for moderate-sized datasets. CNNs capture local patterns, RNNs model sequential state, and transformers represent long-range dependencies through attention. Self-supervised learning reduces dependence on labels, and physics-informed and unrolled models incorporate measurement laws and iterative algorithms. Hybrid systems often provide a practical balance of accuracy, data efficiency, interpretability, and robustness. Reliable deployment requires leakage-resistant evaluation, uncertainty estimation, explainability, privacy protection, monitoring, and attention to domain shift. The central design principle is to combine learned flexibility with the structure and discipline of signal processing.

Key Terms

Feature engineering: Design of informative variables from raw or preprocessed signals.

Kernel: A function measuring similarity while implicitly defining a feature-space representation.

Support vector machine: A margin-based supervised learning method that can use nonlinear kernels.

Convolutional neural network: A neural architecture using local shared filters to learn spatial or temporal patterns.

Recurrent neural network: A sequence model that maintains a hidden state over time.

LSTM: A gated recurrent architecture designed to preserve information over long sequences.

Transformer: An attention-based architecture for modeling relationships among sequence elements.

Attention: A weighted mechanism that selects and combines information from other positions.

Self-supervised learning: Learning from targets generated automatically from the data itself.

Contrastive learning: Representation learning that brings related views together and separates unrelated examples.

Transfer learning: Reusing a learned representation for a new task or domain.

Physics-informed learning: Training constrained by physical laws, equations, or measurement models.

Hybrid model: A system combining analytical signal models with learned components.

Unrolled network: A neural architecture derived by converting an iterative algorithm into learnable layers.

Generalization: Performance on data or conditions not used during training.

Data leakage: Unintended transfer of test or future information into model training.

Calibration: Agreement between predicted confidence and observed correctness.

Distribution shift: A change between training and deployment data distributions.

Explainability: Methods for relating model outputs to inputs, concepts, or decision evidence.

Uncertainty estimation: Quantification of confidence or ambiguity in a model prediction.

Review Questions

Short Answer Questions

1. Why can feature engineering remain useful when deep neural networks are available?

2. State two differences between CNNs and RNNs for signal processing.

3. What is the purpose of a physics-informed loss term?

4. Define data leakage and give one signal-processing example.

5. Why is calibration important in an automated alarm system?

Long Answer Questions

1. Explain the roles of time-domain, frequency-domain, and time-frequency features, with suitable signal examples.

2. Compare kernel methods, CNNs, RNNs, and transformers in terms of data requirements, temporal modeling, and computational behavior.

3. Describe self-supervised learning for signals and discuss how augmentation choices affect the learned representation.

4. Explain unrolled networks and physics-informed learning as forms of hybrid model-based/data-driven DSP.

5. Develop a complete evaluation and deployment plan for a learned biomedical or industrial signal-processing system.

Analytical / Application Questions

1. A classifier achieves excellent accuracy when windows from the same recording are randomly divided between training and test sets but performs poorly on a new machine. Diagnose the evaluation problem and propose a revised protocol.

2. Design a hybrid architecture for channel estimation in a communication receiver when a partially known measurement model and a small labeled dataset are available. Justify each component.

3. A transformer gives higher average accuracy than a CNN but exceeds the latency and memory limits of an edge device. Formulate a decision framework for selecting and adapting the deployed model.

22. Multiple Choice Questions with Answers

Question 1

Which practice most directly prevents overly optimistic evaluation caused by correlated signal windows?

A. Using a larger batch size

B. Splitting data by recording, subject, or operating run

C. Increasing the number of convolutional layers

D. Applying min-max normalization

Correct Answer: B. Splitting data by recording, subject, or operating run

Question 2

Which architecture is most naturally suited to learning local time-frequency patterns in a spectrogram?

A. Convolutional neural network

B. Ordinary least-squares estimator

C. One-class SVM only

D. Kalman filter without learned parameters

Correct Answer: A. Convolutional neural network

Question 3

In a physics-informed objective, the term L_physics is primarily used to

A. Increase the number of training labels

B. Penalize violation of known physical relationships

C. Replace all measurement data

D. Guarantee zero classification error

Correct Answer: B. Penalize violation of known physical relationships

Question 4

What is the main purpose of masking in self-supervised signal learning?

A. To create an automatically generated prediction task from the signal

B. To remove all frequency information permanently

C. To make labels unnecessary during final evaluation

D. To guarantee invariance to every transformation

Correct Answer: A. To create an automatically generated prediction task from the signal

Question 5

A transformer may be preferred over a recurrent model when the task requires

A. Only single-sample amplitude measurements

B. Long-range relationships among sequence elements

C. No positional information

D. A model with strictly sequential training computation

Correct Answer: B. Long-range relationships among sequence elements

Question 6

Which example best represents data leakage in signal classification?

A. Filtering each recording before segmentation

B. Using subject-specific recordings in both training and testing

C. Reporting precision and recall for an imbalanced dataset

D. Testing robustness under additive noise

Correct Answer: B. Using subject-specific recordings in both training and testing

Question 7

An unrolled network differs from an unconstrained deep network because it

A. Contains layers derived from an iterative analytical algorithm

B. Cannot contain learnable parameters

C. Requires no training data under any condition

D. Uses only frequency-domain inputs

Correct Answer: A. Contains layers derived from an iterative analytical algorithm

Question 8

Why may accuracy be inadequate for evaluating a rare-event detector?

A. Accuracy cannot be computed for binary decisions

B. A large number of normal cases can conceal poor rare-event detection

C. Accuracy always measures calibration instead

D. Accuracy is independent of class proportions

Correct Answer: B. A large number of normal cases can conceal poor rare-event detection

Question 9

Which explanation claim is most scientifically cautious?

A. A saliency map always proves causal reasoning

B. Highlighted regions may indicate model attention but require faithfulness testing

C. Any high-confidence prediction is physically valid

D. Feature attribution eliminates the need for test data

Correct Answer: B. Highlighted regions may indicate model attention but require faithfulness testing

Question 10

In the bearing-fault case, order tracking was used mainly to

A. Remove all sensor noise without modeling

B. Compensate for speed variation in interpreting periodic fault structure

C. Convert classification into image recognition

D. Replace the need for labeled examples entirely

Correct Answer: B. Compensate for speed variation in interpreting periodic fault structure