Electrical Engineering Data and Signal Preparation

Introduction

Artificial-intelligence systems in electrical engineering rarely fail because an algorithm cannot be trained. More often, they fail because the data are incorrectly sampled, poorly synchronized, contaminated by noise, incompletely labelled, or interpreted without regard to the physical system that produced them. A machine-learning model trained on distorted voltage waveforms, misaligned phasor measurements, duplicated SCADA records, or images with inconsistent illumination may produce confident but unsafe conclusions.

Electrical-engineering data are heterogeneous. They include continuous-time waveforms, sampled oscillographic records, synchrophasor streams, scalar sensor measurements, thermal and visible-light images, event logs, and supervisory control and data acquisition (SCADA) time series. Each data type has its own sampling characteristics, error mechanisms, units, timestamps, and suitable preparation methods.

Data preparation therefore links physical measurement and AI. It converts raw observations into trustworthy, synchronized, information-rich representations while preserving the phenomena relevant to engineering decisions.

Learning objectives

After studying this chapter, the reader should be able to:

1. Classify common electrical-engineering data and identify their quality risks.
2. Select sampling rates and anti-aliasing strategies for measured signals.
3. Synchronize waveform, phasor, sensor, image, event-log, and SCADA data.
4. Detect and handle missing values, outliers, noise, duplicates, drift, and inconsistent units.
5. Apply time-domain, frequency-domain, and time-frequency transformations.
6. Engineer physically meaningful features for AI models.
7. Use normalization, dimensionality reduction, and data-quality controls appropriately.
8. Evaluate the limitations, safety implications, and governance requirements of prepared engineering data.

1. Electrical-engineering data as measured evidence

A measurement is not the physical quantity itself. It is an observation produced by a sensor, transducer, instrument transformer, data-acquisition system, communication network, and storage process. A useful abstraction is

where x(t) is the underlying physical quantity, h[·] represents the measurement-chain response, n(t) is random or structured noise, and b(t) is bias or slowly varying error. The model is conceptual rather than universal; nonlinear sensors, saturation, quantization, and missing observations may require additional terms.

Major data types

Data preparation should begin with a data dictionary. For every variable, record its name, engineering meaning, unit, sensor location, expected range, sampling or reporting interval, timestamp definition, quality flag, and calibration history. This documentation is essential for reproducible AI development.

2. Sampling and reconstruction of signals

A continuous signal becomes a sequence of samples when measured at times separated by the sampling interval T_s. The sampling frequency is

where f_s is in hertz and T_s is in seconds. For a band-limited signal whose highest significant frequency is f_max, ideal uniform sampling requires

This is the Nyquist condition. It assumes a genuinely band-limited signal, ideal sampling, and adequate reconstruction. In practice, engineers use margin above twice the highest frequency because real filters are not ideal and signals may contain unexpected transients.

If a frequency component exceeds half the sampling frequency, it can appear as a false lower frequency. This phenomenon, aliasing, cannot generally be removed after sampling. An anti-aliasing low-pass filter is therefore placed before the analog-to-digital converter. Its transition band must fit between the highest frequency of interest and the Nyquist frequency.

For example, suppose a current-monitoring system must retain components up to 8 kHz. A sampling rate of 20 kHz gives a Nyquist frequency of 10 kHz, leaving only a 2 kHz transition band. A rate of 40 kHz gives a 20 kHz Nyquist frequency and a more practical filter margin. The higher rate increases storage and computation but reduces aliasing risk.

Quantization introduces amplitude error. With an ideal converter having N equally spaced levels over a full-scale span V_FS, the step size is approximately

where ∆ has the same units as the measured voltage. The quantization error is commonly bounded by approximately ±∆/2 when the signal is not overloaded and quantization behaves approximately as uncorrelated noise. Clipping is different: it is a nonlinear loss of information caused by exceeding the converter range.

Sampling-rate conversion must also be controlled. Before downsampling by an integer factor M, low-pass filtering is required so that retained content lies below the new Nyquist frequency. Interpolation, in contrast, estimates values between samples and does not create genuinely new information.

3. Time bases and synchronization

Synchronization is the process of assigning observations to a common temporal reference. It is essential when estimating phase differences, determining event sequence, calculating propagation delay, or combining a waveform with weather or equipment data.

Time stamps may represent sample start, sample center, acquisition completion, transmission time, or database insertion time. These meanings must not be confused. A communication delay is not necessarily a measurement-time error.

Important time errors include:

• Clock offset: a constant difference between a device clock and reference time.
• Clock drift: a changing offset caused by unequal clock rates.
• Jitter: short-term variation in sampling or timestamp timing.
• Latency: delay between measurement and delivery.
• Time-zone and daylight-saving errors: conversion mistakes in human-readable timestamps.
• Dropped or duplicated records: discontinuities in the observation sequence.

A practical synchronization workflow is:

1. Convert all timestamps to a documented standard, normally UTC with an explicit time scale.
2. Preserve the original timestamp and clock-quality metadata.
3. Estimate offset and drift using a reference clock, pulse-per-second signal, network time protocol, precision time protocol, or known event.
4. Correct timestamps without silently overwriting raw data.
5. Resample only after understanding the original sampling process.
6. Quantify residual timing uncertainty.

For two sampled signals, cross-correlation can estimate relative delay. For discrete sequences x[n] and y[n], a commonly used unnormalized cross-correlation is

where k is an integer lag and the summation is taken over valid overlapping samples. The lag producing the largest physically plausible correlation may estimate delay. This method can be misleading when signals are periodic, weakly related, or affected by changing operating conditions; synchronization should therefore use independent timing evidence whenever safety depends on it.

For sinusoidal steady-state signals, a time error ∆t produces phase error approximately

where f is frequency in hertz and ∆θ is in degrees. At 50 Hz, a 1 millisecond error corresponds to 18°. Thus, millisecond-level timing errors can be significant in phasor applications.

4. Cleaning and quality control

Cleaning is not the indiscriminate removal of unusual values. An unusual value may be a genuine fault transient, switching event, or overload. The correct question is whether the observation is plausible, correctly timed, and suitable for the intended task.

Range and physical-consistency checks

Use engineering limits, instrument limits, and relationships among variables. Examples include:

• temperature cannot change faster than the equipment permits;
• a binary breaker-status variable should use an approved state set;
• three-phase measurements should satisfy approximate physical relationships under stated operating conditions;
• power, voltage, and current should have compatible units and plausible magnitudes;
• a sensor value outside its calibrated range should be flagged, not automatically treated as zero.

Missing data

Missingness may be random, caused by communication failure, or caused by the operating condition itself. The mechanisms are often described as missing completely at random, missing at random, and missing not at random. In electrical systems, missingness is frequently informative: a protection operation may interrupt communications, and a sensor may fail at high temperature.

Common treatments include:

• retaining a missing-value indicator;
• forward filling only when the variable is physically persistent and the time gap is bounded;
• interpolation for short gaps in slowly varying measurements;
• model-based imputation when assumptions are validated;
• excluding records when imputation would invent a transient;
• using algorithms that explicitly handle missing values.

Never interpolate across a fault transient merely to make a plot appear continuous.

Outliers and noise

A statistical outlier is not automatically an error. Robust methods such as the median and median absolute deviation can help identify suspicious observations without being dominated by a few extreme values. A value should be labelled with its reason for exclusion or modification, such as sensor saturation, communication corruption, or confirmed physical event.

Noise may be reduced through filtering, averaging, or state estimation. Each method changes the signal. Filtering can attenuate peaks, shift phase, or remove short-duration evidence. The selected filter must therefore be documented with its passband, stopband, order, phase behavior, edge treatment, and units.

5. Filtering and spectral analysis

A linear time-invariant filter can be represented by its impulse response h[n]. For a discrete input x[n], the output is

The equation assumes linearity, time invariance, and a defined treatment of finite-record boundaries. A filter’s frequency response describes how amplitude and phase vary with frequency.

Finite impulse response filters can provide exactly linear phase when designed with symmetric coefficients, but may require more coefficients. Infinite impulse response filters can achieve sharp responses efficiently but may have nonlinear phase and require stability checks. Applying a causal filter forward in time introduces delay; forward-backward filtering can produce approximately zero phase but is noncausal and unsuitable for real-time protection decisions. It also has edge effects.

The discrete Fourier transform (DFT) of a sequence x[n] of length N is

where j = √(-1), k is the frequency-bin index, and the corresponding frequency is f_k = kf_s/N for the nonnegative-frequency interpretation. The DFT assumes the analysed record is treated as one finite block; discontinuity at the block boundary causes spectral leakage.

The frequency resolution is

where T = N/f_s is record duration. Increasing the record length improves frequency spacing, but may reduce the ability to track rapid changes. Window functions reduce leakage by tapering the record, at the cost of wider main lobes and altered amplitude interpretation. Amplitude correction and one-sided versus two-sided spectrum conventions must be documented.

The fast Fourier transform (FFT) is an efficient algorithm for computing the DFT; it is not a different transform. For nonstationary electrical signals, short-time Fourier transforms, wavelets, or other time-frequency methods may be more informative because they preserve some time localization.

6. Feature engineering for AI

Feature engineering converts prepared observations into variables that represent useful physical or statistical structure. It should be guided by the task, operating regime, and available labels.

Waveform features

Time-domain features include mean, root-mean-square value, peak, peak-to-peak value, crest factor, zero-crossing rate, and energy. For samples x₁,…,x_N, the RMS value is

It has the same units as x_n. For a periodic sinusoid with peak amplitude X_pk, the RMS value is X_pk/√2, provided the waveform is an ideal sinusoid with zero mean.

For a signal with nonzero mean, the standard deviation is

where x̄ is the sample mean. The denominator N−1 is used for the usual unbiased sample-variance estimate under independent, identically distributed sampling assumptions; it is not automatically correct for every time-series or population calculation.

Frequency-domain features may include fundamental magnitude, harmonic ratios, spectral centroid, band power, and interharmonic content. For a power-system waveform, harmonic distortion must specify the reference component and bandwidth. A commonly used total harmonic distortion definition for a periodic waveform is

where V₁ is fundamental RMS voltage, V_h is the RMS magnitude of harmonic order h, and H is the highest included order. The definition should state whether supraharmonics, noise, and measurement bandwidth are included.

Phasor, sensor, and event features

Useful phasor features include magnitude, angle relative to a declared reference, positive-, negative-, and zero-sequence components, rate of change of frequency, and differences between geographically separated measurements. Angles must not be compared unless reference conventions and time alignment are consistent.

Sensor features may include rolling mean, slope, variance, cumulative operating hours, temperature-compensated values, and deviation from a baseline. Event-log features include event count, inter-event time, sequence order, alarm persistence, and time from command to response. Event data require careful distinction between event occurrence time and reporting time.

Image features may be learned directly by convolutional models, or engineered using temperature statistics, hot-spot area, edge density, texture, and region-of-interest measurements. Image normalization should not remove the thermal or visual evidence relevant to the diagnostic task.

Windowing and labels

Time-series records are often divided into windows. Window length should reflect the phenomenon: a protection transient may require milliseconds, whereas transformer ageing may require months. Overlapping windows increase the number of training examples but can cause leakage if windows from the same event appear in both training and test sets. Splitting must occur by event, asset, or time period when appropriate.

Labels must be defined independently of the signal-processing pipeline. If a fault label is assigned using a post-event measurement that is also included as an input feature, the model may obtain unrealistic information unavailable at prediction time.

7. Scaling, representation, and dimensionality reduction

Feature scaling prevents variables with large numerical units from dominating algorithms that use distances or gradients. Standardization is

where μ and σ are calculated from the training set only. Min–max scaling is

and maps values to [0,1] only when the stated minimum and maximum are appropriate. Scaling parameters must be saved and applied unchanged to validation, test, and operational data. Robust scaling may be preferable when extreme values are genuine or frequent.

Principal component analysis (PCA) represents correlated variables using orthogonal directions of decreasing variance. After centering a data matrix, PCA finds eigenvectors of its covariance matrix. The first component captures the largest possible variance among unit-length linear projections; subsequent components are orthogonal and capture the largest remaining variance. PCA is useful for compression, visualization, and denoising, but high variance is not necessarily high diagnostic relevance. PCA components also need not have direct physical meaning.

Other methods include singular value decomposition, autoencoders, and domain-specific transformations such as symmetrical components or modal coordinates. Dimensionality reduction must be fitted only on training data and assessed for reconstruction loss, task performance, stability, and interpretability.

8. Data-quality controls and reproducible pipelines

A production-quality preparation pipeline should be versioned and testable. A typical sequence is:

Figure: A traceable data-preparation pipeline with continuous quality monitoring.

Quality controls should include completeness, timestamp monotonicity, sampling-interval stability, range validity, unit consistency, duplicate detection, missingness rate, sensor health, and distribution drift. Each record should retain provenance: source device, software version, transformation steps, calibration status, and quality flags.

Train–test leakage is a major risk. Imputation, scaling, feature selection, PCA, and threshold selection must be fitted using training data only. Time-dependent data should generally be evaluated using chronological or asset-based splits rather than random shuffling.

9. Worked example: sampling and frequency resolution

An instrument records a 50 Hz voltage waveform at 10 kHz for 0.2 s.

The number of samples is

The DFT frequency spacing is

Thus, a 50 Hz component lies at the tenth frequency bin, because 50/5 = 10. If the instrument must detect a 7.5 kHz component, the Nyquist frequency is 5 kHz, so the component aliases and cannot be recovered from this record. A higher sampling rate and an appropriate anti-aliasing filter are required.

The example also shows that increasing the sample count by increasing duration improves frequency spacing only if the signal remains sufficiently stationary. It does not automatically improve transient timing or eliminate sensor noise.

10. Illustrative case: transformer condition monitoring

Background. A utility combines transformer oil temperature, load current, dissolved-gas measurements, infrared images, SCADA alarms, and relay event records to develop an early-warning model.

Problem. Initial model performance is high in laboratory testing but poor in field operation. Investigation finds that SCADA timestamps represent database arrival time, infrared images have no common time zone, several temperature channels use different units, and windows from the same transformer are randomly divided between training and test sets.

Analysis. The apparent performance is inflated by temporal leakage and asset-specific signatures. Arrival-time alignment also makes the model associate an alarm with measurements that were not available when the alarm decision was required. Some missing temperature values are caused by communications loss during abnormal operation, so treating them as ordinary random gaps is inappropriate.

Approach. Engineers create a data dictionary, preserve raw records, convert timestamps to UTC, retain both event and arrival times, and estimate communication latency. They validate units and sensor ranges, add missingness and quality indicators, and use physically justified interpolation only for short, non-event gaps. Images are corrected for known sensor calibration issues without removing hot-spot information. Feature windows end at the prediction time, and data are split chronologically by transformer. Scaling and PCA are fitted on the training period only.

Outcome. Offline accuracy decreases, but the estimate becomes credible. False alarms are reduced, and the model’s predictions remain more stable during seasonal changes. Operators receive a quality flag when the decision relies on degraded communications or stale measurements.

Learning points. Data preparation is part of system design, not a cosmetic preprocessing stage. Correct timestamps, leakage control, provenance, and operator-facing quality information may matter more than selecting a more complex AI model.

11. Limitations, safety, and governance

Filtering and imputation can hide faults. A preparation pipeline used for protection, control, or safety assessment must distinguish real-time causal processing from offline analysis. A zero-phase filter may be acceptable for retrospective classification but invalid for a real-time trip decision because it uses future samples.

AI datasets may contain sensitive operational information, including grid topology, customer load patterns, industrial processes, and security-relevant events. Access control, encryption, retention limits, and audit logs are therefore appropriate. Data sharing should remove unnecessary identifiers while preserving engineering meaning.

Model developers should document sensor uncertainty, known blind spots, class imbalance, operating regimes, and conditions under which the model must defer to conventional protection or human review. A model should not convert a low-quality measurement into an apparently precise engineering conclusion.

Contemporary systems increasingly use edge computing, streaming analytics, digital twins, physics-informed features, self-supervised learning, and standardized asset-data models. These technologies do not remove the need for sampling theory, calibration, synchronization, and quality control. In fact, distributed AI increases the importance of consistent metadata and versioned preprocessing across devices.

Best-practice checklist

• Define the engineering question before choosing a transformation.
• Preserve raw data and never overwrite original timestamps.
• Record units, calibration, sensor location, and quality flags.
• Use anti-aliasing before downsampling.
• Quantify timing offset, drift, jitter, and latency.
• Treat missingness as potentially informative.
• Distinguish genuine abnormal events from sensor errors.
• Document filter phase, edge handling, and frequency response.
• Fit scaling, imputation, feature selection, and PCA on training data only.
• Split data by time, asset, or event when random splitting causes leakage.
• Validate features against physical knowledge and operational availability.
• Monitor data drift and pipeline failures after deployment.
• Retain provenance so every model input can be traced to its source.

Chapter Summary

Electrical-engineering AI depends on reliable preparation of heterogeneous measurements. Waveforms require appropriate sampling, anti-aliasing, quantization awareness, filtering, and spectral analysis. Phasors and distributed sensors require accurate common time bases because small timing errors can create substantial phase errors. SCADA and event logs require careful interpretation of scan time, occurrence time, reporting time, stale values, duplicates, and quality flags. Images require calibration and preservation of diagnostically meaningful spatial or thermal information.

Cleaning should combine statistical checks with engineering constraints. Missing values, outliers, noise, and drift must be treated according to their physical causes. Feature engineering can expose RMS magnitude, harmonics, trends, event timing, spatial defects, and other useful structure, but features must be available at the intended prediction time. Scaling and PCA can improve computational performance, yet both must be fitted without test-data leakage. A traceable pipeline includes provenance, quality flags, reproducibility, security, and continuous monitoring. Sound preparation does not guarantee a successful AI model, but poor preparation makes trustworthy performance impossible.

Key Terms

Sampling frequency: Number of samples acquired per second.

Nyquist frequency: One-half of the sampling frequency.

Aliasing: False frequency representation caused by insufficient sampling.

Anti-aliasing filter: Analog or digital filtering used to limit frequency content before downsampling or conversion.

Quantization: Mapping of continuous amplitude values to discrete converter levels.

Clock offset: Difference between a device clock and a reference clock.

Clock drift: Change in clock offset over time.

Jitter: Short-term variation in sampling or timing.

Latency: Delay between measurement, transmission, and availability.

SCADA: Supervisory control and data acquisition system.

Phasor: Complex representation of sinusoidal magnitude and phase at a specified frequency.

Cross-correlation: Measure used to compare signals at different relative lags.

Spectral leakage: Spreading of finite-record signal energy across frequency bins.

Frequency resolution: Separation between adjacent DFT frequency bins.

Feature engineering: Construction of informative variables from prepared data.

RMS value: Square root of the mean squared signal value.

Principal component analysis: Linear transformation into orthogonal directions ordered by variance.

Data leakage: Use of information during model development that would not be available at prediction time.

Provenance: Record of a datum’s source, transformations, and history.

Data drift: Change in data distributions or relationships after deployment.

Review Questions

Short Answer Questions

1. What is aliasing, and why can it not generally be removed after sampling?
2. Distinguish clock offset, clock drift, jitter, and latency.
3. Why should raw timestamps and original data be preserved?
4. State two risks of applying a filter to an electrical waveform.
5. Why must scaling parameters be estimated from training data only?

Long Answer Questions

1. Explain a complete preparation workflow for combining waveform, phasor, SCADA, and event-log data for fault classification.
2. Discuss how missingness, outliers, and sensor drift should be handled using both statistical and engineering evidence.
3. Explain DFT frequency resolution, spectral leakage, windowing, and the trade-off between record duration and transient detection.
4. Compare causal filtering, zero-phase offline filtering, FIR filters, and IIR filters in an electrical-engineering application.
5. Describe how feature engineering and dimensionality reduction can improve an AI model while also creating risks of information loss or leakage.

Analytical / Application Questions

1. A system samples at 12 kHz and contains useful signal content up to 4 kHz. Determine whether the nominal Nyquist condition is satisfied and explain what additional practical design margin should be considered.
2. Two 50 Hz phasor measurement units have a relative timing error of 0.5 ms. Estimate the corresponding phase error using the small timing-error relationship and discuss its significance.
3. Design a validation plan for a transformer-monitoring model whose data include seasonal measurements, repeated records from the same assets, alarms, and communication outages. Justify the split strategy, quality controls, and leakage protections.

Multiple Choice Questions with Answers

Question 1

A signal contains significant frequency content up to 6 kHz. Which sampling frequency satisfies the ideal Nyquist condition?

A. 8 kHz

B. 10 kHz

C. 12 kHz

D. 6 kHz

Correct Answer: C. 12 kHz

Question 2

At 50 Hz, what phase error is approximately produced by a 1 ms timing error?

A. 1.8°

B. 9°

C. 18°

D. 50°

Correct Answer: C. 18°

Question 3

What is the principal purpose of an anti-aliasing filter?

A. Increase converter resolution

B. Limit frequency content before sampling

C. Synchronize independent clocks

D. Replace missing timestamps

Correct Answer: B. Limit frequency content before sampling

Question 4

For a 0.25 s record, the DFT frequency spacing is:

A. 0.25 Hz

B. 2 Hz

C. 4 Hz

D. 25 Hz

Correct Answer: C. 4 Hz

Question 5

Which action most directly creates data leakage?

A. Retaining a sensor-quality flag

B. Fitting a scaler using training data

C. Using future post-fault measurements as prediction inputs

D. Converting all timestamps to UTC

Correct Answer: C. Using future post-fault measurements as prediction inputs

Question 6

The RMS value of an ideal zero-mean sinusoid with peak amplitude X_pk is:

A. X_pk

B. X_pk²

C. X_pk/2

D. X_pk/√2

Correct Answer: D. X_pk/√2

Question 7

Why can random window-level splitting overestimate time-series model performance?

A. It always removes all outliers

B. Overlapping windows from one event may occur in both training and test sets

C. It prevents feature scaling

D. It reduces the sampling frequency

Correct Answer: B. Overlapping windows from one event may occur in both training and test sets

Question 8

Which statement about PCA is correct?

A. It guarantees that the highest-variance direction is most diagnostic

B. It creates orthogonal directions ordered by captured variance

C. It eliminates the need for synchronization

D. It can be fitted using test data without consequence

Correct Answer: B. It creates orthogonal directions ordered by captured variance

Question 9

What should be done with a genuine fault transient identified as an unusual value?

A. Automatically delete it as an outlier

B. Replace it with the global mean

C. Retain and label it according to the engineering context

D. Clip it to the median

Correct Answer: C. Retain and label it according to the engineering context

Question 10

Which timestamp distinction is essential in SCADA and event-log analysis?

A. File size versus file name

B. Event occurrence time versus reporting or arrival time

C. Voltage unit versus current unit

D. Image width versus image height

Correct Answer: B. Event occurrence time versus reporting or arrival time