Time-Series Forecasting and State Estimation

Introduction and Context

Electrical power systems are dynamic systems: demand changes with weather and human activity, prices respond to market conditions, renewable output varies with atmospheric conditions, and equipment condition evolves through ageing and stress. Measurements are therefore not independent observations. Their order in time contains information about future values and about the hidden physical state of the system.

Time-series forecasting estimates future observations from historical and explanatory data. State estimation reconstructs unobserved system variables from imperfect measurements and a model of system behaviour. The two tasks are related but distinct. Forecasting asks, “What will happen next?” State estimation asks, “What is happening now, including variables that cannot be measured directly?”

Applications include load forecasting, electricity-price forecasting, wind and photovoltaic generation forecasting, transformer-health prediction, battery-state estimation, and real-time grid monitoring. Artificial intelligence extends classical methods by learning nonlinear relationships and extracting information from high-dimensional data, but physical models, uncertainty analysis, and engineering validation remain essential.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Explain temporal dependence, stationarity, autocorrelation, and seasonality.
2. Formulate classical statistical and AI-based forecasting models.
3. Select appropriate error metrics and prediction intervals.
4. Handle missing, noisy, asynchronous, and anomalous sensor data.
5. Explain Kalman filtering and nonlinear state estimation.
6. Combine physical models, sensor fusion, and machine learning.
7. Evaluate forecasting and estimation systems in realistic electrical-engineering settings.

1. Foundations of Time-Series Data

A time series is an ordered sequence y_t, where t denotes time. A univariate series contains one measured variable, such as hourly demand. A multivariate series contains several variables, such as demand, temperature, wind speed, and solar irradiance.

A useful conceptual decomposition is

where T_t is trend, S_t is a systematic seasonal component, and R_t is an irregular or residual component. This additive form is appropriate when seasonal variation has approximately constant absolute magnitude. A multiplicative form, y_t = T_t S_t R_t, may be more suitable when seasonal variation grows with the level of the series; logarithmic transformation often converts it approximately to an additive form.

Important properties include:

• Trend: persistent long-term movement.
• Seasonality: repeated behaviour at a known period, such as 24 hours or 168 hours.
• Cycle: longer, often less regular fluctuation.
• Autocorrelation: dependence between observations separated by a lag.
• Stationarity: statistical properties such as mean and variance do not change with time. Weak stationarity requires constant mean and variance and autocovariance dependent only on lag.
• Innovation: new information not predictable from the model’s past information.

The lag-one autocorrelation is

where Cov is covariance and σ_y² is variance. More generally, ρ_k measures dependence at lag k. Strong autocorrelation does not by itself establish causation; it may result from trend, seasonality, or common external drivers.

2. Forecasting Models

2.1 Baseline methods

A baseline is essential because a complex model should outperform a simple, transparent alternative. Common baselines are:

• Persistence: ŷ_t+h|t = y_t, often useful for short-term renewable forecasting.
• Seasonal persistence: ŷ_t+h|t = y_t+h−m, where m is the seasonal period.
• Moving average: average recent observations, reducing noise but potentially introducing lag.
• Drift model: extrapolates the average historical change.

Here ŷ_t+h|t denotes the forecast made at time t for horizon t+h.

2.2 Exponential smoothing

Simple exponential smoothing recursively updates a level estimate:

The next forecast is ŷ_t+1|t = ℓ_t. A large α responds rapidly to changes but is more sensitive to noise. Holt’s method adds trend, while Holt–Winters methods add seasonality. These methods are computationally efficient and interpretable, making them useful in operational systems with many individual meters.

2.3 Autoregressive and ARIMA models

An autoregressive model of order p is

where c is an intercept, φ_i are coefficients, and ε_t is a zero-mean innovation. For a stable AR model, the roots of its characteristic polynomial must satisfy the appropriate stationarity condition; in the AR(1) case, this is |φ₁| < 1.

An ARIMA(p,d,q) model applies differencing d times to achieve stationarity, then models the differenced series using autoregressive and moving-average terms. Seasonal ARIMA adds seasonal autoregression, differencing, and moving-average terms. These models work well when temporal structure is strong and relatively stable, but they may struggle with nonlinear weather effects and abrupt regime changes.

2.4 Regression with exogenous variables

Electrical forecasts often improve when external drivers are included:

Here T_t may be temperature, H_t humidity, W_t wind speed, and D_t a demand calendar indicator. Categorical variables can represent hour, weekday, holiday, or season. Lagged demand and lagged weather may also be included. The model is useful only when future values of the explanatory variables are known or forecast separately.

2.5 Machine-learning models

Tree ensembles, support-vector regression, multilayer perceptrons, recurrent neural networks, long short-term memory networks, temporal convolutional networks, and transformers can represent nonlinear and long-range dependencies. Inputs may include lagged measurements, weather forecasts, calendar variables, asset metadata, and spatial information from neighbouring sensors.

A neural forecasting model can be represented abstractly as

where f_θ is a parameterized model, L is the input-window length, H is the forecast horizon, x contains historical observations, and z contains known or forecast future covariates. Parameters θ are learned from training data by minimizing a loss function.

Deep learning is not automatically superior. It generally requires more data, careful regularization, computational resources, and monitoring for distribution shift. For many low-volume engineering datasets, a well-designed statistical model or gradient-boosted tree may be more reliable.

3. Uncertainty and Evaluation

A point forecast gives one value, but operational decisions require uncertainty. A prediction interval gives a range intended to contain the future observation with a stated probability. For a forecast mean with standard error s_h, an approximate 95% interval under suitable normal-error assumptions is

The value 1.96 applies to a normal distribution and is not universally valid. Prediction uncertainty usually includes both uncertainty in the estimated mean and irreducible future disturbance; it is therefore wider than a confidence interval for a mean parameter.

Useful metrics include:

MAE has the same units as the target and treats errors linearly. RMSE also has target units but penalizes large errors more heavily. Mean absolute percentage error, MAPE = (100/n)∑| (y_i−ŷ_i)/y_i |, is problematic when actual values are zero or close to zero. For energy systems, normalized errors, weighted errors, pinball loss for quantiles, and coverage of prediction intervals may be more informative.

Random train-test splitting is usually invalid for time series because it can allow future information to enter training. Use chronological splits, rolling-origin evaluation, or walk-forward validation. Features must be constructed using only information available at the forecast issue time. This is a central protection against data leakage.

For a probabilistic forecast, calibration asks whether stated probabilities match observed frequencies. For example, a nominal 90% interval should contain approximately 90% of future observations over a sufficiently representative evaluation set. Sharp intervals are desirable only when they remain calibrated.

4. Missing Data, Noise, and Anomalies

Electrical data may be missing because of communication failure, sensor maintenance, packet loss, or clock problems. Missingness may be random, periodic, or related to operating conditions. The mechanism matters: filling a systematically missing peak period using ordinary interpolation can bias a model.

Common treatments are:

• short-gap interpolation for slowly changing signals;
• seasonal or model-based imputation for periodic demand;
• forward filling only when the physical variable is expected to remain constant;
• missingness indicators so a model can learn that an observation was absent;
• deletion only when the missing fraction is small and missingness is defensibly harmless.

Imputed values should be flagged and uncertainty should reflect the fact that they are not measurements. Robust filters, physical limits, rate-of-change checks, and redundant sensors help detect anomalies. A voltage outside instrument capability, for example, may be a data error; an unusual but physically feasible voltage may be a genuine event and must not be removed automatically.

Time alignment is equally important. Sensors may use different clocks, sampling intervals, time zones, or daylight-saving conventions. A forecasting pipeline should standardize timestamps, record time quality, and distinguish measurement time from data-arrival time.

5. State Estimation and Sensor Fusion

Let x_t be a hidden state, u_t a known input, and y_t a measurement. A discrete state-space model is

where f describes state evolution, h maps state to measurements, w_t is process noise, and v_t is measurement noise. The noises represent model and sensor uncertainty, not necessarily literal random physical forces.

Examples of states include rotor angle and speed, battery state of charge, transformer winding temperature, and slowly changing sensor bias. Measurements may include voltage magnitudes, power flows, currents, frequency, temperature, or vibration.

5.1 Kalman filter

For a linear model,

with zero-mean mutually independent noises having covariances Q and R, the Kalman filter performs prediction and correction.

Prediction:

Correction:

Here x̂ is the estimated state, P is estimation-error covariance, K is the Kalman gain, A is the state-transition matrix, B is the input matrix, C is the measurement matrix, and I is the identity matrix. The innovation y_t − Cx̂_t|t−1 measures disagreement between the measurement and its prediction.

The filter assumes a sufficiently accurate linear model and known noise covariances. In practice, the Joseph covariance form may be preferred for numerical stability, and covariance matrices must remain symmetric positive semidefinite.

5.2 Nonlinear and power-system estimation

The extended Kalman filter linearizes nonlinear functions using Jacobian matrices. It can fail when nonlinearities are strong or the initial estimate is poor. The unscented Kalman filter propagates carefully selected sigma points and avoids explicit first-order linearization. Particle filters represent the posterior with weighted samples and can handle non-Gaussian distributions, but are computationally more demanding.

Traditional static power-system state estimation commonly uses weighted least squares. If measurements satisfy approximately z = h(x) + e, one solves

where R is the measurement-error covariance. This formulation gives greater weight to more precise measurements. Observability, measurement redundancy, bad-data detection, and numerical conditioning are essential engineering considerations.

6. Hybrid Forecasting and Estimation

Purely data-driven models may violate physical constraints. Purely physical models may omit complex effects or be too expensive for real-time use. Hybrid methods combine them. Examples include:

1. Residual learning: a physical forecast is corrected by a machine-learning model, y_t = y_t^phys + r_θ(x_t).
2. Physics-informed loss: the training objective penalizes both prediction error and violation of conservation or operating constraints.
3. Model-based filtering with learned components: a neural network estimates unknown loads, disturbances, or model parameters, while a Kalman filter enforces state evolution.
4. Ensemble forecasting: multiple statistical, physical, and AI forecasts are combined, often using validation-based weights.

A hybrid design should define which part of the model is trusted, how uncertainty is propagated, and what happens when the learned model encounters an unfamiliar regime.

7. Worked Example: One-Step Kalman Update

Consider a scalar temperature state. Before a measurement, the estimated temperature is 70 °C with variance 4 °C². A sensor reports 74 °C and has measurement variance 9 °C². Assume a static state for this single update, so the predicted state remains 70 °C and predicted variance is 4 °C².

The gain is

The innovation is 74 − 70 = 4 °C. Therefore,

Using the covariance update,

The estimate lies between the prior and the measurement, closer to the prior because the prior variance is smaller than the sensor variance. The result is not a replacement of the sensor value; it is a model-based fusion of two uncertain sources.

8. Illustrative Case: Distribution-Feeder Forecasting and Estimation

Background

A distribution utility operates a feeder with rooftop photovoltaic systems, electric-vehicle chargers, smart meters, and a substation measurement unit. It must forecast feeder demand 15 minutes to 24 hours ahead and estimate feeder voltage states when some low-voltage measurements are missing.

Problem

The existing persistence forecast performs acceptably during stable weather but fails during cloud transients and evening electric-vehicle charging. Missing smart-meter data cause biased feeder totals, while direct use of noisy voltage measurements produces implausible spatial patterns.

Analysis

The engineering team first audits timestamps, meter availability, weather forecasts, and topology. It separates net demand from photovoltaic generation, creates hour-of-day and weekday features, and uses a rolling-origin evaluation. A seasonal baseline, gradient-boosted regression model, and recurrent neural model are compared. For state estimation, a feeder model relates injections to voltage measurements, while a robust estimator identifies suspicious readings.

Approach and solution

The deployed design combines:

• a seasonal persistence baseline;
• a gradient-boosted model using lagged net demand, temperature, irradiance, cloud cover, calendar variables, and charger occupancy;
• probabilistic quantile forecasts to support reserve decisions;
• short-gap imputation with missingness flags;
• weighted least-squares state estimation using substation, smart-meter, and inverter measurements;
• a residual-learning correction for systematic photovoltaic forecast bias;
• alarms based on innovation magnitude and data-quality indicators.

A chronological test set is held out from model selection. The utility reports MAE for operational interpretability, RMSE for extreme errors, interval coverage, and the frequency of physically infeasible voltage estimates.

Outcome

The hybrid system improves peak-period forecasts relative to persistence and gives operators an uncertainty range rather than a single number. State estimation provides a coherent voltage profile even when individual meters fail. However, operators retain override authority because a rare feeder reconfiguration or communications outage may lie outside the training distribution.

Learning points

The case illustrates that model selection is not only an accuracy contest. Data quality, topology, forecast horizon, calibration, physical plausibility, explainability, and failure response determine operational value. A model that is slightly less accurate but stable, auditable, and robust to missing data may be preferable to a black-box model with marginally lower average error.

9. Challenges, Governance, and Best Practices

Forecast and estimation systems can affect dispatch, customer tariffs, maintenance decisions, and reliability. Important risks include biased training data, hidden data leakage, cyberattack, unsafe automation, privacy loss from high-resolution consumption data, and overconfidence in poorly calibrated predictions.

Best practices include:

• maintain a simple baseline and document all feature timestamps;
• use chronological validation and test under extreme weather and outages;
• report uncertainty, not only point accuracy;
• preserve data lineage and version models and parameters;
• monitor drift in inputs, residuals, calibration, and missingness;
• apply access control, encryption, secure update procedures, and anomaly detection;
• keep humans responsible for high-consequence decisions;
• explain model limitations to operators;
• protect customer data through minimization, aggregation, and appropriate retention;
• validate physical constraints before model outputs enter control loops.

Contemporary developments include foundation models for multivariate temporal data, graph neural networks for networked assets, digital twins, federated learning across utilities, and physics-informed neural operators. These approaches are promising but require careful evaluation for computational cost, privacy, reproducibility, out-of-distribution behaviour, and cybersecurity.

Chapter Summary

Time-series forecasting exploits temporal dependence to predict demand, prices, renewable generation, and equipment condition. Classical baselines, exponential smoothing, ARIMA, regression, and machine-learning models each have suitable operating conditions. Reliable evaluation requires chronological validation, leakage prevention, meaningful metrics, and calibrated uncertainty.

State estimation reconstructs hidden electrical variables using dynamic or static models and imperfect measurements. The Kalman filter combines prediction with measurement correction under linear-Gaussian assumptions; extended, unscented, and particle filters address more difficult cases. Weighted least squares is important in power-system estimation. Missing data, anomalies, asynchronous sensors, and model uncertainty must be treated explicitly. Hybrid physical-AI systems can improve accuracy while preserving engineering constraints, provided they are monitored, secure, explainable, and used with appropriate human oversight.

Key Terms

Temporal dependence: Dependence of a variable on its past values.

Time series: Ordered observations indexed by time.

Stationarity: Stability of relevant statistical properties over time.

Autocorrelation: Correlation between observations at different time lags.

Seasonality: Repeated pattern with a regular period.

Innovation: New information not predicted by a model.

Persistence forecast: Forecast that assumes the recent value remains unchanged.

ARIMA: Autoregressive integrated moving-average model.

Exogenous variable: External predictor used in a forecasting model.

Prediction interval: Range intended to contain a future observation with stated probability.

Data leakage: Use of information unavailable at the forecast issue time.

State estimation: Reconstruction of hidden system variables from measurements and models.

State-space model: Dynamic model linking hidden states, inputs, and observations.

Kalman filter: Recursive linear-Gaussian state-estimation algorithm.

Innovation residual: Difference between an observation and its predicted measurement.

Measurement covariance: Quantified uncertainty associated with sensor errors.

Sensor fusion: Combining information from multiple sensors or models.

Weighted least squares: Estimation method that weights measurements according to precision.

Residual learning: Learning the error remaining after a physical or baseline model.

Concept drift: Change in the statistical relationship between inputs and outputs over time.

Review Questions

Short Answer Questions

1. Distinguish trend, seasonality, and cycle in an electrical time series.
2. Why is a persistence forecast an important baseline?
3. State one limitation of MAPE for power-system data.
4. What does the Kalman gain determine?
5. Why can random train-test splitting cause leakage in time-series forecasting?

Long Answer Questions

1. Compare exponential smoothing, ARIMA, regression, and neural forecasting models for short-term load forecasting.
2. Explain the prediction and correction stages of the Kalman filter and define the roles of Q and R.
3. Discuss strategies for handling missing, anomalous, and asynchronous sensor data.
4. Explain how weighted least squares is used in static power-system state estimation.
5. Evaluate the advantages and risks of hybrid physics-AI forecasting systems.

Analytical / Application Questions

1. A feeder’s hourly demand has strong daily seasonality but only six months of data. Design a forecasting and validation strategy, including a baseline, features, metrics, and uncertainty assessment.
2. A battery sensor reports a value inconsistent with the dynamic model while two less precise sensors agree with each other. Explain how a state estimator should combine the evidence and detect possible bad data.
3. A renewable forecast has excellent MAE but prediction intervals cover only 60% of observations when nominal coverage is 90%. Diagnose the problem and recommend corrective actions.

Multiple Choice Questions with Answers

Question 1

Which property means that statistical behaviour is approximately unchanged over time?

A. Stationarity

B. Seasonality

C. Heterogeneity

D. Aliasing

Correct Answer: A. Stationarity

Question 2

For an AR(1) model, which condition supports weak stationarity?

A. φ₁ = 0

B. |φ₁| < 1

C. |φ₁| > 1

D. φ₁ = 1

Correct Answer: B. |φ₁| < 1

Question 3

Which metric penalizes large errors more strongly than MAE?

A. Median absolute error

B. RMSE

C. Prediction coverage

D. Bias only

Correct Answer: B. RMSE

Question 4

In a Kalman filter, the innovation is the difference between:

A. The process noise and measurement noise

B. The current state and previous state

C. The actual measurement and its predicted measurement

D. The forecast horizon and sampling interval

Correct Answer: C. The actual measurement and its predicted measurement

Question 5

If a sensor has larger measurement variance than the prior state estimate, the scalar Kalman update generally places more weight on:

A. The prior estimate

B. The sensor measurement

C. The calendar variable

D. The forecast horizon

Correct Answer: A. The prior estimate

Question 6

Why is random shuffling generally inappropriate for evaluating a forecasting model?

A. It increases sensor resolution

B. It can expose training to future information

C. It removes all seasonality

D. It guarantees underfitting

Correct Answer: B. It can expose training to future information

Question 7

Which method is specifically designed to combine measurements with different precision in static estimation?

A. Weighted least squares

B. One-hot encoding

C. Random oversampling

D. Moving-average smoothing only

Correct Answer: A. Weighted least squares

Question 8

A 95% interval based on ±1.96 standard errors relies primarily on which assumption?

A. Uniform errors

B. Normal-error approximation

C. Zero sampling frequency

D. Perfect measurements

Correct Answer: B. Normal-error approximation

Question 9

Which practice best reduces data leakage?

A. Using future weather observations as historical features

B. Randomly mixing all timestamps before splitting

C. Constructing features only from information available at forecast time

D. Deleting the test set

Correct Answer: C. Constructing features only from information available at forecast time

Question 10

What is a principal benefit of a hybrid physical-AI model?

A. It eliminates the need for validation

B. It can combine learned nonlinear corrections with physical structure

C. It guarantees perfect forecasts

D. It makes uncertainty irrelevant

Correct Answer: B. It can combine learned nonlinear corrections with physical structure