Time-Series, Sequential and Forecasting Algorithms

Introduction and context

A time series is a sequence of observations indexed by time: sales recorded daily, electricity demand measured every 15 minutes, or machine vibration sampled every second. Unlike ordinary independent observations, temporal data may exhibit dependence: the value at time t can be related to earlier values, recent shocks, calendar effects and changing variance. Sequential prediction extends this idea to ordered data in which the objective may be forecasting, classification, anomaly detection or decision-making.

Forecasting is therefore not simply a regression problem with a date column. It requires preservation of temporal order, explicit treatment of information availability, appropriate handling of non-stationarity, and evaluation against realistic operational baselines. This chapter examines classical statistical models, state-space methods, recurrent neural networks and attention-based architectures.

Learning objectives

By the end of this chapter, learners should be able to:

1. Explain trend, seasonality, cycles, stationarity, autocorrelation and temporal dependence.
2. Build and interpret naïve, moving-average, exponential-smoothing, ARIMA and state-space forecasts.
3. Distinguish recursive, direct and multi-output forecasting strategies.
4. Apply recurrent and attention-based models to sequential prediction.
5. Design leakage-resistant validation and select suitable forecasting metrics.
6. Evaluate forecasting systems in operational, ethical and governance contexts.

1. Structure of temporal data

Let y_t denote an observation at time t. A useful conceptual decomposition is:

Here T_t is the trend, S_t is a systematic seasonal component, and R_t is the remainder or irregular component. This additive form is appropriate when seasonal fluctuations have roughly constant absolute size. A multiplicative decomposition is often more suitable when seasonal variation grows with the level:

The components are not always uniquely identifiable. A long-term cycle may be confused with trend, and a changing seasonal pattern may violate the assumption of fixed seasonality.

Important properties include:

• Trend: persistent long-term movement.
• Seasonality: a repeated pattern with a known period, such as 7 days or 12 months.
• Cycle: recurrent movement without a fixed or known period.
• Noise: unpredictable variation.
• Lag: the interval between observations being compared.
• Autocorrelation: correlation between a series and its lagged values.
• Stationarity: broadly, statistical behaviour that does not change over time. Weak stationarity requires constant mean and variance and autocovariance depending only on lag, not calendar time.

For a stationary series, autocovariance at lag k is:

and autocorrelation is:

where γ₀ is the variance. These quantities are meaningful when the relevant moments are stable and the observations are measured at consistent intervals.

A series may be transformed to improve stationarity. First differences are:

Seasonal differences with period m are:

Differencing removes persistent level movement but can also remove useful signal and amplify noise. It should be justified diagnostically rather than applied automatically.

2. Forecasting workflow

A professional workflow begins with the decision and forecast horizon. Predicting tomorrow’s demand for staffing is different from predicting annual capacity. The horizon determines the information available, the acceptable error and the useful model complexity.

A sound process is:

1. Define the target, time granularity, horizon and decision cost.
2. Audit timestamps, missing intervals, duplicates, outliers and revisions.
3. Establish naïve and seasonal-naïve baselines.
4. Explore plots, rolling statistics and autocorrelation.
5. Create time-ordered training, validation and test periods.
6. Fit transformations and features using training data only.
7. Compare candidate models using rolling-origin evaluation.
8. Produce prediction intervals where decisions require uncertainty.
9. Monitor errors, drift and data quality after deployment.

Random shuffling is generally invalid for forecasting because it allows future information to influence the past. A rolling-origin design repeatedly trains on an earlier interval and tests on a later interval.

3. Baselines and classical forecasting

Naïve and seasonal-naïve forecasts

The naïve forecast uses the most recent observation:

for horizon h. A seasonal-naïve forecast uses the observation from the corresponding previous season:

where m is the seasonal period. These baselines are difficult to outperform in highly persistent or strongly seasonal series and are essential for judging whether machine learning adds value.

Moving averages

A trailing moving average of window w is:

It smooths noise but lags behind rapid changes. It is useful for descriptive smoothing and simple level estimation, but it is not automatically an optimal forecast.

Exponential smoothing

Simple exponential smoothing updates an estimated level:

where 0 < α ≤ 1. A one-step forecast is ŷ_t+1|t = ℓ_t. Larger α responds more rapidly to new observations; smaller values produce smoother estimates. Initialisation must be specified, for example by setting the initial level to the first observation or estimating it from the data.

Holt’s method adds a trend. One common additive form is:

where 0 < β ≤ 1 and h is the forecast horizon. Damped-trend variants reduce unrealistic long-range extrapolation. Holt–Winters methods additionally estimate seasonality and may use additive or multiplicative seasonal terms.

AR, MA and ARIMA models

An autoregressive model of order p, AR(p), predicts from its own lags:

where c is an intercept, φ_i are coefficients, and ε_t is an error with mean approximately zero. Stationarity imposes restrictions on the autoregressive coefficients; arbitrary coefficients should not be assumed valid.

A moving-average model of order q, MA(q), models current values through past shocks:

ARMA(p,q) combines both. ARIMA(p,d,q) applies d differences before fitting an ARMA model. Seasonal ARIMA, written SARIMA, adds seasonal autoregressive, differencing and moving-average terms. Model identification commonly uses domain knowledge, autocorrelation and partial autocorrelation, but final selection should use time-aware validation rather than visual rules alone.

A small worked example

Suppose daily demand for the last three days is 100, 110 and 105 units. A three-day moving-average forecast is:

If the previous week’s corresponding demand was 112 units, the seasonal-naïve forecast is 112 units. Neither method is universally superior: the moving average dampens recent variation, whereas the seasonal-naïve method preserves weekly seasonality. Historical rolling-origin errors should determine the choice.

4. State-space models and uncertainty

State-space modelling separates an unobserved state from the measurement process. A general linear Gaussian form is:

Here x_t is the latent state, u_t an optional known input, F_t the state-transition matrix, H_t the observation matrix, and w_t, v_t process and observation errors. In the classical model, these errors are commonly assumed zero-mean Gaussian, mutually independent, and with specified covariance matrices.

The Kalman filter recursively performs prediction and updating. It first predicts the next state from the previous estimate, then corrects that prediction using the new observation. It is computationally efficient and naturally produces uncertainty estimates. Exponential smoothing models can also be expressed in state-space form, making likelihood-based estimation and prediction intervals possible.

State-space methods are valuable when data contain missing observations, changing latent levels, noisy sensors or time-varying components. Nonlinear or non-Gaussian systems may require extended, unscented or particle filters, each introducing additional approximations and computational cost.

5. Sequential machine learning

A supervised learning representation converts a series into input-output windows. With look-back length L, an input may be:

and the target may be y_t+1 or a vector of future values. Exogenous variables—weather, promotions, prices or calendar indicators—may be included only when their future availability is known or separately forecast.

A model can forecast one step recursively, forecast every horizon directly, or produce all horizons jointly.

Tree ensembles and gradient boosting often perform strongly when lag, rolling, calendar and external features are engineered carefully. They do not inherently understand order; the designer must encode temporal structure and prevent leakage in rolling features.

6. Recurrent neural networks

An RNN maintains a hidden state summarising previous inputs:

where x_t is the input, h_t the hidden state, W_x, W_h and W_y are learned parameters, and g is an output function appropriate to the task. Training uses backpropagation through time. Long sequences can cause vanishing or exploding gradients.

LSTM networks address this using gates. A simplified LSTM has input, forget and output gates:

Here σ is the sigmoid function, ⊙ denotes elementwise multiplication, and c_t is the cell state. GRUs use fewer gates and can be faster, sometimes with comparable performance.

Neural sequence models require scaling, adequate data, regularisation, careful window construction and a validation scheme reflecting deployment. Greater capacity does not guarantee better forecasts, particularly for short, noisy or structurally changing series.

7. Attention and transformer-based forecasting

Attention allows a model to weight relevant positions in a sequence rather than compressing all history into one recurrent state. Given query matrix Q, key matrix K and value matrix V, scaled dot-product attention is:

where d_k is the key dimension. The scaling reduces excessively large dot products. Multi-head attention computes several attention transformations and combines them, enabling different relationships across lags or variables.

Transformers require positional information because attention alone is not inherently ordered. Forecasting architectures may use causal masks so that a prediction cannot attend to future targets. Encoder-only models, encoder–decoder models and specialised temporal transformers differ in how they represent history, known future covariates and forecast horizons.

Attention can capture long-range relationships and parallelise training more effectively than recurrence, but it may be data-hungry and computationally expensive. An attention map should not automatically be interpreted as a causal explanation; high attention weight is not proof that a variable caused an outcome.

8. Evaluation, uncertainty and monitoring

Let actual values be y_i, forecasts ŷ_i, and let n be the number of evaluated cases. Mean absolute error is:

It has the same units as the target. Root mean squared error is:

RMSE penalises large errors more strongly. Mean absolute percentage error is:

MAPE is unsuitable when actual values are zero or near zero. Weighted or symmetric alternatives may be preferable. Metrics should reflect business loss: a missed safety threshold may be more serious than an overprediction of equal size.

Point forecasts are incomplete when decisions depend on risk. Prediction intervals should be evaluated for coverage and sharpness. A nominal 90% interval should contain the future observation approximately 90% of the time under stable conditions, although finite samples and drift can reduce coverage. Quantile forecasting estimates conditional quantiles using pinball loss. For quantile level τ:

where I is an indicator and q is the predicted quantile.

After deployment, monitor data distributions, missingness, residual patterns, forecast bias, interval coverage and performance by subgroup or operating regime. Retraining should be triggered by evidence, not merely by calendar habit.

9. Illustrative case: forecasting hospital emergency demand

Background. A regional hospital must forecast emergency arrivals by hour for the next 24 hours to plan clinicians and beds. Historical data contain hourly arrivals, day-of-week, holidays, weather and public-event indicators.

Problem. Demand is strongly intraday and weekly seasonal, occasionally affected by extreme weather, and contains missing sensor records. Underprediction creates safety and staffing risks; moderate overprediction is less costly.

Analysis. The team first verifies time zones and reconstructs missing intervals without fabricating target values. A seasonal-naïve baseline uses arrivals from the same hour one week earlier. Exploratory analysis reveals changing variance and holiday effects. Weather forecasts, rather than future observed weather, are used as covariates because deployment cannot access the future observations.

Approach. The team compares seasonal naïve, Holt–Winters, dynamic regression with ARIMA errors, gradient boosting using lagged and calendar features, and an LSTM. Rolling-origin validation uses several historical evaluation windows. Models are scored with MAE and a weighted loss that penalises underprediction more heavily. A calibrated prediction interval is supplied to the staffing system. The selected model is not the most complex model; it is the model with the best stable validation performance and acceptable latency.

Outcome. In this illustrative scenario, the model is deployed with a dashboard showing forecasts, intervals, missing-data warnings and recent errors. Clinicians retain decision authority. Forecasts are reviewed during unusual events, and performance is monitored separately for ordinary days, holidays and severe weather.

Learning points. Baselines expose whether complexity adds value; covariate availability is a deployment constraint; asymmetric costs affect evaluation; and human oversight is essential when forecasts influence healthcare capacity.

10. Limitations, governance and best practice

Forecasts cannot remove irreducible uncertainty. Structural breaks, pandemics, policy changes and sensor failures can invalidate historical relationships. Correlation in historical sequences does not establish causation. Models may also encode inequities if data quality or service access differs across populations.

Governance should address data minimisation, access control, audit logs, retention, reproducibility and documented model ownership. In high-impact settings, users need understandable uncertainty communication and escalation procedures. Personal or sensitive temporal data require lawful processing, appropriate security and careful consideration of re-identification risks.

Best practice includes:

• begin with a strong naïve baseline;
• preserve temporal order in preprocessing and validation;
• document forecast horizon, information cut-off and missing-data treatment;
• use only deployable future covariates;
• compare accuracy, calibration, latency and maintenance cost;
• inspect residuals and performance across regimes;
• version data, code, features and model parameters;
• monitor drift and define retraining and rollback criteria;
• communicate intervals and limitations to decision-makers.

Contemporary developments

Current forecasting systems increasingly combine probabilistic prediction, foundation models, global models trained across many related series, graph-based representations of spatial networks and hybrid statistical–neural methods. These approaches can transfer information across products, locations or sensors, but transfer may fail when domains differ. Efficient attention, patch-based representations and retrieval of similar historical episodes are active areas of research. Their adoption should be justified by measurable improvement, robust evaluation and governance readiness rather than novelty alone.

Chapter Summary

Temporal forecasting differs from ordinary prediction because order, information availability and dependence matter. Decomposition clarifies trend and seasonality, while differencing and autocorrelation support diagnostic analysis. Naïve methods provide essential baselines; moving averages and exponential smoothing offer transparent level, trend and seasonal models. ARIMA models represent dependence through lagged values and shocks, while state-space models represent latent states and provide recursive uncertainty estimates.

Supervised windowing enables tree-based and neural models, but feature construction must prevent leakage. RNNs, LSTMs and GRUs model sequential state; attention and transformers provide flexible long-range interaction, subject to positional, computational and data requirements. Evaluation should use time-aware validation, decision-relevant metrics and calibrated uncertainty. Successful deployment combines technical accuracy with monitoring, governance, human oversight and explicit recognition of structural change.

Key Terms

Time series: Ordered observations indexed by time.

Temporal dependence: Relationship between observations at different times.

Trend: Persistent long-term movement in a series.

Seasonality: Repeating pattern with a known period.

Stationarity: Stability of relevant statistical properties over time.

Autocorrelation: Correlation between a series and a lagged version of itself.

Lag: Time interval separating observations.

Differencing: Subtracting earlier observations to reduce non-stationarity.

Naïve forecast: Forecast based on the latest observation.

Exponential smoothing: Weighted updating method giving greater or lesser emphasis to recent data.

ARIMA: Autoregressive integrated moving-average model.

State-space model: Model separating latent state dynamics from observations.

Kalman filter: Recursive estimator for linear Gaussian state-space systems.

Autoregressive model: Model using previous values of the series as predictors.

Recursive forecasting: Predicting multiple steps by feeding forecasts back as inputs.

RNN: Neural network that maintains a hidden state across sequence positions.

LSTM: Gated recurrent network designed to retain information over long sequences.

Attention: Mechanism that weights relationships among sequence positions.

Prediction interval: Range intended to contain a future observation with stated probability.

Data leakage: Use of information unavailable at prediction time.

Review Questions

Short Answer Questions

1. What distinguishes seasonality from a general cycle?
2. Why is a seasonal-naïve forecast an important baseline?
3. State one limitation of MAPE.
4. What problem do LSTM gates address?
5. Why must future covariates be available at deployment time?

Long Answer Questions

1. Explain the decomposition of a time series and discuss when additive and multiplicative forms are appropriate.
2. Compare exponential smoothing, ARIMA and state-space methods in terms of assumptions, interpretability and uncertainty.
3. Explain how rolling-origin validation prevents misleading evaluation.
4. Describe the design of an LSTM forecasting system, including windowing, scaling and multi-step prediction.
5. Compare recurrent networks with attention-based architectures for long sequential data.

Analytical / Application Questions

1. A retailer has hourly demand with strong daily and weekly patterns. Design a baseline, validation scheme and metric set for a 24-hour forecast, justifying each decision.
2. A model has lower RMSE but systematically underpredicts peak hospital demand. How should model selection and governance respond?
3. A deployed forecast suddenly fails after a new regulation changes customer behaviour. Diagnose the likely causes and propose a monitoring, retraining and fallback plan.

Multiple Choice Questions with Answers

Question 1

Which forecast is defined by ŷ_t+h|t=y_t?

A. Naïve forecast

B. Seasonal-naïve forecast

C. Moving-average forecast

D. Direct multi-output forecast

Correct Answer: A. Naïve forecast

Question 2

What is the main purpose of differencing a non-stationary series?

A. To increase the number of observations

B. To reduce persistent level movement

C. To guarantee normally distributed errors

D. To remove all random noise

Correct Answer: B. To reduce persistent level movement

Question 3

For a three-period moving average of 8, 10 and 12, what is the forecast?

A. 9

B. 10

C. 11

D. 30

Correct Answer: B. 10

Question 4

Which metric has the same units as the target variable?

A. MAPE

B. MAE

C. A percentage error only

D. A correlation coefficient

Correct Answer: B. MAE

Question 5

What is a principal advantage of a state-space model?

A. It never requires assumptions

B. It can represent latent states and recursive uncertainty

C. It eliminates structural breaks

D. It always outperforms neural networks

Correct Answer: B. It can represent latent states and recursive uncertainty

Question 6

Why is random shuffling usually inappropriate for time-series validation?

A. It reduces the sample size

B. It can allow future information to influence training

C. It prevents calculation of MAE

D. It forces seasonal differencing

Correct Answer: B. It can allow future information to influence training

Question 7

Which mechanism helps an LSTM retain information over longer sequences?

A. Seasonal decomposition only

B. Gated cell-state updates

C. Random shuffling

D. MAPE optimisation

Correct Answer: B. Gated cell-state updates

Question 8

In scaled dot-product attention, why is QKᵀ divided by √d_k?

A. To remove all positional information

B. To control the scale of dot products

C. To convert forecasts into percentages

D. To guarantee causal interpretation

Correct Answer: B. To control the scale of dot products

Question 9

Which situation makes MAPE especially problematic?

A. All actual values are large and positive

B. The forecast horizon is one step

C. Actual values are zero or near zero

D. The series has weekly seasonality

Correct Answer: C. Actual values are zero or near zero

Question 10

A recursive multi-step forecast primarily faces which risk?

A. It cannot use historical observations

B. Errors can accumulate across forecast steps

C. It always requires a transformer

D. It cannot produce a one-step forecast

Correct Answer: B. Errors can accumulate across forecast steps