Time-Series and Sequential Machine Learning

Introduction and Context

A time series is an ordered sequence of observations indexed by time: demand recorded hourly, electricity consumption measured every 15 minutes, transactions observed daily, or patient vital signs sampled continuously. Unlike ordinary independent and identically distributed data, time-series observations are usually dependent. Recent values may contain information about future values, and the data-generating process may change because of seasonality, policy, competition, weather, technological adoption, or abnormal events.

Time-series machine learning therefore differs from ordinary supervised learning in three important ways. First, the order of observations must be preserved. Second, predictors must be constructed using only information available at the forecast origin. Third, evaluation must reproduce the way the model will actually be used. Randomly shuffling observations, computing features from the complete dataset, or selecting a model using future information can produce apparently excellent but operationally invalid forecasts.

Forecasting is not limited to predicting a single future value. A system may produce point forecasts, prediction intervals, quantiles, multiple future steps, event probabilities, or decisions such as inventory replenishment. The appropriate method depends on the forecast horizon, data frequency, scale, business loss, structural stability, and availability of external variables.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Explain trend, seasonality, cycles, autocorrelation, stationarity, and temporal dependence.
2. Construct leakage-safe lag, rolling, calendar, and exogenous features.
3. Distinguish statistical forecasting, machine-learning forecasting, and deep sequential models.
4. Design chronological validation and rolling-origin evaluation.
5. Select and interpret suitable forecasting metrics and prediction intervals.
6. Diagnose model failure under drift, shocks, sparse data, and changing seasonality.
7. Develop a responsible forecasting workflow for professional deployment.

1. Structure of Temporal Data

Let y_t denote the observed value at time t. A useful conceptual decomposition is

where T_t is trend, S_t is seasonal structure, C_t represents longer-term cyclical movement, and R_t is the irregular or remainder component. This additive decomposition is appropriate when seasonal fluctuations have approximately constant absolute magnitude. A multiplicative decomposition is more suitable when seasonal amplitude grows with the level:

The multiplicative form requires positive or otherwise appropriately transformed observations. Taking logarithms converts multiplication into addition, provided values are positive:

The components are descriptive rather than universally identifiable. A gradual rise could be a trend, a portion of a long cycle, or the result of changing explanatory variables. Decomposition should therefore support, not replace, domain analysis.

Important temporal concepts

• Trend: persistent long-term movement in the level or slope.
• Seasonality: systematic and repeating variation linked to a known calendar or operational period.
• Cycle: recurring movement whose period is not necessarily fixed, such as an economic cycle.
• Autocorrelation: dependence between observations separated by a lag.
• Lag: a previous observation, such as y_t-1.
• Forecast horizon: the number of future periods being predicted.
• Forecast origin: the latest time at which information is available when a forecast is produced.
• Exogenous variable: a predictor external to the target series, such as temperature or price.
• Stationarity: a stability condition in which relevant statistical properties do not change over time.
• Concept drift: change in the relationship between predictors and the target.

Autocorrelation and dependence

The lag-k autocorrelation is commonly defined as

where Cov is covariance and Var is variance. This expression assumes a weakly stationary process so that means, variances, and covariances do not depend on calendar time. In practice, the sample autocorrelation is estimated from finite data and may be distorted by trend or seasonality. A large autocorrelation at lag 1 may indicate persistence, but it does not by itself establish causation or guarantee forecastability.

A stationary series has a constant mean, constant finite variance, and autocovariance depending only on the lag rather than the date. Many economic and operational series are not stationary in their raw form. Differencing can remove some nonstationarity:

Seasonal differencing at period m is

Differencing may improve stability, but excessive differencing removes useful signal and increases noise. Stationarity tests, plots, domain knowledge, and residual diagnostics should be considered together; no test should be treated as a mechanical decision rule.

2. Statistical Forecasting Models

Statistical models explicitly represent temporal dependence and often perform strongly with short or moderate data histories.

Baselines

A naïve forecast sets the next forecast equal to the latest observation:

for horizon h. A seasonal naïve forecast uses the most recent value from the corresponding season:

where m is the seasonal period and k is chosen so that the referenced observation is available. Baselines are essential: a complex model that cannot outperform a relevant naïve benchmark may not justify its cost.

Moving averages and exponential smoothing

A simple moving average over window w is

It smooths noise but responds slowly to sudden changes. Simple exponential smoothing updates a level estimate recursively:

where 0 < α ≤ 1. A larger α gives more weight to recent observations. Holt’s method adds a trend component, and Holt–Winters methods add seasonality. Additive seasonality is suitable for approximately constant seasonal magnitude; multiplicative seasonality is suitable when magnitude scales with the level.

Autoregressive models

An autoregressive model of order p is

where c is an intercept, φ_i are coefficients, and ε_t is an error term with conditional mean zero. The model assumes that the selected lag structure captures relevant dependence and that the remaining errors are sufficiently well behaved. Stability conditions depend on the roots of the characteristic polynomial; informally, coefficients must not imply explosive dynamics.

An ARIMA model combines autoregression, differencing, and moving-average errors. It is often written as ARIMA(p,d,q), where p is the autoregressive order, d the number of ordinary differences, and q the moving-average error order. Seasonal ARIMA extends this structure with seasonal terms. These models are interpretable and effective for univariate series, but model identification and maintenance become difficult with many related series or nonlinear effects.

3. Leakage-Safe Temporal Features

Feature engineering is often the strongest determinant of machine-learning forecasting performance. Every feature must be computable at the forecast origin. For a one-step forecast at time t, a feature may use y_t and earlier information only if y_t is already known when predicting y_t+1. For an h-step-ahead forecast, it must not use any value unavailable at the corresponding forecast time.

Lag features

Typical features include y_t−1, y_t−7, and y_t−12 for daily, weekly, and monthly structures, provided those lags are meaningful for the data frequency. Lags should reflect operational mechanisms rather than being added indiscriminately.

Rolling features

A past-only rolling mean with window w is

The current target y_t is excluded. In software, this normally means calculating a rolling statistic and then shifting it by one period before using it to predict the current observation. Rolling standard deviation can represent recent volatility, while rolling minimum and maximum can capture local range.

Calendar and event features

Calendar predictors include hour, day of week, month, holiday indicator, pay-cycle position, and school-term status. Cyclic variables should not always be represented as ordinary integers because December and January are close in time. For a periodic variable [[MATH}q_t]] with period [[MATH}P]], a common encoding is

The two variables preserve circular geometry. Event features should represent only events known at forecasting time. A planned promotion may be known; an unplanned stockout recorded after the fact is not.

Exogenous variables and transformations

Weather forecasts, prices, marketing plans, capacity, and macroeconomic indicators can improve predictions. The model must distinguish observed historical values from future values. If future temperature is unknown, using actual future temperature during evaluation creates optimistic results; the deployed system must use weather forecasts or scenario estimates instead.

Scaling, imputation, target encoding, feature selection, and dimensionality reduction must be fitted within each training window. Fitting them on the complete dataset allows future distributional information to enter the past and constitutes leakage.

4. Machine-Learning Forecasting Strategies

Direct, recursive, and multi-output forecasting

For a forecast horizon [[MATH}H]], a recursive strategy trains one model for one-step prediction and feeds predictions back as future inputs. It is computationally simple but errors can accumulate. A direct strategy trains a separate model for each horizon:

It avoids feeding predictions back into the model but requires multiple models. A multi-output model predicts all horizons jointly. It can learn cross-horizon structure but may be more difficult to train and calibrate.

Tree ensembles such as random forests and gradient-boosted trees are effective when lag, rolling, calendar, and exogenous features are informative. They handle nonlinear interactions and mixed feature types, but ordinary trees do not inherently understand time. Temporal structure must be represented explicitly, and extrapolating a long smooth trend can be difficult.

Regularized linear models provide useful transparency. Ridge regression estimates coefficients by minimizing squared error plus an L2 penalty:

where [[MATH}λ ≥ 0]] controls shrinkage. The penalty can reduce variance when lag and seasonal features are correlated. The intercept may or may not be penalized according to implementation convention; the convention should be documented.

5. Deep Sequential Models

Deep learning is useful when there are large volumes of data, many related series, complex nonlinear relationships, or rich covariates. It is not automatically superior for small, stable univariate datasets.

Recurrent models

A recurrent neural network maintains a hidden state:

where [[MATH}x_t]] is the input, [[MATH}h_t]] is the hidden state, [[MATH}W]] are learned weight matrices, [[MATH}b_h]] is a bias, and [[MATH}σ]] is an activation function. Vanilla RNNs can suffer from vanishing or exploding gradients over long sequences. LSTM and GRU architectures introduce gating mechanisms to preserve or discard information more effectively.

Convolutional and attention-based models

Temporal convolutional networks use causal convolutions, ensuring that output at time [[MATH}t]] does not use future inputs. Dilated convolutions expand the receptive field efficiently. Transformer-style models use attention to assign data-dependent weights to elements of a sequence. They can model long-range relationships, but their computational and data requirements, sensitivity to positional representation, and risk of overfitting must be considered.

Deep forecasting systems may use quantile loss rather than squared error. For quantile level [[MATH}τ ∈ (0,1)]], the pinball loss is

where [[MATH}I(·)]] is an indicator and [[MATH}q]] is the predicted quantile. Minimizing expected pinball loss estimates the conditional [[MATH}τ]]-quantile under standard regularity conditions. Multiple quantiles should be checked for crossing, since a higher quantile should not be below a lower quantile.

6. Temporal Validation and Evaluation

Random train-test splits are generally invalid for forecasting because they permit training on observations later than those used for testing. A chronological split trains on an early interval and tests on a later interval. A stronger design uses rolling-origin evaluation:

1. Select an initial training window.
2. Fit preprocessing and model only on that window.
3. Forecast the next validation period.
4. Move the origin forward.
5. Repeat and aggregate errors.

An expanding window retains all historical observations; a sliding window retains only the most recent observations. Expanding windows are useful when old data remain relevant. Sliding windows can respond better to drift but may increase variance and discard valuable long-run information.

For actual [[MATH}y_i]] and forecast [[MATH}ŷ_i]], mean absolute error is

and root mean squared error is

MAE is in the target’s units and weights errors linearly. RMSE is also in target units but penalizes large errors more strongly. Mean absolute percentage error is

and is undefined when [[MATH}y_i = 0]] and unstable near zero. Weighted absolute percentage metrics or scale-free alternatives may be preferable for intermittent demand.

For an interval [[MATH}[L_i,U_i]]] with nominal coverage [[MATH}1−α]], empirical coverage is

Coverage should be evaluated together with interval width. A very wide interval can achieve high coverage but provide little decision value. Forecast evaluation should be segmented by horizon, geography, product, regime, and event status rather than reduced to one average score.

7. Forecast Uncertainty and Changing Conditions

A point forecast is incomplete when decisions depend on risk. Uncertainty arises from irreducible randomness, parameter estimation, model choice, future covariates, and structural change. Prediction intervals can be generated from statistical model assumptions, bootstrap procedures, ensembles, conformal methods, or quantile models.

Conformal forecasting constructs prediction sets using calibration residuals or nonconformity scores. Its practical appeal is limited reliance on a parametric error distribution, but validity assumptions are most naturally expressed under exchangeability or related stability conditions. Time dependence and drift can weaken coverage; adaptations such as rolling calibration or weighted conformity scores must be evaluated empirically.

Changing conditions require monitoring. Useful signals include residual mean, residual variance, feature distributions, forecast error by horizon, interval coverage, and business outcomes. A residual sequence should not display systematic autocorrelation if the model has captured the major predictable structure. The Ljung–Box test can assess whether a group of autocorrelations is jointly consistent with white noise, but statistical significance is affected by sample size and repeated monitoring.

Possible responses to drift include shorter training windows, time-varying parameters, retraining schedules, change-point detection, regime-specific models, external covariates, and human override procedures. Retraining should not be triggered solely by a generic data-drift alert; the relevant question is whether predictive performance or decision quality has deteriorated.

8. Worked Example: Daily Retail Demand

Suppose a retailer records daily demand for 28 days. The business predicts demand for the next day and replenishes inventory each evening. Assume the latest observed demand is 120 units, demand seven days earlier was 110 units, and the previous six daily observations are 118, 125, 121, 117, 119, and 123 units.

A one-day naïve forecast is 120 units. A seasonal naïve forecast using a seven-day period is 110 units. A past-only six-day rolling mean is

Suppose a fitted model produces 122 units. If the actual next-day demand is 130 units, the absolute errors are 10, 20, 9, and 8 units for the naïve, seasonal naïve, rolling-mean, and fitted model respectively. The fitted model performs best for this one observation, but no conclusion should be based on one day. Rolling-origin evaluation might reveal that the model performs poorly during promotions or weekends.

A leakage-safe feature row for forecasting the next day may include the current demand of 120, demand seven days earlier of 110, the past-only rolling mean of 120.5, day-of-week indicators, and a promotion flag known in advance. It must not include next-day actual demand, the next day’s realized weather, or a rolling mean that accidentally includes the target.

If inventory cost is asymmetric, MAE alone may be inadequate. Underage may cause lost sales while overage causes holding or spoilage cost. The retailer should evaluate forecasts using a decision-oriented loss, service-level targets, and stockout rates, while preserving statistical diagnostics.

9. Applied Case: Illustrative Electricity-Load Forecasting

Background

Consider an illustrative regional electricity operator forecasting hourly load 24 hours ahead. Load exhibits daily and weekly seasonality, temperature sensitivity, holidays, and occasional extreme peaks.

Problem

An initial gradient-boosting model achieved low random-split RMSE but failed during summer peaks. Investigation showed that the data pipeline randomly mixed hours and used realized future temperature, which would not be available at prediction time.

Analysis

The team created chronological rolling-origin backtests. Features included lagged load at 1, 24, 48, and 168 hours; past-only rolling averages; hour-of-day and day-of-week cyclic encodings; holiday indicators; and forecast temperature rather than realized temperature. A seasonal naïve benchmark based on load 168 hours earlier was retained. Models were assessed separately for 1–6, 7–12, and 13–24 hour horizons.

Approach and solution

A direct multi-horizon boosted-tree system was compared with exponential smoothing and a recurrent neural network. Preprocessing was fitted independently in each training window. Prediction intervals were generated using rolling residual calibration. The operator selected a model not only by average RMSE but also by peak-load error, interval coverage, and operational cost.

Outcome

The corrected evaluation produced a less impressive but more credible performance estimate. The model reduced peak error relative to the seasonal naïve benchmark while maintaining acceptable coverage. A monitoring dashboard tracked horizon-specific error, temperature forecast error, residual autocorrelation, and coverage. During an unusual heatwave, the system flagged a regime change and invoked a wider-interval policy and expert review.

Learning points

The case demonstrates that leakage control can reduce apparent accuracy while improving real-world validity. Exogenous forecasts must match deployment information. Benchmarks expose whether complexity is useful. Evaluation should reflect high-impact periods, not only average conditions. Finally, uncertainty communication and override governance are part of the forecasting system, not optional additions.

10. Professional, Ethical, and Governance Considerations

Forecasts influence staffing, credit, healthcare capacity, energy dispatch, and public services. A forecast can therefore redistribute risk even when it is statistically accurate on average. Practitioners should document the target definition, forecast origin, data cut-off, horizon, missing-data treatment, model version, training window, and evaluation protocol.

Sensitive variables may create discriminatory effects in sequential decisions. For example, a demand or risk forecast used to allocate service capacity may systematically under-serve a population because historical data reflect unequal access. Fairness analysis should examine errors, coverage, and downstream outcomes across relevant groups, while respecting privacy and data-protection law.

Data provenance, access controls, retention limits, and audit logs are necessary when time series contain personal, location, medical, or financial information. Human oversight is particularly important during unprecedented events, when historical relationships may no longer apply. Forecasts should be presented with uncertainty, assumptions, and known failure modes rather than as unquestionable predictions.

11. Best-Practice Workflow

1. Define the decision, target, horizon, frequency, and forecast origin.
2. Establish naïve and seasonal-naïve benchmarks.
3. Inspect missingness, outliers, aggregation, trend, seasonality, and regime changes.
4. Build leakage-safe lag, rolling, calendar, and exogenous features.
5. Choose a model family consistent with data volume, complexity, and interpretability needs.
6. Use chronological or rolling-origin validation.
7. Fit all preprocessing inside each training fold.
8. Evaluate point accuracy, uncertainty, horizon effects, subgroup performance, and business loss.
9. Diagnose residual dependence and drift.
10. Deploy monitoring, retraining rules, versioning, documentation, and escalation procedures.

Contemporary Developments

Current forecasting research increasingly addresses global models trained across many related series, probabilistic forecasting, foundation models for sequences, hierarchical reconciliation, causal forecasting, and hybrid systems that combine statistical structure with neural representation learning. These developments are promising but require careful evidence. Large pretrained sequence models may transfer useful representations, yet transfer can fail when the target domain, frequency, measurement process, or decision loss differs from pretraining data. Research-grade evaluation should include strong baselines, realistic information sets, multiple forecast origins, uncertainty assessment, and reproducible data pipelines.

Chapter Summary

Time-series and sequential machine learning model ordered observations in which dependence, seasonality, trend, and changing conditions matter. Decomposition provides a useful vocabulary, while autocorrelation and stationarity clarify temporal structure. Statistical approaches such as naïve benchmarks, exponential smoothing, autoregression, and ARIMA remain valuable, especially with limited data. Machine-learning methods can exploit nonlinear lag, calendar, and exogenous relationships, but they require explicit temporal features. Deep recurrent, convolutional, and attention-based models can represent complex sequences when sufficient data and engineering resources are available.

The central operational principle is leakage prevention: every feature, transformation, and external variable must reflect information available at the forecast origin. Chronological and rolling-origin evaluation provide more credible evidence than random splitting. MAE, RMSE, percentage metrics, quantile loss, interval coverage, and decision-oriented costs measure different aspects of performance. Because relationships change, monitoring, drift adaptation, uncertainty communication, governance, and human review are essential components of a professional forecasting system.

Key Terms

Time series: Ordered observations indexed by time.

Forecast origin: The latest time at which information is available for producing a forecast.

Forecast horizon: Number of future periods being predicted.

Trend: Persistent long-term movement in a series.

Seasonality: Repeating variation associated with a known period.

Cycle: Recurrent movement with a potentially changing or non-fixed period.

Autocorrelation: Dependence between observations separated by a time lag.

Stationarity: Stability of key statistical properties over time.

Lag feature: A previous observation used as a predictor.

Rolling feature: A statistic computed over a past moving window.

Exogenous variable: An external predictor used to explain or forecast the target.

Data leakage: Use of information unavailable at prediction time.

Recursive forecasting: Iterative multi-step forecasting using previous predictions as inputs.

Direct forecasting: Separate forecasting model for each future horizon.

Exponential smoothing: Recursive method assigning greater weight to recent observations.

ARIMA: Autoregressive integrated moving-average model.

Concept drift: Change in the relationship between inputs and target over time.

Prediction interval: Range intended to contain a future observation with specified coverage.

Pinball loss: Asymmetric loss used to estimate conditional quantiles.

Rolling-origin evaluation: Repeated chronological training and testing at advancing forecast origins.

Review Questions

Short Answer Questions

1. Distinguish trend, seasonality, and cycle in a time series.
2. Why is random shuffling generally inappropriate for forecasting evaluation?
3. Define a leakage-safe rolling mean for predicting y_t.
4. State one difference between recursive and direct multi-step forecasting.
5. Why can MAPE be problematic when actual values are zero or close to zero?

Long Answer Questions

1. Explain the additive and multiplicative decomposition of a time series, including when each is appropriate.
2. Compare exponential smoothing, ARIMA, tree-based machine learning, and deep sequential models.
3. Design a rolling-origin evaluation procedure for a 24-hour-ahead load-forecasting problem.
4. Discuss how trend, seasonality, exogenous variables, and concept drift affect feature engineering.
5. Explain why uncertainty intervals and governance are necessary in operational forecasting.

Analytical / Application Questions

1. A retailer uses a seven-day seasonal lag and a rolling mean that includes the current day’s target. Identify the leakage and specify how the feature pipeline should be corrected.
2. A model has lower average RMSE than a benchmark but much worse error during demand peaks. Recommend an evaluation and model-selection framework for this business.
3. Design a forecasting solution for a series with daily seasonality, intermittent zeros, known future promotions, and frequent regime changes. Justify the target transformation, features, validation scheme, metrics, and monitoring plan.

Multiple Choice Questions with Answers

Question 1

Which feature is leakage-safe when predicting y_t?

A. A rolling mean including y_t

B. The realized value of y_t+1

C. A rolling mean of [[MATH:y_{t−1}], y_{t−2}, …]

D. The future realized temperature at time t+1

Correct Answer: C. A rolling mean of [[MATH:y_{t−1}], y_{t−2}, …]

Question 2

What is the main purpose of a seasonal-naïve benchmark?

A. To guarantee the best forecast

B. To provide a relevant simple comparison using the previous season

C. To eliminate all seasonal variation

D. To estimate neural-network weights

Correct Answer: B. To provide a relevant simple comparison using the previous season

Question 3

In the expression ∆y_t = y_t − y_t−1, what operation is performed?

A. Seasonal scaling

B. First-order differencing

C. Exponential smoothing

D. Quantile estimation

Correct Answer: B. First-order differencing

Question 4

Which metric is expressed in the same units as the target and penalizes large errors more strongly than MAE?

A. RMSE

B. MAPE

C. Coverage

D. Pinball-loss level

Correct Answer: A. RMSE

Question 5

Which forecasting strategy trains a separate model for each future horizon?

A. Recursive

B. Direct

C. Seasonal naïve

D. Moving average

Correct Answer: B. Direct

Question 6

For a periodic variable with period P, why are sine and cosine features often used together?

A. They encode circular proximity

B. They remove all missing values

C. They guarantee stationarity

D. They convert forecasts into intervals

Correct Answer: A. They encode circular proximity

Question 7

What does a sliding training window do?

A. Uses only the most recent fixed-length observations

B. Uses every observation from the beginning of time

C. Randomly selects training observations

D. Removes the forecast horizon

Correct Answer: A. Uses only the most recent fixed-length observations

Question 8

A prediction interval has high empirical coverage but is extremely wide. What does this most directly indicate?

A. Perfect calibration and maximum usefulness

B. High coverage with potentially poor sharpness

C. Certain data leakage

D. Zero forecast uncertainty

Correct Answer: B. High coverage with potentially poor sharpness

Question 9

Which condition is central to a standard weakly stationary series?

A. Its mean and variance change every period

B. Its autocovariance depends on calendar date rather than lag

C. Its mean and variance are stable and dependence is lag-based

D. It contains no random component

Correct Answer: C. Its mean and variance are stable and dependence is lag-based

Question 10

Why must realized future weather generally not be used in deployed demand forecasting?

A. Weather is never predictive

B. It is unavailable at the forecast origin

C. It always creates seasonality

D. It prevents calculation of MAE

Correct Answer: B. It is unavailable at the forecast origin