Machine Learning for Time Series

Introduction

Machine learning (ML) has become an important complement to classical time-series methods. Its value is greatest when forecasting depends on many predictors: lagged demand, promotions, prices, weather, holidays, inventory, web traffic, and product or customer attributes. Unlike ordinary cross-sectional prediction, time-series ML must respect temporal order. A model that uses information unavailable at the forecast origin can appear highly accurate in testing and fail in production.

The central engineering task is therefore not simply selecting an algorithm. It is to construct a forecasting dataset in which every feature is computable at the time a forecast would actually be issued, validate it through realistic temporal splits, and select a model whose errors and explanations support the business decision.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Convert a time series into a supervised-learning dataset using lag, rolling, calendar, event, and external features.
2. Identify and prevent temporal leakage.
3. Compare regularized regression, tree ensembles, random forests, and gradient boosting.
4. Design rolling-origin validation and evaluate forecasts with suitable metrics.
5. Interpret global and local models using feature importance and explainability methods.
6. Decide between global and local forecasting architectures.
7. Combine statistical forecasts and ML models in hybrid systems.
8. Evaluate practical, ethical, and governance risks in ML forecasting.

1. From a Time Series to a Supervised-Learning Problem

Let y_t denote the observed value at time t. A one-step-ahead forecasting problem seeks an estimate of y_t+1 based only on information available at time t. More generally, for horizon h, the target is y_t+h.

A feature vector may contain past observations, known future information, and contemporaneous variables that are available before the forecast is made:

Here ŷ_t+h is the forecast, f_h is a learned function for horizon h, X_t is the information set available at the forecast origin, and ε_t+h is the forecast error. The relationship is predictive rather than necessarily causal.

For a series with observations at times 1,2,…,T, a lagged row might be written as:

where the rolling statistic must be calculated using observations no later than t. The target could be y_t+1. Rows near the beginning of the series are lost because their required lags do not exist.

The forecasting horizon determines feature eligibility. A promotion known six weeks in advance can be used for a six-week forecast; an observed weather value cannot be used unless a weather forecast was available at the forecast origin. This distinction separates known-future variables from merely future observations.

2. Feature Engineering for Temporal Data

2.1 Lag features

A lag feature is a past value:

where k is a positive integer measured in periods. For hourly electricity demand, useful lags may include 1, 2, 24, 48, and 168 hours. For weekly sales, lags of 1, 4, 13, and 52 weeks may represent short-term persistence, quarterly patterns, and annual seasonality.

Lag selection should reflect the data-generating process, operating cycle, and forecast horizon. Excessive lags increase dimensionality and can produce unstable estimates, especially when many lags are strongly correlated. Lags should also be aligned with the target. For a direct forecast of y_t+7, using y_t+6 may be valid only if that value would be known when the seven-period forecast is issued.

Differences can represent change rather than level:

Differences are useful for trend-dominated series, but they can discard level information. A model may therefore use both levels and differences, subject to regularization and validation.

2.2 Rolling and expanding features

Rolling features summarize recent history. A trailing mean of window length w is:

The index begins at t−1, not t, when predicting the next observation. Excluding y_t is often preferable when the feature is intended to represent information available before the target period and when the forecast is generated before period t closes.

Other useful statistics include trailing median, minimum, maximum, standard deviation, exponentially weighted mean, and rolling quantiles. An exponentially weighted mean gives greater weight to recent observations. One common recursion is:

where α controls responsiveness. Rolling standard deviation can indicate changing volatility, but it should be interpreted carefully when the window is short.

Expanding features use all historical observations available before the forecast origin, for example a historical mean or cumulative count. They can be useful for stable long-run behavior but may adapt slowly after structural change.

2.3 Calendar features

Calendar features encode recurring timing effects: hour, day of week, week of year, month, quarter, public holiday, month-end, financial year, and school term. Some are categorical; others can be represented cyclically. For a periodic variable with period P and position c, use:

The pair preserves the circular relationship between the final and first positions. For example, December and January are close in annual phase, whereas integer coding would make them appear far apart.

Calendar variables are not automatically causal. A day-of-week effect may proxy operating schedules, customer behavior, or staffing. Their predictive usefulness should be assessed under temporal validation.

2.4 Event and intervention features

Events include promotions, product launches, strikes, outages, sporting events, policy changes, and marketing campaigns. Useful representations include:

• binary indicators for whether an event is active;
• event type or category;
• days until and since an event;
• event intensity, budget, or discount depth;
• interaction between an event and product, location, or weekday;
• lead and lag indicators for anticipation and persistence.

Event features require a data-availability audit. A campaign label entered after the event is not valid historical information unless the same label would have been available at forecast time. Missing event records can cause the model to attribute effects to unrelated variables.

2.5 External variables

External regressors may include weather, macroeconomic indicators, competitor prices, exchange rates, traffic, or epidemiological measures. Each must be classified as one of three types:

External variables can improve accuracy but introduce dependency risk. A forecast system should record the data vintage—the version available at the forecast origin—so that backtests do not use revised historical values.

2.6 Missingness, transformations, and scale

Missing values should be interpreted, not merely imputed. Missing demand may mean a closure, a reporting failure, or true zero sales. A missingness indicator may be informative, but it can also encode operational artifacts that will not persist.

For positive and highly skewed targets, a log transformation may stabilize variance:

where c is a non-negative offset used when zeros are possible. Predictions must be transformed back carefully; simply exponentiating a prediction of log demand can underestimate the arithmetic mean when residual variance is material. Transformations must be chosen using validation and must respect the business meaning of the target.

Scaling is important for regularized regression but usually not required for decision trees. Scaling parameters must be fitted on training data only.

3. Algorithms for Time-Series ML

3.1 Regularized regression

A linear model provides a transparent baseline:

where x_tj is feature j, β_j is its coefficient, and p is the number of features. With correlated lags and many calendar indicators, ordinary least squares may be unstable. Ridge regression minimizes:

Lasso instead penalizes ∑|β_j| and can set coefficients exactly to zero. Elastic net combines both penalties. The parameter λ ≥ 0 controls shrinkage and must be selected using time-aware validation. Coefficients are conditional associations; with correlated features, their signs and magnitudes may be unstable.

Regression is valuable when effects are approximately additive, extrapolation is required, and explanation is important. It may underperform when thresholds and interactions dominate.

3.2 Decision trees and random forests

A decision tree recursively partitions feature space into regions and predicts a constant or simple statistic in each leaf. Trees capture nonlinear thresholds and interactions without requiring explicit transformations.

A random forest averages many trees trained on bootstrap samples and randomized feature subsets. Averaging reduces variance, although forests can be poor at extrapolating trends beyond the training range. They may also be computationally heavy with thousands of highly redundant lag features.

Random forests should not be treated as automatically time-aware. The algorithm can be applied to lagged data, but temporal ordering must still govern data splitting and feature construction.

3.3 Gradient boosting

Gradient boosting builds an additive sequence of weak learners. At iteration m, a simplified squared-error update is:

where h_m is a fitted tree, η is the learning rate, and F_m is the ensemble after m stages. Small learning rates with sufficient trees often improve generalization, but depth, number of trees, subsampling, and regularization must be validated.

Boosting can model nonlinearities and interactions effectively and is often a strong tabular forecasting method. It can nevertheless overfit, especially when temporal dependence reduces the effective sample size. Early stopping must use a validation period that follows the training period.

3.4 Direct, recursive, and multi-output strategies

For multiple horizons, a recursive strategy trains a one-step model and feeds predictions back as inputs. Errors can accumulate because future lag features become predicted rather than observed.

A direct strategy trains a separate model for each horizon h, estimating f_h(X_t). It avoids recursive error propagation but requires more models and may produce inconsistent forecasts across horizons. A multi-output model predicts several horizons jointly and can share information, but its assumptions and implementation must be examined carefully.

4. Leakage-Safe Validation and Evaluation

Random train-test splitting is generally inappropriate for forecasting because it allows future regimes to influence training and destroys the deployment sequence. A basic chronological split trains on earlier observations and tests on later observations. Stronger evaluation uses rolling-origin or expanding-window validation:

1. Train on an initial historical window.
2. Forecast the next validation block.
3. Move the forecast origin forward.
4. Refit or update the model.
5. Aggregate errors across origins.

Every operation—imputation, scaling, feature selection, hyperparameter selection, and target transformation—must be fitted within each training fold. A gap may be inserted between training and validation when publication delays, label latency, or operational lead times create an information boundary.

For actual value y_t and forecast ŷ_t, common metrics include:

MAE has the target's units and weights errors linearly. RMSE also has the target's units but gives greater influence to large errors. Percentage metrics can be unstable near zero. WAPE, for example, is:

provided the denominator is non-zero. Metrics should reflect the cost of over- and under-forecasting. A business may need weighted errors, service-level measures, or asymmetric loss rather than a single aggregate score.

Always compare ML models with meaningful baselines: last value, seasonal naive forecast, moving average, and a classical model such as exponential smoothing or ARIMA. Improvement should be measured across origins, segments, and regimes—not only by one average.

5. Feature Importance and Explainability

Feature importance answers different questions, so no single method is universally sufficient.

• Permutation importance measures the increase in validation error after disrupting a feature. It is model-agnostic but can understate the importance of correlated features.
• Gain or split importance in tree models summarizes training reductions in loss or split usage. It may favor high-cardinality or frequently selected variables.
• Partial dependence describes average predictions as a feature varies, holding other features according to a specified averaging procedure. Correlated predictors can make its interpretation unrealistic.
• Accumulated local effects are often more reliable under correlated features because they use local changes.
• SHAP-style additive explanations decompose a prediction into a baseline plus feature contributions. The result is an explanation of model behavior, not proof of causality.

Local explanations should be inspected for individual forecasts, especially large errors and high-impact decisions. Explanations must include the feature vintage and the model version. A forecast attributed to “temperature” may actually reflect a data pipeline change or a correlated seasonal variable.

Interpretability has two audiences: technical users need diagnostic detail, while managers need a faithful explanation of drivers, uncertainty, and actions. Explanations should never conceal that the model is associational.

6. Global versus Local Models

A local model is trained separately for each series, such as one model per product-store combination. A global model is trained across many related series, using identifiers, static attributes, and time-varying features. A global model may learn common seasonal or promotional patterns and borrow strength for sparse series.

A middle path uses clusters or hierarchical models: series are grouped by behavior, and models are shared within groups. Model choice should be evaluated empirically, with performance reported by series size, volatility, geography, and business importance.

7. Hybrid Statistical–ML Forecasting

Statistical models often represent level, trend, autocorrelation, and seasonality efficiently. ML models are strong at nonlinear effects, interactions, events, and rich external data. Hybrid systems combine these strengths.

A simple additive hybrid is:

where the ML model predicts a residual or adjustment. To train a residual model without leakage, statistical forecasts must be generated as rolling out-of-sample forecasts, not fitted values from a model trained on the entire history.

An alternative is forecast blending:

Weights can be fixed, optimized on validation data, or varied by horizon and segment. They must be estimated without using the final test period. Blending can improve robustness when models make different errors, but it adds operational complexity.

8. Illustrative Case: Retail Grocery Demand

Background and problem

A grocery chain forecasts daily demand for 2,000 store-product combinations. Forecasts drive replenishment, where under-forecasting causes stockouts and over-forecasting creates waste. The chain has sales history, promotions, prices, holidays, store attributes, weather forecasts, and product categories.

Analysis

A data audit finds that sales are recorded after midnight, promotions are scheduled in advance, and historical weather observations are available retrospectively. Therefore, the model may use scheduled promotions and weather forecasts, but not realized future weather. A seasonal-naive forecast using demand seven days earlier is established as the baseline.

Features include lags of 1, 7, 14, and 28 days; trailing means and standard deviations computed from prior days; day-of-week and holiday indicators; promotion depth; days since promotion start; store and product attributes; and weather forecasts. Products with intermittent demand receive a zero-demand indicator and are evaluated separately.

Approach

An expanding-window design uses three monthly validation origins, each forecasting the next 14 days. Preprocessing is refitted at every origin. Candidate models are ridge regression, random forest, gradient boosting, and a hybrid in which an ML model predicts residuals from a seasonal statistical forecast.

The evaluation uses MAE, WAPE, and a business-weighted loss that penalizes stockout risk more heavily for high-margin products. Performance is also reported by store size and product category. Feature importance is computed on held-out periods, and large errors are reviewed against stockouts, closures, and data outages.

Outcome and interpretation

The gradient-boosting model improves average error over the seasonal-naive baseline, particularly during promotions, while the statistical model is more stable for products with little history. A blended forecast performs best overall. The organization deploys the blend with a rule that falls back to the seasonal-naive forecast when promotion or weather data are missing.

The result is not interpreted as proof that a promotion causes a particular increase; promotional timing may be targeted to expected demand. Managers receive prediction intervals or scenario ranges, feature-vintage information, and an exception report for unusual forecasts.

Learning points

The main gains come from temporal feature design, realistic validation, and operational fallback—not from algorithm selection alone. The case also illustrates why segment-level evaluation, data availability, and asymmetric business costs are essential.

9. Challenges, Governance, and Contemporary Practice

Concept drift occurs when relationships change because of new competitors, policies, technology, or customer behavior. Drift monitoring should examine feature distributions, residuals, calibration, and business outcomes. Retraining frequency should be selected using evidence rather than an arbitrary calendar.

Forecasting systems can amplify historical inequities. If resource allocation or staffing forecasts systematically under-serve particular locations, the error has social as well as technical consequences. Governance should document data provenance, model purpose, exclusions, owners, retraining triggers, and human override procedures.

Privacy is relevant when forecasts use customer-level behavior, location, or transaction data. Data minimization, access controls, retention limits, and aggregation should be applied. External data may impose licensing restrictions. Automated decisions based on forecasts require clear accountability and audit trails.

Contemporary systems increasingly combine gradient boosting with probabilistic forecasting, hierarchical reconciliation, representation learning, and automated feature pipelines. Deep sequence models can be useful for very large, complex datasets, but they do not remove the need for leakage-safe design, strong baselines, uncertainty assessment, and governance. For many structured business datasets, well-engineered tabular ML remains competitive and easier to operate.

Best-Practice Workflow

1. Define the forecast origin, horizon, decision, and cost of errors.
2. Establish simple seasonal and statistical baselines.
3. Build a data-availability and feature-vintage specification.
4. Generate lagged, rolling, calendar, event, and external features using only permissible information.
5. Split data chronologically and use rolling-origin validation.
6. Fit preprocessing and feature selection inside each training fold.
7. Compare regularized regression, tree ensembles, boosting, and hybrids.
8. Evaluate overall and segment-level accuracy, uncertainty, stability, and operational impact.
9. Explain forecasts with validation-based importance and local diagnostics.
10. Monitor drift, missing inputs, residuals, and business outcomes after deployment.

Chapter Summary

Machine learning forecasting begins with a time-aware supervised-learning representation. Lag and rolling features encode persistence and local dynamics; calendar and event features represent recurring or intervention effects; external regressors add information only when their forecast-time availability is established. Regularized regression provides a transparent additive baseline, random forests capture nonlinear interactions with variance reduction, and gradient boosting often performs strongly on structured tabular features.

Leakage-safe rolling-origin validation is indispensable. All preprocessing and feature construction must respect the forecast origin, and evaluation should include meaningful baselines, appropriate error metrics, segments, and business costs. Feature importance and explainability support diagnosis and communication but do not establish causality. Global models borrow strength across series, whereas local models specialize; hybrid statistical–ML systems can combine reliable structural forecasts with nonlinear adjustments. Successful deployment additionally requires monitoring, uncertainty, privacy, documentation, and accountability.

Key Terms

Forecast origin: The time at which a forecast is issued.

Forecast horizon: The distance between the forecast origin and the target period.

Lag feature: A predictor formed from an earlier observation of a series.

Rolling feature: A statistic calculated over a trailing historical window.

Expanding feature: A statistic calculated using all available history up to a cutoff.

Calendar feature: A variable representing recurring time structure.

Event feature: A variable representing a promotion, intervention, or special occurrence.

External regressor: A predictor originating outside the target series.

Temporal leakage: Use of information unavailable at the forecast origin.

Rolling-origin validation: Repeated chronological training and testing at advancing origins.

Ridge regression: Regression with an L2 coefficient penalty.

Lasso regression: Regression with an L1 coefficient penalty.

Random forest: An ensemble of randomized decision trees.

Gradient boosting: Sequential construction of weak learners that reduce loss.

Direct forecasting: Training a separate model for each forecast horizon.

Recursive forecasting: Feeding predictions back into later forecasts.

Global model: One model trained across multiple related time series.

Local model: A separate model trained for an individual series.

Hybrid forecast: A forecast combining statistical and ML components.

Concept drift: Change in the data-generating relationship over time.

Review Questions

Short Answer Questions

1. Why is random train-test splitting usually unsuitable for time-series forecasting?
2. What distinguishes a known-future variable from an unknown-future variable?
3. State one advantage and one limitation of gradient boosting for forecasting.
4. Why should rolling means exclude observations unavailable at the forecast origin?
5. What is the difference between a global model and a local model?

Long Answer Questions

1. Explain how lag, rolling, calendar, event, and external features transform a time series into a supervised-learning dataset.
2. Compare ridge regression, random forests, and gradient boosting in terms of assumptions, strengths, and limitations.
3. Design a leakage-safe rolling-origin validation procedure for a 14-day demand forecast.
4. Discuss how permutation importance, partial dependence, and local explanations should be interpreted for correlated time-series features.
5. Explain the rationale, construction, and risks of a hybrid statistical–ML forecasting system.

Analytical / Application Questions

1. A retailer uses actual future temperature in a historical backtest but only weather forecasts in production. Diagnose the problem and redesign the evaluation.
2. A global model improves average accuracy but performs poorly for a small group of volatile products. Propose an evaluation and model-selection strategy.
3. A promotion model reduces MAE but increases stockouts. Identify possible reasons and recommend metrics, features, or decision rules that address the asymmetric business cost.

Multiple Choice Questions with Answers

Question 1

Which feature is generally valid for a forecast issued before period t?

A. The realized value of y_{t+1}

B. A trailing mean calculated from observations through t−1

C. The final revised value of an external variable published after t+1

D. A sales total entered after the forecast was issued

Correct Answer: B. A trailing mean calculated from observations through t−1

Question 2

What is the primary purpose of rolling-origin validation?

A. To randomize observations across folds

B. To ensure every fold has equal sample size

C. To reproduce the chronological sequence of training and forecasting

D. To eliminate the need for a baseline

Correct Answer: C. To reproduce the chronological sequence of training and forecasting

Question 3

Which regularization penalty can set some regression coefficients exactly to zero?

A. L1 penalty

B. L2 penalty only

C. Seasonal differencing

D. Bootstrap aggregation

Correct Answer: A. L1 penalty

Question 4

Why are sine and cosine often used together for hour-of-day or month-of-year?

A. They remove all missing values

B. They encode the circular nature of the calendar variable

C. They guarantee causal interpretation

D. They make tree models unnecessary

Correct Answer: B. They encode the circular nature of the calendar variable

Question 5

A recursive multi-step forecast differs from a direct forecast because it:

A. Uses a separate model for every feature

B. Feeds earlier predictions into later prediction steps

C. Cannot use lag features

D. Always has lower error

Correct Answer: B. Feeds earlier predictions into later prediction steps

Question 6

Which metric has the same units as the target and weights large errors more heavily?

A. RMSE

B. WAPE

C. MAPE

D. A correlation coefficient

Correct Answer: A. RMSE

Question 7

What is a principal limitation of tree-based feature importance based on split gain?

A. It cannot be calculated for training data

B. It may favor variables with many possible split points

C. It requires normally distributed residuals

D. It always identifies causal variables

Correct Answer: B. It may favor variables with many possible split points

Question 8

Why might a global model be useful for a product with little historical data?

A. It ignores all other products

B. It can borrow information from related series

C. It guarantees zero forecast error

D. It requires no validation

Correct Answer: B. It can borrow information from related series

Question 9

In a residual hybrid, the ML component should ideally be trained using:

A. Residuals based on in-sample fitted statistical forecasts only

B. Residuals from leakage-free rolling out-of-sample statistical forecasts

C. Randomly shuffled residuals

D. Future realized values unavailable at prediction time

Correct Answer: B. Residuals from leakage-free rolling out-of-sample statistical forecasts

Question 10

Which statement about explainability is correct?

A. A feature contribution automatically proves causality

B. Permutation importance is unaffected by correlated predictors

C. Explanations describe model behavior and must be interpreted with data availability and correlation in mind

D. Local explanations replace validation

Correct Answer: C. Explanations describe model behavior and must be interpreted with data availability and correlation in mind