Machine Learning and AI for Forecasting

Introduction

Forecasting has traditionally relied on statistical models whose structure is explicitly specified: exponential smoothing represents level, trend, and seasonality; ARIMA models dependence on past observations and shocks; and regression models relate an outcome to explanatory variables. Machine learning (ML) and artificial intelligence (AI) extend forecasting by learning complex, nonlinear, high-dimensional relationships from data.

The central question is not whether ML is universally superior to statistical forecasting. It is when a particular method improves decisions, given the available data, forecast horizon, computational resources, interpretability requirements, and consequences of error. A small, stable, seasonal series may be forecast more reliably by exponential smoothing than by a deep neural network. Conversely, thousands of related demand series with promotions, prices, weather, web traffic, and calendar effects may benefit from global ML models.

This chapter examines regularized regression, decision trees, random forests, boosting, neural networks, recurrent architectures, temporal convolutional networks, transformers, automated forecasting, and foundation models. It also compares these methods with statistical approaches through the dimensions of predictive accuracy, explainability, robustness, data requirements, and governance.

Learning Objectives

After studying this chapter, you should be able to:

1. Construct time-aware features and explain why ordinary random cross-validation is inappropriate for forecasting.
2. Apply regularized regression, trees, ensembles, boosting, and neural architectures to forecasting problems.
3. Distinguish recurrent, convolutional, attention-based, automated, and foundation-model approaches.
4. Compare ML methods with statistical models using accuracy, interpretability, robustness, and data requirements.
5. Design a defensible forecasting workflow and communicate its limitations to decision-makers.

1. The Forecasting Problem in an ML Setting

Let y_t denote the observed value at time t. A forecast made at origin t for h periods ahead is written as \hat{y}_t+h|t. The information set available at the forecast origin is \mathcal{I}_t. A forecasting model estimates a function

where f_h is the forecasting function and \theta represents learned parameters. The information set may include lagged outcomes, calendar variables, prices, promotions, weather, inventory, economic indicators, and forecasts from other systems.

A supervised ML formulation converts the time series into training examples. For a one-step model, a feature vector may be

and the target may be y_t+1. For multiple horizons, either a separate model is trained for each horizon, or one model receives the horizon as an additional feature. Features must be available at the time the forecast is issued. Using a value that becomes known only later is leakage and produces falsely optimistic evaluation.

Forecasting-specific sources of difficulty

Time series differ from ordinary cross-sectional data in several ways:

• Temporal dependence: nearby observations are often correlated.
• Nonstationarity: relationships, means, variances, and seasonal patterns can change.
• Changing availability: future explanatory variables may be unknown and require their own forecasts.
• Intervention and regime change: policy, competition, technology, and crises can alter the data-generating process.
• Unequal costs: under-forecasting may be more costly than over-forecasting.
• Hierarchies and intermittency: products may aggregate into regions, while many item-level series contain zeros.

The loss function should reflect the decision. Common measures include mean absolute error,

and root mean squared error,

where n is the number of forecast cases and both y_i and \hat{y}_i use the same units. MAE weights all absolute errors proportionally; RMSE gives greater influence to large errors. Percentage measures can be unstable when actual values are zero or close to zero. For probabilistic forecasts, interval coverage and proper scoring rules such as the continuous ranked probability score are often more appropriate than point accuracy alone.

2. Regularized Regression

Regression is often the strongest baseline for ML forecasting because it is transparent, fast, and effective when relationships are approximately additive. A feature-based forecasting regression can include lags, rolling statistics, seasonal indicators, trend terms, and external variables.

Ordinary least squares estimates coefficients by minimizing squared residuals. With correlated lags or many features, coefficients can become unstable. Ridge regression adds an L2 penalty:

Here \beta₀ is the intercept, \boldsymbol{\beta} contains p coefficients, \lambda\geq0 controls shrinkage, and features should generally be standardized when penalty comparisons are intended. Larger \lambda reduces coefficient magnitude and variance but may increase bias.

Lasso regression uses an L1 penalty:

Lasso can set some coefficients exactly to zero, providing feature selection. Elastic net combines L1 and L2 penalties. Regularization does not automatically solve leakage, nonstationarity, or poor feature design. It must be tuned using time-ordered validation, not random shuffling.

Feature engineering

Useful features include:

• lags such as y_t-1 and y_t-7;
• rolling means and standard deviations computed using past data only;
• differences, growth rates, and logarithms for suitable positive variables;
• day-of-week, month, holiday, and fiscal-period indicators;
• price, promotion, stock availability, weather, and macroeconomic variables;
• interactions, such as price multiplied by promotion status.

A rolling mean over the previous k observations is

The omission of y_t is essential when the feature is used to predict y_t or a future value. If future covariates are uncertain, the forecasting system should either forecast them, use scenarios, or restrict the model to variables known at issuance.

3. Trees, Random Forests, and Boosting

Decision trees recursively partition the feature space. A regression tree assigns a constant prediction to each terminal region. Trees capture thresholds and interactions without requiring the analyst to specify them in advance. For example, demand may increase sharply when temperature falls below a threshold and only during weekends.

A single tree is easy to visualize but can overfit. A random forest averages many trees trained on bootstrap samples and randomized feature subsets. If the individual tree errors are not perfectly correlated, averaging reduces variance. Forests are robust to nonlinearities and mixed feature types, but they may be less effective at extrapolating a trend beyond the range observed in training data.

Boosting builds an additive sequence of weak learners. In a simplified squared-error formulation, the prediction after m stages is

where h_m is the next tree and \eta is the learning rate. The new tree is fitted to remaining errors or a related gradient signal. Gradient boosting, XGBoost, LightGBM, and CatBoost are widely used tabular forecasting tools. Important controls include tree depth, number of trees, learning rate, row and feature subsampling, and regularization.

Trees and boosting are especially useful when many external drivers are available and interactions matter. They require careful temporal validation, treatment of missing values, and monitoring for drift. Feature importance is not the same as causality: a variable may predict demand because it is a proxy for another process.

4. Neural Networks for Forecasting

A feed-forward neural network maps inputs through weighted layers and nonlinear activation functions. A typical hidden unit is

where w_jk are weights, b_j is a bias, and \phi is an activation function such as ReLU. Parameters are learned by minimizing a loss using gradient-based optimization.

Neural networks can represent nonlinear interactions and learn shared patterns across many series. However, they have high variance when the dataset is small, are sensitive to scaling and optimization choices, and can be difficult to explain. Dropout, weight decay, early stopping, and validation-based hyperparameter selection reduce overfitting but do not guarantee reliable forecasts.

Recurrent neural networks

Recurrent neural networks process observations sequentially using a hidden state. A simplified recurrence is

where \mathbf{h}_t summarizes prior information. Vanilla RNNs can suffer from vanishing or exploding gradients over long sequences. Long short-term memory networks (LSTMs) and gated recurrent units (GRUs) use gates to regulate information retention and updating. They can model complex sequential dynamics, but they are not automatically superior to simpler models and may be slower to train.

Temporal convolutional networks

A one-dimensional temporal convolution applies learned filters to local sequences. Dilated convolutions expand the receptive field by skipping input positions according to a dilation factor. Causal convolutions ensure that the output at time t uses no future observation. Temporal convolutional networks can train in parallel and model long contexts through stacked dilations. They are useful when local motifs, repeated patterns, and stable sequence structure are important.

Transformers and attention

Attention allows a model to assign different weights to elements of a sequence. A scaled dot-product attention operation is

where Q, K, and V are query, key, and value matrices, and d_k is the key dimension. The scaling reduces excessively large dot products. Positional information is required because attention alone does not inherently encode order.

Transformers can connect distant events and support multivariate, multi-horizon forecasting. Their costs include substantial data and computation requirements, sensitivity to distribution shift, and the possibility of learning spurious correlations. Attention weights should not automatically be interpreted as causal explanations; they indicate a mechanism of information weighting, not necessarily variable importance in a causal sense.

5. Automated Forecasting and Foundation Models

Automated forecasting systems search across candidate models, transformations, seasonal specifications, hyperparameters, and ensembles. A responsible automated workflow includes data-quality checks, leakage detection, time-series cross-validation, baseline comparison, residual diagnostics, and monitoring. Automation accelerates experimentation; it does not remove the need for judgment about business context, availability, hierarchy, and loss functions.

A common validation design is rolling-origin evaluation. Train on an initial period, forecast a later validation window, expand or roll the training period, and repeat. If origins are t₁, t₂, \ldots, t_r, an average validation loss can be written as

where H is the horizon and L is the chosen loss. The design should resemble how forecasts will actually be produced.

Foundation models are large pretrained models intended for adaptation across tasks. In time-series forecasting, they may be pretrained on many series and then used zero-shot, prompted with context, fine-tuned, or combined with domain-specific models. Their potential advantages include transfer learning, reduced task-specific training, and useful performance when related patterns exist. Their limitations include uncertain training-data provenance, weak performance on unfamiliar regimes, computational expense, limited transparency, and possible privacy or licensing concerns. “Foundation model” is not a guarantee of accuracy; it is an architectural and training paradigm.

6. Statistical Models versus ML Models

Statistical models often impose a stronger structure, which can be advantageous with limited data. ARIMA-family models represent autocorrelation and differencing explicitly; exponential smoothing represents level, trend, and seasonality; state-space models support updating and uncertainty estimation. ML models generally provide greater flexibility in nonlinear, multivariate, and high-dimensional settings.

The best practice is empirical comparison against strong baselines, not methodological allegiance. A seasonal naive forecast, for example, predicts the value from the corresponding previous season and is difficult to beat in stable seasonal environments. Forecast combinations frequently improve robustness because different models make different errors.

Explainability should operate at several levels: global model behavior, individual forecast explanations, sensitivity to scenarios, and uncertainty communication. For tree models, permutation importance or carefully interpreted SHAP values may help; for neural models, perturbation tests, integrated gradients, or feature ablation can provide evidence. Explanations should be checked for stability and should not be presented as causal claims without a causal design.

7. A Defensible End-to-End Workflow

1. Define the decision. Specify the forecast target, horizon, frequency, granularity, cost of errors, and required output.
2. Audit the data. Check timestamps, duplicates, missingness, outliers, revisions, stockouts, interventions, and changes in measurement.
3. Establish baselines. Use naive, seasonal naive, moving-average, and suitable statistical models.
4. Construct leakage-safe features. Record when each feature becomes available and calculate rolling features using past data only.
5. Create temporal validation splits. Preserve order and include multiple forecast origins.
6. Train candidates. Compare regularized regression, boosting, statistical models, and neural approaches only when justified by data volume and complexity.
7. Tune and calibrate. Select hyperparameters using validation loss; assess prediction intervals or quantiles where decisions require uncertainty.
8. Diagnose errors. Examine errors by product, location, horizon, season, promotion status, and regime.
9. Select or combine models. Consider accuracy, stability, explainability, latency, cost, and operational maintainability.
10. Deploy and monitor. Track data drift, forecast error, bias, coverage, feature availability, and business outcomes. Establish retraining and override procedures.

8. Illustrative Case: Regional Grocery Demand

Background and problem

A grocery retailer forecasts daily demand for 2,000 products across 40 stores, seven days ahead. Demand is seasonal by weekday, affected by promotions and price, and distorted by stockouts. The operations team needs forecasts for replenishment and wants explanations when a forecast changes sharply.

Analysis

The retailer first distinguishes true low demand from stockout-censored demand. It records promotion calendars, prices, store characteristics, holidays, weather forecasts, and product hierarchies. A random train-test split is rejected because it allows future promotion regimes and demand patterns to influence training.

Approach

The team establishes seasonal-naive and exponential-smoothing baselines. It then trains a global gradient-boosting model with lag features, rolling statistics, product and store identifiers, promotion variables, and weather forecasts. Numeric features are scaled where required; categorical variables are encoded consistently. A regularized regression model provides a transparent comparison. For products with long histories and stable seasonality, statistical models are retained. For sparse products, the retailer evaluates pooled models and intermittent-demand methods rather than forcing a deep network.

Rolling-origin validation is conducted over several months. Performance is assessed using MAE and a weighted business loss that penalizes stockout-inducing under-forecasts more heavily. Forecasts are reconciled or jointly evaluated across store and regional levels so that operational plans do not contradict aggregate expectations.

Outcome

The boosting model improves average validation accuracy for promoted and weather-sensitive products, while exponential smoothing remains competitive for stable high-volume products. The retailer adopts a segmented model portfolio rather than one universal algorithm. Forecast explanations report recent sales, promotion status, seasonal position, and weather contribution, with a warning that these are predictive associations rather than causal effects.

Learning points

The case illustrates that model selection depends on data structure and operational costs. Data cleaning, feature availability, temporal validation, and monitoring contribute as much to success as algorithm choice. A portfolio can be more robust than a single complex model, and a simple model may be the best choice for a particular segment.

9. Limitations, Ethics, and Governance

Forecasts can influence staffing, credit, inventory, public services, and pricing. Historical data may encode discrimination or unequal service. A model that predicts lower demand in an underserved area may reinforce reduced availability, making future demand lower and apparently validating the model. Analysts should examine performance and error distributions across relevant groups and locations.

Important governance practices include documented data lineage, reproducible training, versioned models, access controls, privacy protection, human accountability, and an audit trail for overrides. Personal data should be minimized and processed according to applicable law and organizational policy. Vendor claims about AI performance should be tested on representative local data.

Robustness testing should include missing covariates, delayed data, outliers, holidays, structural breaks, extreme but plausible scenarios, and adversarial or manipulated inputs. Stress tests do not predict the future, but they reveal dependence on fragile assumptions. Forecast intervals should be calibrated on data resembling deployment conditions, and users should understand that uncertainty generally increases with horizon.

10. Contemporary Best Practices

• Start with a decision-relevant baseline and a clearly defined loss function.
• Prefer global ML models when many related series share information; prefer local statistical models when series are few and stable.
• Use leakage-safe feature pipelines and time-aware validation.
• Treat external-variable forecasts as uncertain inputs, not fixed truth.
• Use ensembles or model portfolios when regimes and series differ.
• Separate predictive explanation from causal explanation.
• Evaluate both average accuracy and worst-group, worst-horizon, and high-impact errors.
• Monitor drift and recalibrate after major changes.
• Keep models simple enough to operate, audit, and replace.
• Use foundation models selectively, validating their provenance, privacy, cost, and out-of-distribution behavior.

Chapter Summary

ML forecasting converts temporal data into supervised learning problems while preserving time order and information availability. Regularized regression provides a strong, interpretable baseline for high-dimensional lag and calendar features. Trees and boosting capture nonlinearities and interactions effectively, particularly with tabular external variables. Neural networks, recurrent architectures, temporal convolutions, and transformers can learn complex sequential representations but require more data, tuning, and governance. Automated forecasting accelerates model search, while foundation models offer transfer learning but introduce questions about provenance, generalization, and transparency.

Statistical models remain valuable because they can perform strongly with limited data, represent seasonality and autocorrelation explicitly, and often provide understandable uncertainty estimates. Model choice should therefore be based on rigorous rolling-origin evaluation, decision-relevant loss, robustness, explainability, operational constraints, and ethical governance. In practice, a validated portfolio of statistical and ML methods is often more reliable than an assumption that the newest model must be best.

Key Terms

Forecast origin: The time at which a forecast is produced.

Forecast horizon: The number of periods between forecast origin and target.

Data leakage: Use of information unavailable at the time the forecast is issued.

Regularization: A penalty that constrains model complexity to reduce overfitting.

Ridge regression: Regression using an L2 penalty on coefficients.

Lasso regression: Regression using an L1 penalty that can produce sparse coefficients.

Decision tree: A model that partitions feature space through sequential rules.

Random forest: An ensemble of randomized decision trees whose predictions are aggregated.

Boosting: Sequential construction of learners that focus on remaining errors.

Global model: One model trained across multiple related time series.

RNN: A neural network that maintains a sequential hidden state.

LSTM: A gated recurrent architecture designed to retain information over longer sequences.

Temporal convolution: A convolutional operation designed for ordered time observations.

Attention: A mechanism that weights information according to learned relevance.

Transformer: An architecture primarily based on attention and feed-forward layers.

Foundation model: A broadly pretrained model adapted to multiple downstream tasks.

Rolling-origin validation: Repeated time-ordered training and evaluation at successive origins.

Concept drift: Change in the relationship between inputs and the forecasting target.

Probabilistic forecast: A forecast represented by a distribution, quantiles, or intervals.

Explainability: The ability to communicate and examine how a model produces predictions.

Review Questions

Short Answer Questions

1. Why is random shuffling usually inappropriate for evaluating a forecasting model?
2. Distinguish ridge regression from lasso regression.
3. What problem does a seasonal-naive baseline help identify?
4. Why must future availability be considered when selecting explanatory variables?
5. State one advantage and one limitation of boosting for forecasting.

Long Answer Questions

1. Explain how regularized regression can be used to forecast demand with lags, rolling features, and calendar variables. Include the principal assumptions and risks.
2. Compare statistical models, tree ensembles, and deep neural models in terms of data requirements, explainability, nonlinear representation, and uncertainty estimation.
3. Describe the architecture and forecasting implications of RNNs, temporal convolutional networks, and transformers.
4. Design a rolling-origin evaluation strategy for a seven-day-ahead retail forecasting problem.
5. Discuss ethical, privacy, and governance issues that arise when AI forecasts influence resource allocation.

Analytical / Application Questions

1. A company has 18 monthly observations for each of 50 products and strong annual seasonality. Would a transformer be your first choice? Justify a model portfolio and validation plan.
2. A boosting model has lower average MAE but substantially worse errors during stockouts than an exponential-smoothing model. Recommend a selection or combination strategy using business loss and explainability.
3. A foundation model performs well on historical data but fails during a regulatory regime change. Analyze possible causes and propose robustness, monitoring, and retraining measures.

Multiple Choice Questions with Answers

Question 1

Which practice most directly prevents leakage when calculating a rolling mean to forecast y_t?

A. Include y_t in the rolling window

B. Use only observations before t

C. Randomly shuffle the observations first

D. Replace missing values with future averages

Correct Answer: B. Use only observations before t

Question 2

What is the principal effect of increasing the ridge penalty λ, all else equal?

A. It forces every coefficient to zero

B. It increases coefficient magnitudes

C. It shrinks coefficients toward zero

D. It removes the intercept in every model

Correct Answer: C. It shrinks coefficients toward zero

Question 3

Which method is especially designed to capture nonlinear thresholds and interactions in tabular forecasting features?

A. Decision-tree ensemble

B. Seasonal naive forecasting

C. Simple moving average

D. Random timestamp permutation

Correct Answer: A. Decision-tree ensemble

Question 4

Why are seasonal-naive forecasts valuable in model evaluation?

A. They always outperform neural networks

B. They provide a strong, interpretable benchmark

C. They eliminate the need for validation

D. They estimate causal promotion effects

Correct Answer: B. They provide a strong, interpretable benchmark

Question 5

In a recurrent neural network, the hidden state primarily serves to:

A. Store a representation of sequential information

B. Guarantee causal interpretation

C. Remove all seasonal variation

D. Replace the forecast target

Correct Answer: A. Store a representation of sequential information

Question 6

What is the main purpose of scaling the dot product by √d_k in attention?

A. To create calendar features

B. To reduce excessively large dot products

C. To ensure all forecasts are unbiased

D. To remove the need for positional information

Correct Answer: B. To reduce excessively large dot products

Question 7

Which validation design best reflects repeated real-world forecasting?

A. Random k-fold shuffling

B. Validation on the training observations

C. Rolling-origin evaluation

D. Evaluation using future features unavailable at issuance

Correct Answer: C. Rolling-origin evaluation

Question 8

Which statement about attention weights is most accurate?

A. They automatically establish causal effects

B. They are always identical to feature importance

C. They show learned information weighting but are not by themselves causal explanations

D. They can only be used for univariate data

Correct Answer: C. They show learned information weighting but are not by themselves causal explanations

Question 9

A forecast has actual values 10 and 14, and predictions 12 and 11. What is the MAE?

A. 1.5

B. 2.0

C. 2.5

D. 3.0

Correct Answer: A. 1.5

Question 10

Which consideration is especially important before deploying a foundation model for forecasting?

A. Whether its outputs use decorative graphics

B. Training-data provenance, privacy, and out-of-distribution performance

C. Whether it avoids all validation

D. Whether it always replaces statistical baselines

Correct Answer: B. Training-data provenance, privacy, and out-of-distribution performance