Forecasting with Drivers and Contemporary AI

Introduction

Many business outcomes are influenced by more than their own historical values. Retail demand responds to price, promotions, holidays, weather and advertising. Electricity consumption depends on temperature, daylight and industrial activity. Call-centre volume is affected by product releases, service failures and billing cycles. A forecasting model that uses such explanatory information is often more useful than a model based only on the past values of the variable being forecast.

This chapter examines forecasting with external drivers, event variables and lag features. It then considers machine-learning forecasting, automated forecasting systems, deep learning and generative artificial intelligence. The central principle is that advanced technology does not remove the need for sound data design, time-aware validation, domain knowledge and responsible judgement.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish univariate forecasting from forecasting with external drivers;
• formulate and interpret dynamic regression models;
• construct event variables and lagged predictors without data leakage;
• evaluate machine-learning forecasts using time-ordered validation;
• explain when automated forecasting and deep learning are appropriate;
• identify the limitations, ethical risks and governance requirements of contemporary AI forecasting;
• communicate forecast results and uncertainty to business decision-makers.

1. Forecasting Beyond the Historical Series

A univariate forecast uses the past of the target variable, such as sales, to predict its future. A driver-based forecast also uses variables that help explain or predict changes in the target. These variables are commonly called exogenous variables, external regressors, predictors, covariates or features.

Suppose y_t is weekly demand at time t, p_t is price, a_t is advertising expenditure and h_t indicates a holiday. A simple regression with drivers can be written as:

Here, β₀ is the intercept, β₁, β₂ and β₃ are coefficients, and ε_t is the unexplained error at time t. The units of each coefficient depend on the units of the variables. For example, β₁ measures the expected change in demand units associated with a one-unit price change, holding the other included variables constant.

This equation is a useful starting point, but ordinary regression assumptions may be inappropriate for time series. Errors can be correlated over time, the relationship can change, predictors may be unavailable in the forecast period, and the target may have trend or seasonality. Dynamic regression addresses these issues by combining explanatory variables with time-series structure.

2. Dynamic Regression

A dynamic regression model allows the target and predictors to have delayed effects. A general form is:

In this model, y_t-i represents lagged target values, x_t-j represents current or lagged external drivers, p and q are selected maximum lag orders, s_t represents deterministic effects such as trend, seasonality or events, and ε_t is the error term. The summation notation means that the model may include several lags rather than only one.

A dynamic regression may be used when:

• past demand influences current demand;
• a promotion has effects that continue for several periods;
• advertising affects sales with a delay;
• operational variables have lead or recovery effects;
• calendar patterns are predictable.

A key forecasting condition is that future values of the external predictors must be known, reliably forecast, or represented through scenarios. A model using tomorrow’s temperature is practical if a weather forecast is available. A model using next month’s competitor price is more difficult unless that price is contractually fixed or separately forecast.

2.1 Interpreting coefficients

Coefficient interpretation depends on the model form. In a level-level model, a coefficient expresses a change in the target’s units for a one-unit change in the predictor. In a log-level model, the coefficient represents an approximate percentage change in the target for a one-unit predictor change. In a log-log model, the coefficient is an elasticity: the approximate percentage change in the target for a 1% change in the predictor.

For example, if:

then β₁ is an elasticity under the usual interpretation, assuming the model is appropriate and other relevant factors are controlled. The logarithm requires positive values, so zero or negative observations need careful treatment rather than an automatic log transformation.

Regression coefficients should not automatically be interpreted as causal effects. If advertising expenditure rises when managers expect high demand, a positive advertising coefficient may partly reflect reverse causality or omitted factors. Forecasting usefulness and causal explanation are related but distinct goals.

3. Events, Calendar Effects and Interventions

Events are often more informative than a smooth historical trend. Common examples include public holidays, product launches, strikes, system outages, sporting events, school terms and changes in regulation.

An event can be represented in several ways:

A binary event indicator may be written as:

If the event is expected to increase sales by a temporary amount, the model may include β_I I_t. If the event changes price sensitivity, an interaction can be included:

The coefficient β₃ indicates how the price relationship differs during the event. Interaction terms should be included only when they have a defensible business interpretation and sufficient observations.

Event data must be historically accurate. A common error is to encode an event using information that became known only after the forecast origin. For example, using the final number of days a strike lasted would leak future information into a historical training record. The feature should reflect what was knowable at the time.

4. Lag Features and Rolling Features

A lag feature uses an earlier observation to predict a later one. For a target series, typical features include:

• y_t-1: value one period earlier;
• y_t-7: value seven periods earlier for daily data with weekly seasonality;
• rolling mean over the preceding four periods;
• rolling standard deviation measuring recent variability;
• cumulative or exponentially weighted measures of recent activity.

A four-period trailing mean, calculated without using the current observation, is:

The feature is available at time t because it uses only information from periods before t. A rolling feature that includes y_t when predicting y_t is leakage.

Lag features are powerful because they allow flexible algorithms to learn temporal patterns. However, they create practical problems. The first observations may have missing lag values, highly correlated lags may create unstable models, and a long list of lags can overfit. Feature selection should be guided by business cycles, sampling frequency and validation performance.

5. Machine-Learning Forecasting

Machine-learning models are often applied to forecasting by converting the time series into a supervised learning dataset. Each row represents a forecast origin, the columns contain features available at that origin, and the target is a future value.

For a one-step-ahead forecast, a row might contain:

Suitable models include regularized linear regression, decision trees, random forests, gradient-boosted trees and neural networks. Tree-based models can represent nonlinear effects and interactions, such as a promotion being especially effective when inventory is high. They do not, however, understand time automatically; lag and calendar features must be designed correctly.

5.1 Multi-step forecasting strategies

For a horizon of several periods, three common strategies are used:

1. Recursive strategy: predict the next period, feed that prediction into the feature set, and continue. It is simple but errors can accumulate.
2. Direct strategy: train a separate model for each horizon, such as one model for one week ahead and another for four weeks ahead. It avoids repeatedly feeding predictions back but requires more models.
3. Multi-output strategy: train one model to predict several future periods simultaneously. It can represent relationships across the horizon but may require more data and specialized algorithms.

The strategy should reflect the decision. Inventory replenishment may require several daily forecasts, while a staffing decision may focus on a particular weekly peak.

5.2 Evaluation metrics

For observations y_t and forecasts ŷ_t, common metrics include:

Here, n is the number of evaluated forecasts. MAE is expressed in the target’s units. RMSE also uses target units but penalizes large errors more heavily. MAPE is percentage-based but is undefined when y_t = 0 and unstable when actual values are close to zero. For intermittent or zero-heavy demand, alternatives such as MAE, weighted measures or scaled errors are usually safer.

A business metric may be more useful than a statistical metric. For example, under-forecasting may create stockouts and lost sales, while over-forecasting may create holding costs. Forecast evaluation should therefore consider asymmetric consequences.

6. Time-Aware Validation and Leakage Control

Randomly shuffling time-series records is usually inappropriate because it allows information from the future to influence the past. Instead, use a chronological holdout or rolling-origin evaluation.

In a rolling-origin design, a model is trained on an initial period, tested on the next period, then the training window is expanded or moved forward. The process is repeated. This estimates how the model performs under repeated historical forecasting conditions.

A robust workflow is:

1. define the forecast horizon and forecast origin;
2. separate training, validation and final test periods chronologically;
3. create every feature using only information available at each origin;
4. fit preprocessing steps inside each training window;
5. tune hyperparameters using time-aware validation;
6. compare with simple baselines;
7. evaluate once on the untouched final test period;
8. monitor performance after deployment.

Baselines are essential. A seasonal-naïve forecast, for example, may use the value from the same season in the previous cycle. A complex model should earn its complexity by improving accuracy, reliability or business usefulness.

7. Automated Forecasting

Automated forecasting systems can select models, transform data, tune parameters, generate forecasts and produce prediction intervals. Automation improves consistency and scalability, particularly when an organisation must forecast thousands of products or locations.

Automation is most useful when:

• the data pipeline is stable;
• many related series require repeatable treatment;
• model selection rules are transparent;
• baselines and validation are built into the process;
• exceptions are escalated to analysts.

Automation is not a substitute for problem formulation. It cannot determine whether a sudden fall represents genuine demand loss, a stockout, a data-entry failure or a change in measurement. It may also select a model that performs well historically but fails after a structural change.

A professional automated system should record data versions, feature definitions, training dates, model versions, validation results and overrides. Monitoring should include forecast error, missing data, feature drift, changes in event frequency and prediction-interval coverage.

8. Deep Learning for Time Series

Deep-learning methods include recurrent neural networks, long short-term memory networks, temporal convolutional networks and transformer-based architectures. They can learn nonlinear and long-range relationships from large collections of related time series and covariates.

Deep learning is more defensible when:

• there is a large volume of observations or many related series;
• relationships are nonlinear and complex;
• rich covariates, text, images or high-frequency signals are available;
• computing infrastructure and specialist skills exist;
• the organisation can maintain monitoring and retraining.

It is often unnecessary for a small, stable business series with limited history. Deep models can be difficult to explain, sensitive to data quality and computationally expensive. A simpler model may be more accurate, more stable and easier to audit.

Forecast uncertainty remains important. A highly accurate point forecast does not reveal the probability of demand being above a capacity threshold. Prediction intervals, quantile forecasts or scenario ranges should be considered when decisions involve risk.

9. Generative AI in Forecasting Workflows

Generative AI can assist forecasting work, but it should not be treated as an autonomous source of numerical truth. Appropriate uses include:

• translating a business question into a forecasting plan;
• generating draft code that is reviewed and tested;
• documenting feature definitions and model assumptions;
• summarising forecast changes for managers;
• helping analysts explore plausible scenarios;
• providing a natural-language interface to approved analytical systems.

Inappropriate uses include asking a language model to invent missing historical values, accept unverified forecasts, expose confidential data, or generate unsupported causal explanations. Generative models can produce fluent but false statements, misread units, omit uncertainty and reproduce bias present in their training data.

A controlled workflow separates generation from verification. The model may draft an explanation, but the numerical values should be retrieved from the governed forecasting system. Code should be executed in a controlled environment, tested against known results and reviewed by a qualified person. Prompts and outputs may need to be retained for audit purposes.

10. Illustrative Case: Forecasting Grocery Demand

Background

A regional supermarket wants to forecast weekly demand for a chilled meal. The product experiences weekly seasonality, price promotions, public-holiday effects and occasional stockouts.

Problem

The existing forecast uses only the previous week’s sales. Managers report that it under-forecasts promotional weeks and over-forecasts weeks when shelves were empty. The business needs a four-week forecast for purchasing and staffing.

Analysis

The analytics team first distinguishes observed sales from unconstrained demand. Sales during stockouts are not reliable measures of demand. The team creates:

• lags of one and seven weeks;
• a four-week trailing mean using only prior observations;
• promotion indicators and discount depth known in advance;
• holiday and holiday-adjacent indicators;
• planned distribution-centre constraints;
• a stockout flag for historical diagnosis.

The stockout flag is not used as a future predictor unless future stockouts are known or separately forecast. Otherwise, it would create leakage or an unrealistic input requirement.

Approach and solution

Three approaches are compared: a seasonal-naïve baseline, dynamic regression and a gradient-boosted model using lag, calendar and promotion features. Rolling-origin validation is performed with a four-week horizon. MAE is selected as the primary metric because it is interpretable in meal units, while under-forecast frequency is monitored separately because stockouts are costly.

The boosted model has the lowest average MAE, but the dynamic regression is easier to explain and performs nearly as well. The organisation chooses the dynamic regression for routine planning and retains the boosted model as a challenger. Forecasts are accompanied by prediction intervals and a warning when planned promotions differ substantially from historical patterns.

Outcome

The most important improvement comes not from algorithmic complexity but from correct promotion encoding, stockout diagnosis and time-aware validation. Managers receive a forecast, an expected range and a short explanation of the main drivers. When a new promotion has no historical analogue, the system presents a scenario range rather than pretending to have precise knowledge.

Learning points

• recorded sales may understate true demand;
• future driver availability must be checked before model selection;
• a baseline provides a meaningful benchmark;
• validation must reproduce the real forecasting process;
• model transparency and operational fit can outweigh a small accuracy advantage.

11. Governance, Ethics and Professional Practice

Forecasts influence staffing, credit decisions, inventory access, pricing and public services. Poor forecasts can therefore create unequal effects. Governance should address:

• privacy: restrict personal or sensitive data and apply appropriate access controls;
• fairness: test whether errors differ systematically across customer or geographic groups;
• transparency: document data sources, variables, model purpose and limitations;
• accountability: assign responsibility for approval, deployment and intervention;
• security: protect forecast systems, credentials and sensitive commercial drivers;
• reproducibility: preserve code, data versions, parameters and evaluation results;
• human oversight: require review for high-impact decisions and unusual forecasts.

A forecast should not be presented with false precision. If the future driver is uncertain, that uncertainty should be propagated through scenarios or prediction intervals. Decision-makers should know whether a result is an observed pattern, a model-based extrapolation or a management assumption.

12. Best-Practice Implementation Checklist

Before deployment, an analyst should be able to answer the following questions:

1. What exactly is being forecast, at what frequency and for what horizon?
2. Which drivers will genuinely be available at forecast time?
3. Are event dates and historical feature values recorded as they were known then?
4. Have lag and rolling features excluded future information?
5. Does the model outperform a relevant naïve baseline?
6. Has validation reproduced the intended operational use?
7. Are uncertainty and asymmetric business costs reported?
8. What happens when drivers are missing or outside historical ranges?
9. Who reviews unusual forecasts and approves model changes?
10. How will drift, bias, accuracy and data quality be monitored?

Chapter Summary

Forecasting with drivers extends time-series analysis by incorporating variables such as price, advertising, weather, calendar effects and planned events. Dynamic regression combines these predictors with lagged targets and other temporal structure. Event variables must reflect the timing and duration of interventions, while lag and rolling features must be created without leakage.

Machine-learning models can capture nonlinear relationships and interactions, but they require carefully designed features and time-aware validation. MAE, RMSE and percentage measures answer different questions, and business costs should influence metric selection. Automated forecasting improves scale and consistency but cannot replace data governance or domain judgement. Deep learning is valuable for large, complex datasets but is not automatically superior to simpler methods. Generative AI can support coding, documentation and communication, provided that numerical outputs, privacy, security and assumptions are independently verified.

The professional forecaster therefore combines statistical reasoning, feature engineering, operational knowledge, model evaluation, uncertainty communication and responsible AI governance.

Key Terms

Dynamic regression: A regression model that includes time-dependent structure such as lagged targets or lagged predictors.

External driver: A variable outside the target series that helps predict its future values.

Exogenous variable: A predictor treated as determined outside the forecasting model.

Event variable: A feature representing a holiday, intervention, promotion or other special occurrence.

Pulse variable: An indicator for an event occurring in a limited period, often one period.

Step variable: A variable representing a persistent level change after an intervention.

Lag feature: A predictor formed from an earlier observation.

Rolling feature: A summary calculated over a preceding moving window.

Data leakage: The use of information unavailable at the time a historical forecast would have been made.

Forecast origin: The time at which information is collected and a forecast is produced.

Recursive forecasting: A multi-step approach that feeds earlier predictions into later predictions.

Direct forecasting: Training separate models for separate forecast horizons.

MAE: Mean absolute error, the average absolute difference between actuals and forecasts.

RMSE: Root mean squared error, a metric that penalises larger errors more strongly.

Prediction interval: A range intended to contain a future observation with a stated probability or coverage level.

Automated forecasting: A system that performs model selection, estimation or forecasting with limited manual intervention.

Deep learning: Machine learning using layered neural-network architectures.

Generative AI: AI that produces new text, code, images or other outputs from learned patterns.

Model drift: A decline in performance caused by changes in data patterns or relationships.

Scenario forecast: A forecast conditional on specified assumptions about future drivers.

Review Questions

Short Answer Questions

1. What is the main difference between a univariate forecast and a forecast with external drivers?
2. Why must future values of external predictors be available or forecastable?
3. Define data leakage and give one forecasting example.
4. State one strength and one limitation of RMSE.
5. Why can a seasonal-naïve forecast be useful when evaluating a complex model?

Long Answer Questions

1. Explain how dynamic regression incorporates both external drivers and temporal dependence. Include an example of a lagged driver.
2. Compare pulse, step, ramp and window event variables, and describe a business situation for each.
3. Discuss recursive, direct and multi-output strategies for multi-step forecasting.
4. Evaluate the circumstances in which deep learning is preferable to a simpler forecasting model.
5. Explain how generative AI can be used responsibly in a forecasting workflow and identify at least four risks.

Analytical / Application Questions

1. A retailer forecasts daily demand using a seven-day lag, a trailing seven-day mean and a promotion indicator. Describe how you would construct these features for a forecast made at the end of Tuesday, and identify any leakage risks.
2. A model has lower RMSE but higher MAE than a competing model. What might this reveal about its errors, and which additional business information would you request before selecting a model?
3. A company plans a promotion unlike any previous promotion and does not know the competitor’s response. Design a forecasting and communication approach that uses drivers, scenarios and uncertainty without claiming unjustified precision.

Multiple Choice Questions with Answers

Question 1

Which feature is most likely to create data leakage when forecasting demand for day t?

A. Demand on day t − 1

B. A promotion planned before day t

C. A rolling mean calculated from days t − 1 through t − 7

D. Demand recorded on day t itself

Correct Answer: D. Demand recorded on day t itself

Question 2

In a dynamic regression model, a lagged target variable is included mainly to represent what?

A. Historical temporal dependence

B. A guarantee of causal interpretation

C. Randomisation of observations

D. Removal of all forecast uncertainty

Correct Answer: A. Historical temporal dependence

Question 3

Which metric is expressed in the same units as the target and averages absolute errors?

A. MAPE

B. MAE

C. RMSE percentage

D. R-squared percentage

Correct Answer: B. MAE

Question 4

Why can MAPE be unsuitable for a series containing zero actual values?

A. It requires normally distributed errors

B. It ignores the forecast horizon

C. It divides by the actual value

D. It is measured in target units

Correct Answer: C. It divides by the actual value

Question 5

Which event representation is most appropriate for a permanent level change after a policy intervention?

A. Pulse variable

B. Step variable

C. Random lag

D. Seasonal-naïve variable

Correct Answer: B. Step variable

Question 6

What is the main risk of recursive multi-step forecasting?

A. It cannot use lag features

B. It always requires separate models for every horizon

C. Prediction errors can accumulate through later steps

D. It prevents the use of external drivers

Correct Answer: C. Prediction errors can accumulate through later steps

Question 7

Which validation design is generally appropriate for time-series forecasting?

A. Randomly shuffle all observations before splitting

B. Use chronological rolling-origin evaluation

C. Train on the test period and evaluate on the past

D. Evaluate only on the training data

Correct Answer: B. Use chronological rolling-origin evaluation

Question 8

When is deep learning most defensible for forecasting?

A. When there is a small series with no historical variation

B. When a simple baseline already meets all requirements

C. When large, rich and complex data support the additional complexity

D. Whenever managers prefer advanced terminology

Correct Answer: C. When large, rich and complex data support the additional complexity

Question 9

Which is a responsible use of generative AI in forecasting?

A. Inventing missing historical observations

B. Producing unverified numerical forecasts as final results

C. Drafting documentation that is checked against the governed model

D. Uploading confidential customer data to an unknown service

Correct Answer: C. Drafting documentation that is checked against the governed model

Question 10

Why should a complex forecasting model be compared with a naïve baseline?

A. To ensure complexity produces useful improvement

B. To eliminate the need for validation

C. To guarantee causal conclusions

D. To make all forecast errors equal

Correct Answer: A. To ensure complexity produces useful improvement