Forecast Evaluation and Experimental Design

Introduction

Forecasting is not complete when a model produces a future value. A forecast is useful only when its quality can be assessed honestly and when competing models can be compared under conditions that resemble deployment. Forecast evaluation therefore concerns more than selecting a convenient accuracy statistic. It requires an experimental design that respects temporal order, prevents information leakage, represents the forecast horizon of interest, includes credible benchmarks, and recognises that statistical error and business cost are not always identical.

In time-series forecasting, the central experimental difficulty is that observations are ordered in time. A randomly selected training and test split can allow information from the future to influence model selection or feature construction. The resulting estimate may be optimistically biased. This chapter develops rolling-origin evaluation and time-series cross-validation, explains important error measures, and shows how uncertainty, bias, business-weighted loss, and operational constraints should influence model selection.

Learning objectives

By the end of this chapter, students should be able to:

1. Explain why ordinary random cross-validation is generally inappropriate for time series.
2. Design rolling-origin and expanding- or sliding-window evaluation procedures.
3. Select and interpret MAE, RMSE, MASE, RMSSE, sMAPE, bias, and business-weighted loss.
4. Construct and assess naïve, seasonal-naïve, and other benchmark forecasts.
5. Identify leakage in data preparation, feature engineering, tuning, and model comparison.
6. Compare models while accounting for sampling uncertainty, forecast horizons, and business consequences.
7. Communicate an evaluation protocol that is reproducible and defensible.

1. Forecast evaluation as a decision problem

Let y_t denote the observed value at time t, and let \hat{y}_t|T denote a forecast made using information available through time T. For a forecast horizon h, the target is y_T+h and the forecast error is:

The sign convention matters. A positive error means under-forecasting; a negative error means over-forecasting. Evaluation should be conducted on observations that were not used to estimate the particular forecast being assessed.

A useful evaluation design answers five questions:

• What forecast origin dates will be used?
• Which observations are available at each origin?
• What horizon or horizons matter operationally?
• Which loss function represents the decision being supported?
• How will uncertainty in the comparison be reported?

A model can be best under RMSE and inferior under MAE; it can be statistically accurate but commercially poor if stockouts are much more costly than excess inventory. Evaluation is therefore conditional on a purpose, a horizon, a data-generating environment, and a loss function.

2. Why random cross-validation fails for many time series

In ordinary cross-validation, observations are often shuffled before folds are formed. This is inappropriate when temporal dependence exists. A randomly selected training set may contain observations after the validation observation. Even if the response itself is not directly included, future information can enter through lag construction, rolling statistics, revised data, imputation, normalization, feature selection, or hyperparameter tuning.

The issue is not merely dependence between observations. It is also the direction of information flow. In deployment, a forecast made on 1 March cannot use sales recorded on 15 March. Evaluation must reproduce this information set.

Random splitting may be acceptable only under a carefully justified assumption that the observations are exchangeable for the intended task—for example, a genuinely cross-sectional problem with no meaningful temporal ordering. It should not be the default for forecasting.

3. Rolling-origin evaluation

Rolling-origin evaluation repeatedly simulates forecasting from historical dates. At each origin, the model is trained only on data available at that date, forecasts one or more future periods, and is then evaluated after the actual outcome becomes known.

Suppose the observed series is available through time T, the minimum training size is m, and the maximum forecast horizon is H. Let the origins be \tau₁,\tau₂,\ldots,\tau_K. At origin \tau_i, the model uses observations through \tau_i and generates forecasts for \tau_i+1 through \tau_i+H. The evaluation errors are:

Only pairs for which the actual value exists are included. If the operational requirement is a twelve-month forecast, the experiment should report twelve-month performance rather than only one-step performance.

Expanding-window evaluation

An expanding window retains all observations available before the origin. The training sets are:

This design is appropriate when older observations remain relevant and more data generally improve estimation. It is common for stable economic series and long-running demand histories.

Sliding-window evaluation

A sliding, or rolling, window uses only the most recent w observations:

It is useful when structural change, product substitution, inflationary shifts, or changing customer behaviour makes distant history less representative. The window length is itself a modelling choice and should be selected using validation data, not by inspecting the final test period.

Blocked and multi-step designs

For a fixed-horizon decision, one may evaluate non-overlapping future blocks. Overlapping horizons provide more error observations but create dependence among errors: forecasts from nearby origins can concern some of the same actual periods. This does not invalidate the evaluation, but it affects uncertainty calculations. Naively treating all errors as independent can produce confidence intervals that are too narrow.

A practical design often has three chronological parts:

1. Training period: estimates model parameters.
2. Validation period: selects model form, features, window length, or hyperparameters.
3. Final test period: used once for an unbiased estimate of the selected procedure.

Within the training and validation periods, rolling-origin resampling may be used. The final test period must remain untouched until decisions are frozen.

Figure: Expanding-window rolling-origin evaluation. The training set grows as the forecast origin advances.

4. Benchmark forecasts

A benchmark is a transparent reference against which a sophisticated model must demonstrate value. The simplest benchmark is the naïve forecast:

It forecasts every future period using the latest observation. For a seasonal series with period m, the seasonal-naïve forecast is:

The seasonal-naïve benchmark is often difficult to beat for strongly seasonal demand. Other useful benchmarks include a drift forecast, a historical mean, a last-value forecast for intermittent demand, and an existing organisational planning forecast.

Benchmarks serve three purposes. They identify whether a model adds predictive value, provide a scale for interpreting errors, and expose data or implementation problems. A complex model that cannot outperform a seasonal-naïve forecast under the relevant loss function should not be adopted without a strong operational justification.

5. Error measures and their interpretation

Let there be N evaluated forecast errors e₁,\ldots,e_N. The metric must be selected with attention to scale, outliers, zero values, aggregation, and the decision context.

Mean absolute error

MAE is expressed in the units of the target, such as units sold or dollars. It gives equal weight to absolute deviations and is less sensitive to extreme errors than RMSE. It is often easy to explain to managers: the average forecast misses by approximately MAE units.

Root mean squared error

RMSE has the same units as the target but penalises large errors more heavily. It is appropriate when large misses have disproportionately high consequences or when squared-error estimation is substantively justified. Because squaring magnifies outliers, it should be interpreted alongside MAE.

Bias and mean error

Forecast bias, often reported as mean error, is:

Under the error definition used here, positive bias indicates systematic under-forecasting and negative bias indicates systematic over-forecasting. Bias near zero does not imply accuracy: large positive and negative errors may cancel. Therefore bias should accompany, not replace, an absolute-error measure.

A useful supplementary measure is the mean absolute percentage error, but percentage measures become unstable or undefined when actual values are zero or close to zero. They should not be used automatically for intermittent or low-volume series.

Symmetric mean absolute percentage error

One common form of sMAPE is:

Here y_i is the actual value and \hat{y}_i is the forecast. The denominator must be nonzero; when both values are zero, the term requires an explicit convention, commonly assigning zero error. sMAPE is bounded under this definition between 0% and 200%, but it is not perfectly symmetric in all practical interpretations and can behave oddly near zero. It should not be treated as universally superior to MAE.

Mean absolute scaled error

MASE compares forecast errors with the average absolute error of a naïve benchmark calculated on the training data. For a non-seasonal series:

For seasonal period m, the seasonal scaling denominator is:

The numerator is the evaluation MAE and the denominator is computed from the relevant training observations. MASE is dimensionless and generally interpretable across series: values below 1 indicate improvement over the corresponding naïve scaling benchmark. The denominator must be nonzero; a constant training series requires a documented alternative treatment.

Root mean squared scaled error

RMSSE scales squared forecast errors using the mean squared change of the training series:

For seasonal data, replace one-period differences with y_t-y_t-m. RMSSE is dimensionless and retains RMSE’s sensitivity to large misses. It is particularly useful when evaluating many series with different units or scales. As with MASE, the scaling denominator must be calculated without using future evaluation observations.

Business-weighted loss

A business loss function assigns different costs to different errors. For example, under-forecasting may cause a stockout while over-forecasting causes holding cost. A simple asymmetric linear loss is:

where c_u is the cost per unit of under-forecasting and c_o is the cost per unit of over-forecasting. Expected business loss is the mean of L(e_i). This formula assumes that the error-to-cost relationship is linear over the relevant range and that the sign convention is actual minus forecast. Real decisions may require nonlinear costs, service-level penalties, capacity constraints, or lead-time effects.

A model should not be declared best solely because it minimises a generic metric. The primary metric should reflect the decision, while secondary metrics reveal trade-offs such as bias, volatility, and extreme misses.

6. Worked example: comparing two forecasting procedures

Consider six holdout errors, measured in units, for two models:

For Model A, the absolute errors sum to 11 and squared errors sum to 23. Thus:

Its bias is (2-2+1-1+3-2)/6=0.167. For Model B, absolute errors sum to 11 and squared errors sum to 31:

The bias of Model B is (0+3-1+2-4+1)/6=0.167. MAE and bias are identical, but Model B has the larger RMSE because it contains a larger extreme error. If large misses are costly, Model A is preferable. If only average absolute deviation matters, the models tie; further evidence is required.

Suppose under-forecasting costs 3 monetary units per unit and over-forecasting costs 1. Since positive errors are under-forecasts, Model A’s business loss is 3(2+1+3)+1(2+1+2)=25, or 4.167 per period. Model B’s loss is 3(3+2+1)+1(4)=22, or 3.667 per period. Thus Model B is preferred under this asymmetric cost despite its worse RMSE. This illustrates why metric choice is a decision-design issue.

7. Leakage prevention

Data leakage occurs when information unavailable at forecast time influences model estimation, feature creation, model selection, or reported performance. Common examples include:

• scaling or imputing the complete dataset before splitting;
• constructing a rolling average that accidentally includes the current or future target;
• using revised economic data when the historical forecast would have used an earlier vintage;
• selecting variables after examining the final test period;
• tuning hyperparameters on the test period;
• using future promotions, realised prices, or confirmed cancellations as if they were known at the origin;
• calculating MASE or RMSSE denominators using evaluation-period observations;
• allowing target-derived aggregates to cross entity or time boundaries.

A safe pipeline fits every learned transformation inside each training fold. At origin \tau, lagged features may use values through \tau, while a future-known calendar variable may be used if it was genuinely available at that time. Features must be classified as either known in advance, forecast separately, or unavailable.

Leakage also occurs through repeated experimentation. If many model variants are compared against one test set, the test set gradually becomes part of the selection process. A final holdout, nested time-series validation, or a fresh future period is needed when extensive tuning has occurred.

8. Accounting for uncertainty in model comparisons

A single average metric conceals variation across origins, products, and horizons. Evaluation should report the distribution of errors and, where feasible, uncertainty intervals for differences.

For two models, define the paired loss difference at evaluation case i as:

The mean difference is \bar d=N^-1\sum_i d_i. Pairing is important because both models are evaluated on the same origins and actual outcomes. A positive value favours Model B under the selected loss. Because time-series loss differences are often autocorrelated, ordinary independent-sample standard errors may be inappropriate. Block bootstrap methods, heteroskedasticity-and-autocorrelation-consistent methods, or forecast-comparison tests designed for dependent forecast errors may be used, provided their assumptions are examined.

Uncertainty can also be represented through quantiles of origin-level loss, prediction intervals, and performance by horizon. A model with a slightly better mean loss but severe instability in certain regimes may be operationally less attractive than a robust competitor. Statistical significance is not the same as practical significance: a tiny improvement may not justify additional cost, latency, interpretability loss, or maintenance burden.

Evaluation should be stratified when appropriate by:

• forecast horizon;
• product, region, or customer segment;
• high- and low-demand periods;
• ordinary versus promotion periods;
• regime or volatility level.

Aggregation requires care. A macro-average gives each series equal weight; a volume-weighted or revenue-weighted average gives greater influence to important series. The chosen aggregation must match the decision and be reported explicitly.

9. A disciplined experimental protocol

A defensible forecasting experiment can follow this sequence:

1. Define the decision. Specify the forecast horizon, update frequency, available information, and cost of errors.
2. Audit the data. Record timestamps, missingness, revisions, outliers, aggregation rules, and data availability times.
3. Choose a chronological evaluation design. Select expanding or sliding windows, origins, and horizons before viewing final results.
4. Implement benchmarks. Include naïve, seasonal-naïve, and any incumbent planning method.
5. Build leakage-safe pipelines. Fit preprocessing, feature selection, and hyperparameter tuning only within permitted training data.
6. Evaluate with primary and secondary metrics. Use a business-relevant loss plus measures such as MAE, RMSE, scaled error, and bias.
7. Examine error distributions. Inspect horizon-specific, segment-specific, and regime-specific results.
8. Quantify uncertainty. Use paired differences and methods suitable for dependent observations.
9. Select parsimoniously. Prefer the simplest model that delivers meaningful and stable improvement.
10. Lock the procedure and test once. Refit on permitted historical data, evaluate on the final untouched period, and document the result.

Reproducibility requires recording random seeds where applicable, software versions, data vintages, origin dates, feature availability rules, metric definitions, and all model-selection decisions.

10. Illustrative case: regional retail inventory forecasting

Background

A retailer forecasts weekly demand for 2,400 product–region combinations. The supply chain has a two-week replenishment lead time. Stockouts have a high service cost, while overstock creates holding and markdown costs. Demand is seasonal and promotional activity is known one week in advance but not two weeks in advance.

Problem

The analytics team proposes a gradient-boosted model that appears substantially better than the existing seasonal-naïve system under random five-fold cross-validation. Operations managers doubt the result because the model uses promotion and inventory features whose availability timing is unclear.

Analysis

An audit reveals that the random folds mix future promotional outcomes with earlier observations. In addition, a rolling demand feature was calculated using a centred window, which included future demand. The reported improvement is therefore contaminated by leakage.

The team redesigns the experiment. Origins are weekly, the minimum training history is 52 weeks, and forecasts are evaluated at horizons one and two. An expanding window is used for stable products; a 26-week sliding window is tested for products with evidence of recent structural change. The seasonal-naïve model is the primary benchmark.

The primary loss is asymmetric: under-forecasting is assigned a cost three times the unit cost of over-forecasting. MAE, RMSE, MASE, bias, and stockout rate are secondary measures. All transformations are fitted separately at each origin. Promotion information is used only at the horizon for which it was genuinely known.

Approach and outcome

The complex model no longer dominates on every measure. It improves two-week business-weighted loss for high-volume products and reduces under-forecasting bias, but offers little gain for intermittent low-volume items. The seasonal-naïve model remains competitive for stable seasonal products. A segmented policy is adopted: the boosted model is deployed for high-volume promotional products, while seasonal-naïve forecasts are retained elsewhere. Forecast intervals and an exception dashboard are added for unusually high predicted stockout risk.

Learning points

The case demonstrates that model evaluation is inseparable from information availability and business objectives. A sophisticated algorithm did not fail because machine learning was inappropriate; the initial experiment failed because it did not reproduce deployment. Segmentation, benchmarks, leakage controls, asymmetric loss, and operational review produced a more credible decision than a single overall accuracy ranking.

11. Limitations and contemporary practice

No evaluation design guarantees future performance. Structural breaks, new products, policy changes, supply disruptions, and changes in measurement can make historical validation unrepresentative. Backtesting should therefore be supplemented by stress scenarios and monitoring after deployment.

Probabilistic forecasts require proper scoring rules, such as interval scores or the continuous ranked probability score, rather than evaluation of point forecasts alone. When decisions depend on uncertainty—for example, safety stock or staffing—calibration and sharpness should be assessed.

Automated machine-learning systems can make leakage easier to overlook because feature-generation and tuning pipelines may be opaque. Governance should require data lineage, time-of-availability documentation, reproducible experiments, human review of business loss functions, and monitoring for subgroup or regional degradation. Forecasts used in employment, credit, healthcare, or public services may also raise fairness, privacy, and accountability concerns. A lower numerical loss does not justify a system that cannot be explained, audited, or safely operated.

Chapter Summary

Forecast evaluation must simulate the information flow of real forecasting. Rolling-origin evaluation preserves temporal order and can use expanding or sliding training windows. The forecast horizon, origin schedule, benchmark, loss function, and final test policy should be specified before model comparison.

MAE measures average absolute error; RMSE emphasises large errors; bias detects systematic under- or over-forecasting; sMAPE provides a percentage-style measure but has zero and near-zero limitations; MASE and RMSSE scale errors against naïve in-sample differences; and business-weighted loss connects forecast performance to operational consequences. Naïve and seasonal-naïve forecasts are essential reference points.

Leakage can arise through preprocessing, feature construction, data revisions, tuning, or inappropriate scaling denominators. It is prevented by fitting transformations within each training fold and respecting data availability. Model comparisons should use paired losses, uncertainty-aware methods, horizon and segment analysis, and practical significance. The best forecasting procedure is not necessarily the most complex or the one with the lowest generic error; it is the procedure that delivers stable, leakage-free, decision-relevant performance.

Key Terms

Forecast origin: The time at which a forecast is produced.

Forecast horizon: The number of periods between the origin and the target observation.

Rolling-origin evaluation: Repeated historical simulation of forecasting from successive origins.

Expanding window: A training set that retains all observations available before each origin.

Sliding window: A training set containing only the most recent fixed number of observations.

Benchmark forecast: A transparent reference forecast used to judge added model value.

Naïve forecast: A forecast that extends the latest observed value into the future.

Seasonal-naïve forecast: A forecast that repeats the value from the corresponding previous season.

Forecast error: Actual value minus forecast under the convention used in this chapter.

MAE: Mean absolute error, the average absolute forecast deviation.

RMSE: Root mean squared error, which gives extra weight to large errors.

Bias: Mean signed forecast error.

sMAPE: Symmetric mean absolute percentage error.

MASE: Mean absolute scaled error relative to naïve in-sample differences.

RMSSE: Root mean squared scaled error relative to naïve in-sample squared differences.

Business-weighted loss: A loss function that assigns costs according to operational consequences.

Data leakage: Use of information in evaluation or modelling that would not be available at forecast time.

Forecast comparison uncertainty: Sampling or temporal uncertainty in the estimated difference between forecasting procedures.

Data vintage: The version of a dataset available at a particular historical time.

Review Questions

Short Answer Questions

1. Why is random cross-validation generally inappropriate for time-series forecasting?
2. Distinguish an expanding window from a sliding window.
3. What does a positive bias indicate when error is defined as actual minus forecast?
4. Why should naïve and seasonal-naïve forecasts be included in an evaluation?
5. State one important limitation of sMAPE.

Long Answer Questions

1. Explain how to design a rolling-origin experiment for a monthly demand series requiring forecasts six months ahead.
2. Compare MAE, RMSE, MASE, and RMSSE in terms of scale, sensitivity to outliers, and cross-series comparability.
3. Explain how data leakage can enter a forecasting pipeline through preprocessing and feature engineering.
4. Discuss why a model with the best average statistical metric may not be the best operational model.
5. Explain how dependent forecast errors affect uncertainty in model comparisons and identify suitable responses.

Analytical / Application Questions

1. A retailer has equal unit costs for over- and under-forecasting but a very high penalty for errors exceeding 100 units. Which evaluation metrics and experimental summaries should be prioritised, and why?
2. Design a leakage-safe validation procedure for a model using lagged sales, price, promotions, and a rolling average. Specify which variables can be used at one- and two-week horizons.
3. Two models have equal MAE, but one has lower RMSE and the other has lower under-forecasting cost. Recommend how a supply-chain manager should choose between them, including additional evidence that should be collected.

Multiple Choice Questions with Answers

Question 1

Which evaluation design most directly preserves the information flow of forecasting?

A. Randomly shuffled k-fold cross-validation

B. Rolling-origin evaluation

C. Evaluating only fitted values

D. Selecting the model with the largest training R²

Correct Answer: B. Rolling-origin evaluation

Question 2

Under the chapter’s error convention, a positive forecast error indicates:

A. Over-forecasting

B. Under-forecasting

C. Zero demand

D. A seasonal adjustment

Correct Answer: B. Under-forecasting

Question 3

Which metric penalises large errors more strongly because errors are squared before averaging?

A. MAE

B. Bias

C. RMSE

D. sMAPE

Correct Answer: C. RMSE

Question 4

A MASE value below 1 generally indicates that the forecast:

A. Has zero bias

B. Improves on the corresponding naïve scaling benchmark

C. Has lower RMSE than every competing model

D. Uses a seasonal model

Correct Answer: B. Improves on the corresponding naïve scaling benchmark

Question 5

Which action is an example of data leakage?

A. Using a lagged value available at the forecast origin

B. Evaluating a seasonal-naïve benchmark

C. Fitting a scaler separately within each training fold

D. Computing a centred rolling feature that includes future demand

Correct Answer: D. Computing a centred rolling feature that includes future demand

Question 6

Why can two models with equal MAE have different RMSE values?

A. RMSE ignores all large errors

B. RMSE is calculated only on training observations

C. RMSE gives greater influence to the sizes of large errors

D. MAE is always a percentage measure

Correct Answer: C. RMSE gives greater influence to the sizes of large errors

Question 7

For a seasonal series with period m, a seasonal-naïve forecast uses:

A. The overall sample mean

B. The observation from m periods earlier

C. The latest observation for every horizon

D. The largest historical observation

Correct Answer: B. The observation from m periods earlier

Question 8

If under-forecasting costs more than over-forecasting, the evaluation should include:

A. Only an unweighted RMSE

B. An asymmetric business loss

C. Only the training error

D. A random train-test split

Correct Answer: B. An asymmetric business loss

Question 9

Why should uncertainty in forecast-model comparisons account for paired errors?

A. Both models are evaluated on the same origins and actual outcomes

B. Paired errors eliminate all temporal dependence

C. Paired errors make benchmarks unnecessary

D. Pairing converts MAE into RMSE

Correct Answer: A. Both models are evaluated on the same origins and actual outcomes

Question 10

What is the main purpose of a final untouched test period?

A. To tune hyperparameters repeatedly

B. To increase the training sample before validation

C. To provide an honest estimate after model-selection decisions are frozen

D. To calculate future values of the target

Correct Answer: C. To provide an honest estimate after model-selection decisions are frozen