Forecast Evaluation, Selection and Reconciliation

Introduction

Forecasting is not complete when a model produces a number. A forecast becomes useful only when its quality is assessed against an appropriate benchmark, its errors are understood in operational terms, and the resulting forecasts are consistent with the decisions and structures of the organisation. In a supply chain, for example, a statistically accurate forecast may still be commercially poor if it systematically underestimates demand for an item with severe stock-out consequences. Similarly, forecasts produced independently for stores, regions and the national market may not add up, creating contradictory plans.

This chapter develops an advanced framework for evaluating, selecting, combining and reconciling forecasts. It focuses on rolling-origin validation, accuracy and bias measures, service-oriented metrics, forecast value added, forecast combination, hierarchical and grouped reconciliation, and intermittent-demand forecasting. The central principle is that forecast evaluation must reflect the decision context: the relevant horizon, loss function, data frequency, constraints and consequences of error.

Learning objectives

By the end of this chapter, the reader should be able to:

1. Design rolling-origin and time-series cross-validation schemes.
2. Calculate and interpret accuracy, scale-free, bias and service-oriented measures.
3. Select models without confusing in-sample fit with out-of-sample performance.
4. Diagnose whether human or process interventions add forecast value.
5. Combine forecasts using principled weighting methods.
6. Explain hierarchical, grouped and temporal reconciliation.
7. Select suitable methods for intermittent demand.
8. Evaluate forecast systems using statistical, operational and governance criteria.

1. Forecast evaluation as a decision problem

Let actual demand at time t be yₜ and the forecast made for that period be ŷₜ|origin, where the origin identifies when the forecast was produced. The forecast error is:

A positive error means that demand exceeded the forecast; this is an underforecast. A negative error indicates an overforecast. The sign convention must be stated because some organisations define error as forecast minus actual.

Forecast evaluation has four related purposes:

• Description: How large and variable have errors been?
• Comparison: Which candidate model performs better under a common test design?
• Diagnosis: Are errors biased, autocorrelated, seasonal or related to demand size?
• Decision support: What forecast is best for inventory, staffing, capacity or revenue decisions?

A forecast evaluation system should specify the forecast horizon, evaluation window, aggregation level, data revisions, treatment of missing observations, outliers and the loss function. Comparing one model on a one-step horizon with another on a twelve-month horizon is not a valid model comparison.

Training, validation and test periods

A time series cannot normally be randomly shuffled because doing so allows information from the future to influence the past. A chronological design is required:

• Training period: data used to estimate model parameters.
• Validation period: data used to tune choices such as model class or hyperparameters.
• Test period: a final, untouched period used to estimate likely live performance.

In operational systems, a single holdout period may be unstable if it contains an unusual promotion or disruption. Rolling-origin validation addresses this limitation by evaluating forecasts across multiple historical origins.

2. Rolling-origin validation

In rolling-origin evaluation, the model is repeatedly fitted using information available at a particular origin and then tested on future observations. Suppose origins are t₁, t₂, …, tₖ and the evaluation horizon is h. At origin tᵢ, only observations through tᵢ are available. A forecast is generated for tᵢ+h and compared with the actual value.

Two common designs are:

1. Expanding window: the training sample grows as each new observation is added.
2. Sliding window: the training sample has fixed length and moves through time.

An expanding window is appropriate when older observations remain relevant. A sliding window is useful when demand relationships change and distant history is less informative.

Figure: Each origin uses only past information and tests the next forecast period.

For multiple horizons, calculate errors separately for h = 1, 2, …, H. Aggregating all horizons into one score can conceal deterioration at longer lead times. A weighted score may be used when some horizons matter more operationally, but the weights should be documented before examining results.

Avoiding leakage

Leakage occurs when information unavailable at forecast time enters model estimation or feature construction. Examples include using revised sales data, future stock-out flags, promotional information known only retrospectively, or a normalisation parameter calculated from the entire dataset. Leakage produces deceptively strong validation results and is particularly common in machine-learning pipelines.

3. Accuracy, scale-free and bias measures

No single metric is universally best. Each expresses a different loss function.

Mean absolute error and root mean squared error

For n forecast errors:

MAE is expressed in the units of demand and gives equal weight to each absolute error. RMSE is also in demand units but penalises large errors more heavily because errors are squared. RMSE is useful when occasional large failures are especially costly, but it can be dominated by outliers.

Percentage and scaled measures

Mean absolute percentage error is:

MAPE is undefined when yₜ = 0 and becomes unstable when actual demand is close to zero. It is therefore unsuitable for many intermittent-demand series.

Weighted absolute percentage error is:

WAPE is well defined when the denominator is positive and weights periods by actual volume. It can, however, allow high-volume items to dominate a portfolio score.

The mean absolute scaled error compares a model with a naïve benchmark:

Here m is the seasonal period; m = 1 for a non-seasonal naïve benchmark. MASE is dimensionless. A value below 1 indicates improvement over the corresponding seasonal naïve method, provided the scaling denominator is non-zero and calculated from an appropriate training sample.

Bias and forecast control

Mean error measures signed bias:

A negative ME means systematic overforecasting under the convention eₜ = yₜ − ŷₜ. Mean percentage error can be misleading near zero and should not be used as the sole bias measure.

Cumulative forecast error is:

A tracking signal is often defined as:

where MAD is the mean absolute deviation, often equivalent to MAE over the monitoring window. Tracking-signal thresholds are heuristic, not universal statistical laws. A persistent signal should trigger diagnosis rather than automatic model replacement.

Probabilistic forecast evaluation

When a forecast includes a prediction interval, calibration and sharpness matter. A nominal 90% interval should contain approximately 90% of future observations over a sufficiently large and comparable sample. Calibration concerns coverage; sharpness concerns the narrowness of intervals subject to calibration. A narrow interval that frequently misses actual demand is not useful.

For quantile forecasts, the pinball loss for quantile α is:

where qα is the predicted α-quantile and I(·) is an indicator equal to 1 when its condition is true and 0 otherwise. This loss penalises underprediction and overprediction asymmetrically, making it suitable for service-level decisions.

4. Service-oriented evaluation

Statistical accuracy is a proxy for operational performance. Inventory decisions depend on lead-time demand, replenishment policy, safety stock, order quantities and target service levels. A forecast with slightly higher MAE may produce fewer stock-outs if it better represents tail risk or avoids systematic underforecasting.

Important service measures include:

• Cycle service level: probability of no stock-out during a replenishment cycle.
• Fill rate: proportion of total demand fulfilled immediately from stock.
• Ready rate: proportion of demand occasions satisfied immediately.
• Stock-out frequency and duration: operational measures of shortage occurrence.
• Bias during lead time: cumulative underforecasting over the period in which corrective action is difficult.

These measures should be calculated through an inventory simulation or actual replenishment records, not inferred solely from MAE. Observed sales may be censored by stock-outs: sales of zero do not necessarily mean demand was zero. Using sales without accounting for lost demand can reward forecasts that reproduce constrained supply rather than true customer demand.

A decision-oriented loss function can represent asymmetric costs:

where cᵤ is the cost per unit of underforecasting and cₒ is the cost per unit of overforecasting. The units of L are monetary cost when cᵤ and cₒ are monetary costs per unit. This simplified expression excludes nonlinear effects such as fixed ordering costs, capacity thresholds and lost goodwill; those require simulation or a richer cost model.

5. Forecast selection and forecast value added

Model selection should be based on out-of-sample performance, stability, interpretability and operational fit. A complex model should not be selected merely because its in-sample residual error is smaller. Information criteria such as AIC can help compare likelihood-based models, but they do not replace time-series validation when the business objective is forecast accuracy at a specified horizon.

A sensible benchmark set often includes:

• naïve and seasonal naïve forecasts;
• moving-average or exponential-smoothing methods;
• trend and seasonal statistical models;
• causal or regression models where predictors are genuinely available at forecast time;
• machine-learning models, evaluated under exactly the same origins and horizons.

Forecast value added

Forecast value added (FVA) asks whether each step in the forecasting process improves the final result relative to a defined baseline. If the baseline forecast has loss L₀ and the revised forecast has loss L₁, an additive value measure is:

Positive FVA indicates improvement when lower loss is better. Percentage improvement may be reported as 100 × (L₀ − L₁)/L₀ when L₀ > 0.

FVA analysis should compare forecasts generated at the same time, for the same horizon and using the same actual outcomes. It should be segmented by product, market, horizon and intervention type. Human overrides may add value for known promotions, product launches or supply disruptions, but routinely changing statistically generated forecasts without evidence often adds noise. Governance should record the reason, author, timestamp and expected effect of each override.

6. Combining forecasts

Different models capture different aspects of uncertainty. Combining forecasts can improve robustness when model errors are imperfectly correlated.

A weighted combination is:

where ŷₜᶜ is the combined forecast, ŷₜⱼ is forecast j, wⱼ is its weight and m is the number of forecasts. Non-negative weights are common because they improve interpretability, although constrained or unconstrained regression combinations may be appropriate in some settings.

Equal weighting is a strong benchmark because estimated weights can be unstable, especially with few validation observations and highly correlated models. Data-driven weights may be estimated by minimising validation loss, but the estimation must occur within the training or validation process; using the final test period to estimate weights invalidates the test.

For unbiased forecasts with error variance σⱼ² and covariance matrix Σ, the variance of the combined error is:

Thus, a model with the smallest individual variance need not be the best component if its errors are highly correlated with the other forecasts. The covariance structure is a central reason combination can work.

7. Hierarchical, grouped and temporal reconciliation

The coherence problem

A hierarchy may contain total company demand, regional demand and store-level demand. A coherent set of forecasts must satisfy the aggregation relationships. If bottom-level forecasts are collected in a vector b and the summing matrix is S, all-level observations can be represented as:

Base forecasts produced independently are denoted ŷ. They may be statistically reasonable but incoherent. Reconciliation transforms them into forecasts ỹ that satisfy:

for some reconciled bottom-level vector ĝ.

Main approaches

• Bottom-up: forecast the lowest level and aggregate upward. It preserves detailed forecasts but can be noisy for sparse series.
• Top-down: forecast the aggregate and allocate it using proportions. It may lose local information and requires stable allocation proportions.
• Middle-out: forecast at an intermediate level, then aggregate upward and disaggregate downward.
• OLS reconciliation: projects base forecasts onto the coherent space using ordinary least squares assumptions.
• Weighted least squares: accounts for unequal forecast-error variances.
• MinT (minimum trace): chooses a reconciliation transformation intended to minimise the total variance of reconciled forecast errors, using an estimate of the base-error covariance matrix.

In general, reconciliation can be written as:

where G is a reconciliation matrix estimated under the selected method. The dimensions of S, G and ŷ must conform, and the resulting ỹ must satisfy the aggregation constraints. Reconciliation does not guarantee lower error at every node; it trades local performance against coherence and can improve portfolio-level decisions.

Grouped structures differ from strict hierarchies because series can be aggregated simultaneously by region, product category, channel or customer segment. A product may belong to one category but be sold through several channels, creating overlapping aggregation constraints. Reconciliation must then respect all valid linear relationships rather than assuming a single tree.

Temporal hierarchies aggregate time at multiple frequencies, such as hourly, daily, weekly and monthly. Forecasts reconciled across temporal levels can support decisions made at different frequencies and reduce contradictions between, for example, a monthly plan and the sum of weekly plans.

8. Intermittent-demand forecasting

Intermittent demand contains many zero observations and occasional positive demands. It is common for spare parts, maintenance components and slow-moving inventory. Standard percentage measures and ordinary seasonal models may perform poorly because zeros dominate the sample.

A useful decomposition represents demand as an occurrence process and a size process. Let p be the probability of a non-zero demand in a period and μ be the mean size conditional on a non-zero demand. Under a simple independence assumption, expected period demand is:

This identity is descriptive; it does not by itself provide a complete forecasting method.

Croston-type methods separately update the estimated interval between non-zero demands and the non-zero demand size, then divide the size estimate by the interval estimate. Variants such as SBA and TSB address known biases or changing demand occurrence. The choice should depend on demand pattern, horizon and inventory policy. For probabilistic planning, a two-part model can estimate occurrence probability and conditional size distribution, then simulate lead-time demand.

Evaluation should use MAE, MASE, scaled measures, quantile loss, bias and inventory outcomes. MAPE should generally be avoided. Intermittent series also require careful aggregation: aggregating several sparse items can produce a less intermittent series, while disaggregation can reintroduce zeros and increase uncertainty.

9. Illustrative case: Northstar Components

Background

Northstar Components, an illustrative industrial distributor, supplies 2,400 spare parts through three regional warehouses. The planning team uses statistical forecasts, sales-manager overrides and a monthly executive forecast. It observes frequent stock-outs for high-criticality parts and excess inventory for slow movers. Forecasts at warehouse, region and national levels do not reconcile.

Problem

The company compares models using MAPE over a single six-month holdout. This favours forecasts for high-volume products, treats zero-demand items badly and provides no evidence about lead-time service. Managers also override forecasts without recording reasons.

Analysis

The analytics team creates twelve monthly rolling origins with horizons of one, three and six months. It compares seasonal naïve, exponential smoothing, a causal promotion model and an equal-weight combination. Evaluation uses MAE, MASE, ME, weighted absolute percentage error and simulated fill rate. Stock-out periods are flagged, and lost demand is estimated from substitution and back-order records.

The causal model performs best during planned promotions but is no better in ordinary periods. The combination is more stable across products. For critical parts, the service simulation shows that a mildly underbiased forecast produces unacceptable stock-outs; an upper quantile forecast is selected for replenishment while the median forecast is retained for financial planning.

For the product-region hierarchy, Northstar generates base forecasts at all levels and applies covariance-informed reconciliation. It then reconciles weekly forecasts with the monthly sales-and-operations plan. Slow-moving parts are segmented using demand occurrence and interval statistics; Croston-type candidates are compared with a probabilistic two-part model.

Approach and outcome

After three months of controlled deployment, Northstar reports lower MAE at the national level, improved coherence across planning reports and a reduction in critical-part stock-out frequency. Total inventory does not fall uniformly: safety stock rises for a small group of high-cost, high-criticality parts and falls for several items previously inflated by manual overrides. FVA monitoring shows that overrides add value mainly for documented promotions and engineering shutdowns.

Learning points

1. A single metric and holdout period cannot represent the forecasting problem.
2. Service measures must be linked to inventory simulation and business criticality.
3. Reconciliation improves planning consistency but does not eliminate uncertainty.
4. Interventions should be logged and evaluated as experiments.
5. A portfolio of models and a stable combination can be more reliable than a single champion model.

10. Implementation, governance and contemporary practice

A production evaluation framework should maintain a forecast archive containing forecast origin, horizon, model version, inputs, overrides and actual outcomes. Without an immutable archive, retrospective evaluation is vulnerable to revision bias.

Model governance should address:

• data lineage and access controls;
• reproducible feature construction;
• versioning of models and reconciliation matrices;
• monitoring for drift, bias and changing error distributions;
• human accountability for overrides;
• privacy and fairness where forecasts affect customers, workers or suppliers;
• explainability appropriate to the decision risk.

Artificial intelligence and machine learning can improve forecasting when they exploit valid exogenous information, nonlinearities or cross-series relationships. They do not remove the need for temporal validation, leakage control, calibration, reconciliation or service evaluation. A highly accurate black-box model that cannot be maintained, audited or reconciled may be inferior to a slightly less accurate but governable alternative.

Best practice is therefore to use a layered process: establish naïve benchmarks; design leakage-free rolling validation; evaluate several error and service measures; inspect bias and residual structure; compare combinations; reconcile related forecasts; test business outcomes; and monitor live performance with predefined intervention rules.

Chapter Summary

Forecast evaluation is a structured decision process rather than a search for one universally best metric. Rolling-origin validation respects temporal order and reveals performance across origins and horizons. MAE and RMSE measure error magnitude, while MASE and WAPE provide useful scale-free comparisons under appropriate conditions. MAPE is unsafe when actual demand is zero or near zero. Bias, cumulative error and tracking signals identify systematic over- or underforecasting, but should be interpreted diagnostically.

Operational evaluation extends beyond statistical accuracy to fill rate, stock-outs, lead-time performance and asymmetric costs. Forecast value added tests whether human or process interventions genuinely improve a baseline. Forecast combination can improve robustness because model errors are not perfectly correlated. Hierarchical, grouped and temporal reconciliation produces coherent forecasts that respect organisational constraints. Intermittent demand requires specialised methods, suitable metrics and careful treatment of zero observations. Effective forecasting systems combine statistical discipline, operational simulation, governance, documentation and continuous monitoring.

Key Terms

Rolling-origin validation: Repeated time-ordered evaluation using several forecast origins.

Expanding window: A training window that grows as new observations become available.

Sliding window: A fixed-length training window that moves through time.

Forecast error: The difference between actual demand and forecast under a stated sign convention.

MAE: Mean absolute error, the average absolute forecast error.

RMSE: Root mean squared error, which penalises large errors more strongly.

MAPE: Mean absolute percentage error; unstable or undefined when actuals are zero or small.

WAPE: Weighted absolute percentage error based on total absolute error divided by total actual volume.

MASE: Mean absolute scaled error relative to a naïve benchmark.

Bias: Systematic tendency to overforecast or underforecast.

Tracking signal: A heuristic ratio of cumulative error to mean absolute deviation.

Pinball loss: An asymmetric loss function for quantile forecasts.

Forecast value added: Improvement in forecast loss attributable to a process or intervention.

Forecast combination: A weighted aggregation of forecasts from multiple models.

Coherence: Satisfaction of specified aggregation relationships among forecasts.

Reconciliation: Transformation of base forecasts into coherent forecasts.

Summing matrix: A matrix that maps bottom-level series to all levels of an aggregation structure.

Intermittent demand: Demand with many zero periods and occasional positive observations.

Calibration: Agreement between stated probabilistic coverage and observed coverage.

Forecast leakage: Use of information in validation that would not have been available at forecast time.

Review Questions

Short Answer Questions

1. Why is random shuffling generally inappropriate for time-series cross-validation?
2. Distinguish MAE from RMSE.
3. Under what conditions can MAPE be misleading?
4. What does forecast coherence mean?
5. Why should forecast overrides be recorded?

Long Answer Questions

1. Explain how an organisation would design an expanding-window rolling-origin evaluation for one-step and multi-step forecasts.
2. Compare accuracy, bias and service-oriented measures, including their advantages and limitations.
3. Explain the logic of forecast value added and propose a governance process for measuring it.
4. Compare bottom-up, top-down, middle-out and covariance-informed reconciliation.
5. Discuss why intermittent demand requires specialised evaluation and forecasting methods.

Analytical / Application Questions

1. A retailer has zero sales in several periods because of stock-outs. Design an evaluation plan that distinguishes low demand from censored demand and identifies the operational consequences of forecast error.
2. Three models have similar MAE but different error correlations. Explain how you would decide whether to combine them and what additional validation is required.
3. A company’s monthly national forecast, regional forecasts and store forecasts do not add up. Specify a reconciliation strategy, the data required, and the risks of evaluating only the national total.

Multiple Choice Questions with Answers

Question 1

Which validation design best prevents future information from entering model estimation?

A. Randomly shuffle all observations before splitting

B. Use chronological rolling origins

C. Evaluate only fitted values from the training sample

D. Select the model with the most parameters

Correct Answer: B. Use chronological rolling origins

Question 2

Under the convention eₜ = yₜ − ŷₜ, a negative mean error indicates:

A. Systematic underforecasting

B. Systematic overforecasting

C. Perfect calibration

D. Increasing seasonality

Correct Answer: B. Systematic overforecasting

Question 3

Which measure is generally most problematic when actual demand can equal zero?

A. MAE

B. RMSE

C. MAPE

D. ME

Correct Answer: C. MAPE

Question 4

If a forecast has MAE 8 and the benchmark scaling MAE is 10, its MASE is:

A. 0.8

B. 1.25

C. 2

D. 18

Correct Answer: A. 0.8

Question 5

Positive forecast value added under a loss-based definition means that:

A. The revised forecast has higher loss

B. The baseline and revised forecasts are identical

C. The revised forecast has lower loss than the baseline

D. The forecast has zero bias

Correct Answer: C. The revised forecast has lower loss than the baseline

Question 6

Why can equal-weight forecast combination be useful?

A. It guarantees zero error

B. It avoids dependence on estimated weights and can diversify model errors

C. It always outperforms the best component

D. It requires no validation data

Correct Answer: B. It avoids dependence on estimated weights and can diversify model errors

Question 7

A coherent hierarchical forecast must:

A. Use only the top-level model

B. Have identical errors at every level

C. Satisfy the specified aggregation relationships

D. Contain no uncertainty intervals

Correct Answer: C. Satisfy the specified aggregation relationships

Question 8

Which demand pattern is most associated with Croston-type methods?

A. Continuous demand with strong daily seasonality

B. Intermittent demand with many zero observations

C. Constant demand with no variation

D. Only rapidly growing demand

Correct Answer: B. Intermittent demand with many zero observations

Question 9

For a 90% prediction interval, calibration primarily concerns whether:

A. The interval is always as narrow as possible

B. Approximately 90% of observations fall inside over a suitable sample

C. The point forecast equals the median

D. The model has the lowest in-sample error

Correct Answer: B. Approximately 90% of observations fall inside over a suitable sample

Question 10

In a forecast combination, error covariance matters because:

A. It determines whether errors diversify or reinforce one another

B. It makes actual demand unnecessary

C. It eliminates the need for a benchmark

D. It converts MAE into MAPE

Correct Answer: A. It determines whether errors diversify or reinforce one another