Forecast Accuracy, Validation and Model Selection

Introduction

Forecasting is useful only when its outputs are sufficiently accurate, reliable, timely, and economically valuable for decisions. In logistics, forecast quality affects procurement, replenishment, inventory positioning, transport capacity, warehouse labour, production scheduling, and customer-service performance. A model that has the lowest average error may nevertheless be unsuitable if it systematically under-forecasts peak demand or performs poorly for high-priority customers.

Forecast evaluation therefore has two purposes. First, it measures how closely forecasts match observations. Second, it determines whether a forecasting process improves operational and financial outcomes relative to credible alternatives. This chapter develops methods for holdout testing and rolling-origin validation, explains major accuracy metrics, and presents a disciplined approach to benchmarking, model selection, forecast value added, and economic evaluation.

Learning Objectives

By the end of this chapter, the reader should be able to:

• design chronological holdout and rolling-origin validation schemes;
• calculate and interpret bias, MAE, RMSE, MAPE, WAPE, MASE, and service-oriented metrics;
• select metrics appropriate to demand characteristics and managerial decisions;
• benchmark competing models against simple and operationally relevant baselines;
• diagnose overfitting, instability, data leakage, and segment-specific performance;
• assess forecast value added and translate forecast improvements into economic impact.

1. Forecast errors and the evaluation problem

Let y_t denote actual demand in period t, and let \hat{y}_t|t-h denote a forecast made at time t-h for period t. The forecast error is:

A positive error means that actual demand exceeded the forecast; the system under-forecast. A negative error means that the system over-forecast. This sign convention must be stated because some organisations define error as forecast minus actual.

Evaluation must reproduce the information available when the forecast would actually have been produced. Using future observations, future promotions, revised master data, or a randomly selected training sample can create leakage and produce unrealistically good results. In time-series forecasting, chronological order is part of the data-generating process.

Forecast quality has several dimensions:

1. Accuracy: closeness of forecasts to actual demand.
2. Bias: systematic over- or under-forecasting.
3. Stability: consistency across time, products, locations, and demand regimes.
4. Calibration: whether predicted uncertainty corresponds to observed outcomes.
5. Operational usefulness: effect on inventory, capacity, service, and cost.
6. Timeliness and usability: whether forecasts arrive in time and can be acted upon.

No single metric captures all these dimensions.

2. Holdout design and rolling-origin validation

2.1 Chronological holdout testing

A simple evaluation divides a time series into a training period and a test period. The model is fitted using observations from the training period and evaluated on later observations that were not used in fitting.

For example, a 36-month demand history might use months 1–27 for training and months 28–36 for testing. The test horizon should correspond to the operational decision horizon. A replenishment system with a three-week effective lead time should not be evaluated only at a one-day horizon.

A holdout is useful because it is transparent and computationally inexpensive. However, one particular test window may be atypical. A promotion-heavy test period can unfairly penalise a model that was not given promotion information; a stable test period can conceal poor performance during disruption.

2.2 Rolling-origin validation

Rolling-origin, or walk-forward, validation repeats the forecast exercise at several historical origins. At each origin, only information available by that date is used. The model may be refitted as the origin advances, and forecasts are generated for one or more future horizons.

For origins o₁, o₂, ..., o_K and horizon h, the evaluation set consists of forecasts \hat{y}_ok+h|ok. A multi-step design evaluates each horizon separately, because one-period-ahead accuracy and twelve-period-ahead accuracy can differ substantially.

A practical design is:

1. reserve a final untouched test period for confirmation;
2. use earlier observations for rolling-origin model selection;
3. specify the forecast horizon and update frequency before analysing results;
4. refit models only with information that would have been available at each origin;
5. calculate metrics by origin, horizon, product, and aggregate portfolio;
6. investigate whether conclusions are robust across time and demand segments.

Expanding-window validation retains all historical observations as the origin advances. A sliding-window design uses only the most recent fixed number of observations, which can be preferable when relationships change or old data are no longer representative.

Figure: In rolling-origin validation, the training window advances while each test horizon remains chronologically after its corresponding training data.

2.3 Choosing validation windows

The validation window should contain enough observations to represent the relevant patterns: seasonality, promotions, replenishment cycles, and disruption periods. If annual seasonality is important, a window shorter than one year may be inadequate. Conversely, a very long window may dilute recent structural changes.

Validation must also respect grouped data. For a network forecast, information about one warehouse may legitimately be available when forecasting another warehouse, but information derived from the future demand of the target warehouse is not. Hierarchical aggregation, product substitutions, and intermittent demand require explicit rules to avoid hidden leakage.

3. Accuracy metrics

Assume that evaluation contains n forecast-observation pairs, errors are e_t = y_t - \hat{y}_t, and, unless stated otherwise, actual demand is non-negative.

3.1 Mean error and bias

Mean error is:

It has the same units as demand. A positive ME indicates aggregate under-forecasting; a negative ME indicates aggregate over-forecasting. Mean error can be close to zero even when forecasts are individually poor because positive and negative errors cancel.

A scaled bias percentage is sometimes reported as:

This denominator is meaningful only when total actual demand is positive. Bias is a directional diagnostic, not a complete accuracy measure.

3.2 Mean absolute error

MAE measures average absolute error in physical units, such as units, cartons, or tonnes. It is easy to interpret and gives each observation a linear penalty. It is appropriate when an error of 20 units is considered twice as serious as an error of 10 units.

3.3 Root mean squared error

RMSE has the same units as demand but penalises large errors more heavily than MAE. It is useful when occasional severe misses create disproportionate operational consequences, such as stockouts during a major launch. RMSE is sensitive to outliers and should not be interpreted as a robust measure in highly disrupted data without investigation.

3.4 Mean absolute percentage error

The usual MAPE is:

Here n^* is the number of observations with positive actual demand. MAPE expresses error as a percentage of actual demand. It is undefined when y_t=0 and can become extremely large when actual demand is close to zero. Omitting zero-demand periods changes the population being evaluated and must be reported.

MAPE is also asymmetric in practical interpretation: the same absolute miss can produce very different percentages depending on the denominator. It is therefore unsuitable as the sole metric for intermittent or low-volume demand.

3.5 Weighted absolute percentage error

WAPE is:

WAPE weights periods by actual volume and is defined when total actual demand is positive. It is often useful for portfolio or network reporting because high-volume items contribute more than low-volume items. However, it can conceal poor performance on low-volume or strategically important items. WAPE is not the arithmetic average of item-level percentage errors.

3.6 Mean absolute scaled error

MASE compares forecast error with the error of a benchmark, commonly the in-sample one-step naïve forecast. For a seasonal series with period m, a suitable scaling denominator is the mean absolute seasonal difference in the training data:

Here N is the number of training observations used to calculate the scale and m is the seasonal period. MASE is dimensionless. A value below 1 means the evaluated forecasts have lower MAE than the selected seasonal-naïve benchmark; a value above 1 means they are worse. The scale denominator must be non-zero. The benchmark and scaling scheme must be documented, because changing them changes the interpretation.

3.7 Metric comparison

Metrics should be reported with the aggregation level and denominator. Micro-averaging pools all observations before calculation; macro-averaging gives each item or location equal weight. Neither is universally correct.

4. Service-oriented forecast evaluation

Accuracy metrics describe forecast errors; service metrics describe operational consequences. Important measures include:

• Cycle service level: proportion of replenishment cycles in which no stockout occurs.
• Fill rate: proportion of total demand fulfilled immediately from available stock.
• Item fill rate: proportion of demand lines fulfilled completely, depending on the organisation’s definition.
• Stockout rate: proportion of demand or demand occasions not served immediately.
• Backorder days: time between requested and fulfilled demand.
• On-time-in-full (OTIF): proportion of orders delivered by the promised date and in the promised quantity.

Definitions must be fixed before comparison. For example, cycle service level and fill rate are not interchangeable: one counts cycles, while the other weights shortages by demand quantity.

A forecast can have low MAE but poor service if its errors occur during lead-time demand peaks. Conversely, a model with slightly higher MAE may improve service by reducing under-forecasting in critical periods. Evaluation should therefore segment errors by lead-time bucket, priority class, promotion status, and stockout exposure.

When forecast distributions are available, service-oriented decisions can use prediction intervals or quantiles. A replenishment decision may require a high quantile of lead-time demand rather than the mean forecast. This links forecast evaluation to inventory policy rather than treating forecasting as an isolated statistical exercise.

5. Benchmarking and model selection

5.1 Baselines are essential

A complex model should be compared with credible simple benchmarks, including:

• last-value naïve forecast;
• seasonal-naïve forecast;
• moving average;
• current operational forecast;
• forecast generated by the existing planning process.

A model that cannot outperform a seasonal-naïve or incumbent forecast consistently should not be adopted merely because it is more sophisticated. Baselines also reveal whether apparent gains result from an easy test period or from genuine information.

5.2 Selection criteria

Model selection should consider:

1. performance on rolling-origin validation;
2. performance on the untouched final test set;
3. variation across products, locations, horizons, and time periods;
4. bias and service outcomes, not only average error;
5. computational cost, latency, interpretability, and maintainability;
6. robustness to missing data, promotions, new products, and disruptions;
7. economic impact after forecast-driven decisions are simulated.

Selecting the model with the smallest validation error can itself overfit the validation set when many models are tried. A final test set, nested validation, or a time-based confirmation period reduces this risk.

5.3 Statistical significance and practical significance

Small metric differences may be noise. Forecast errors are often autocorrelated and correlated across products, so ordinary independent-sample tests may be inappropriate. A researcher may use forecast comparison procedures such as a Diebold–Mariano test, provided assumptions, loss functions, horizons, and dependence structures are handled carefully. Statistical significance does not establish economic significance. A one-percent improvement may be operationally valuable for a large network, but not worth implementation cost for a small item group.

5.4 Ensemble and segmented selection

One model need not serve every series. Intermittent spare-parts demand may require a different method from stable fast-moving consumer goods. Segmented selection can classify items by volume, intermittency, lifecycle, or promotional intensity. Ensembles combine forecasts and can reduce model-specific risk, but they add governance and monitoring requirements. Weights should be estimated without using future test outcomes.

6. Forecast value added and economic impact

Forecast Value Added (FVA) asks whether each process step improves the forecast relative to a baseline. Let L(B) be the loss of the baseline and L(P) the loss after a process intervention, using the same observations and loss function. A percentage FVA can be written as:

Positive FVA indicates improvement; negative FVA indicates deterioration. The loss may be MAE, weighted absolute error, a service penalty, or an economic cost. FVA analysis should compare like with like: same horizon, demand population, information set, and evaluation period.

For economic evaluation, define a cost function. A simple under- and over-forecast loss is:

Here c_u is the unit cost of under-forecasting and c_o is the unit cost of over-forecasting. These costs may include emergency transport, lost margin, production disruption, holding cost, markdown, disposal, or customer penalties. The formula is a simplified decision model: real systems may include nonlinear capacity constraints, inventory carryover, substitution, and service-level penalties.

A broader net benefit is:

The new model should be evaluated through a decision simulation whenever forecasts influence inventory or capacity. Simply multiplying MAE reduction by a unit cost can be misleading because inventory policies, safety stock, lead time, and order constraints mediate the relationship between error and outcome.

7. Worked example

An illustrative product has five weekly actual demands and forecasts:

The sum of errors is 10, and total actual demand is 500. Therefore:

Because every absolute error is 10:

All actual values are positive, so:

WAPE is:

The positive ME indicates slight under-forecasting overall, despite alternating errors. If under-forecasting costs £4 per unit and over-forecasting costs £1.50 per unit, under-forecasted units total 30 and over-forecasted units total 20:

This cost is not implied by MAE alone; it depends on the organisation’s asymmetric consequences.

8. Applied case: regional distribution network

Background

The following is an illustrative case. A distributor supplies 2,400 stock-keeping units through six regional warehouses. Planners use a spreadsheet forecast, while the analytics team proposes a gradient-boosting model using sales history, promotions, holidays, prices, and weather variables.

Problem

The proposed model achieves lower aggregate WAPE in an offline experiment, but planners report that stockouts remain frequent for fast-moving promotional items. The experiment used random train-test splits and included revised promotion fields that were not known at forecast creation time.

Analysis

The organisation redesigns validation using twelve rolling origins, a four-week horizon, and a final eight-week untouched test period. The incumbent forecast, seasonal-naïve forecast, and machine-learning model are compared. Metrics include WAPE, MAE, bias, MASE, fill rate, and shortage cost. Results are segmented by warehouse, ABC volume class, promotion status, and forecast horizon.

The machine-learning model improves WAPE for stable, high-volume items but under-forecasts promotion weeks. The incumbent process performs better for a group of promotional items because planners possess event information not captured in the model. The seasonal-naïve model is strongest for several highly seasonal products. Portfolio results initially favour the machine-learning model, but service results do not.

Approach and solution

The firm adopts a segmented strategy. The machine-learning model forecasts stable high-volume items; a promotion-aware model is used for event-driven items; and seasonal naïve forecasts are retained where they are competitive. Planner overrides are logged, time-stamped, and evaluated as a separate process step. The system uses a final human review only when an exception rule is triggered, rather than allowing unrestricted manual adjustments.

A decision simulation converts forecasts into replenishment orders under actual lead times and capacity constraints. The selected portfolio reduces shortage cost and improves fill rate, although its aggregate WAPE is only modestly better than the incumbent forecast.

Learning points

• Random splits can invalidate time-series evaluation.
• Aggregate accuracy can conceal segment-level service failures.
• Planner knowledge is valuable only when its contribution is measured.
• Forecast selection should be connected to the inventory and service decision.
• A portfolio of models may be more appropriate than one universal model.

9. Governance, technology, and contemporary practice

Forecast evaluation requires clear data ownership and version control. Each forecast should retain its creation timestamp, input data version, model version, overrides, and final outcome. This audit trail supports reproducibility, accountability, and investigation of disputes.

AI and machine-learning models can improve nonlinear pattern recognition, but they also increase risks of leakage, instability, opaque decisions, and performance deterioration under distribution shift. Monitoring should track metric drift, bias drift, missing features, forecast latency, and segment-specific service. Champion–challenger testing can compare the deployed model with a stable incumbent before full replacement.

Ethically and commercially, forecasts can affect allocation of scarce stock among customers. A model that consistently prioritises high-margin customers may disadvantage smaller customers unless the policy is explicit and justified. Governance should therefore define service classes, override authority, escalation rules, and fairness considerations. Forecast accuracy must not be used to disguise a policy choice about whose demand receives priority.

10. Best-practice evaluation workflow

A robust workflow is:

1. define the decision, horizon, unit of measure, and service objective;
2. document the information available at each forecast origin;
3. establish naïve, seasonal-naïve, and incumbent benchmarks;
4. design rolling-origin validation and preserve a final test period;
5. select metrics that reflect volume, bias, large misses, and service;
6. report results by horizon, segment, and time regime;
7. test robustness and investigate exceptional periods;
8. simulate inventory, transport, or capacity consequences;
9. assess FVA for every human or algorithmic process step;
10. deploy with monitoring, version control, and scheduled review.

Chapter Summary

Forecast evaluation is a chronological, decision-oriented activity rather than a search for one universally best statistic. Holdout testing provides a simple independent assessment, while rolling-origin validation better represents repeated operational forecasting and reveals changes across horizons and time regimes. Bias, MAE, RMSE, MAPE, WAPE, and MASE measure different aspects of error and have different assumptions, especially concerning zero demand and scale.

Service metrics connect forecast performance with customer outcomes, while benchmarking protects against unjustified complexity. Model selection should consider robustness, interpretability, operational feasibility, and economic impact, not only the smallest average error. Forecast Value Added measures whether a process step improves a defined loss relative to a baseline. Ultimately, the best forecasting system is the one that produces better decisions and measurable net benefits under realistic constraints.

Key Terms

Forecast error: Difference between actual demand and forecast demand.

Bias: Systematic tendency to over-forecast or under-forecast.

Holdout set: Chronologically later observations reserved for evaluation.

Rolling-origin validation: Repeated chronological forecasting from advancing historical origins.

Expanding window: Validation design that retains all prior observations as training data.

Sliding window: Validation design using a fixed-length recent training window.

MAE: Mean absolute error, the average absolute forecast miss.

RMSE: Root mean squared error, which gives greater weight to large errors.

MAPE: Mean absolute percentage error.

WAPE: Weighted absolute percentage error based on total absolute error divided by total actual demand.

MASE: Mean absolute scaled error relative to a benchmark error scale.

Seasonal naïve forecast: Forecast equal to demand from the corresponding prior season.

Forecast horizon: Number of future periods between forecast origin and target period.

Data leakage: Use of information unavailable at the time the forecast would have been made.

Fill rate: Proportion of demand fulfilled immediately from available stock.

Cycle service level: Proportion of replenishment cycles without a stockout.

Forecast Value Added: Improvement in a defined loss relative to a baseline process.

Economic impact: Change in relevant costs and benefits resulting from forecast-driven decisions.

Forecast drift: Deterioration or change in forecast performance over time.

Champion–challenger testing: Controlled comparison of the deployed model with a candidate alternative.

Review Questions

Short Answer Questions

1. Why is chronological ordering essential in forecast validation?
2. Distinguish MAE from RMSE.
3. State one important limitation of MAPE.
4. What does a positive mean error indicate under the convention e_t=y_t-\hat{y}_t?
5. Why should forecasts be compared with naïve and incumbent benchmarks?

Long Answer Questions

1. Explain how to design a rolling-origin validation scheme for a four-week logistics forecasting horizon.
2. Compare MAPE, WAPE, and MASE for a portfolio containing intermittent and high-volume demand.
3. Discuss why accuracy metrics and service metrics may lead to different model rankings.
4. Explain how data leakage can arise in machine-learning forecasting projects and how it can be prevented.
5. Develop a model-selection framework that includes statistical performance, operational feasibility, and economic impact.

Analytical / Application Questions

1. A model has MAE of 12 units, while the benchmark has MAE of 15 units. Calculate the percentage improvement in MAE and explain why this is not necessarily equivalent to a 20% reduction in logistics cost.
2. Two models have equal WAPE. Model A has positive bias during promotions; Model B has neutral bias but slightly worse fill rate for critical customers. Recommend the additional analyses required before selection.
3. Design an FVA study for a planning process containing statistical forecasts, planner overrides, and management adjustments. Specify the baseline, evaluation periods, metrics, and controls needed for a credible conclusion.

Multiple Choice Questions with Answers

Question 1

Which validation design best preserves the information available at the time of each historical forecast?

A. Random train-test splitting

B. Rolling-origin validation

C. Duplicating the test observations in training

D. Evaluating only the fitted training values

Correct Answer: B. Rolling-origin validation

Question 2

Under the convention e_t=y_t-\hat{y}_t, what does a positive mean error indicate?

A. Systematic over-forecasting

B. Systematic under-forecasting

C. Perfect calibration

D. High forecast variance only

Correct Answer: B. Systematic under-forecasting

Question 3

Which metric is especially sensitive to large individual errors because it squares errors before averaging?

A. MAE

B. WAPE

C. RMSE

D. ME

Correct Answer: C. RMSE

Question 4

What is a principal limitation of MAPE?

A. It cannot be expressed as a percentage

B. It is undefined when actual demand is zero

C. It always penalises under-forecasting more than over-forecasting

D. It is measured in physical units

Correct Answer: B. It is undefined when actual demand is zero

Question 5

Actual demand totals 500 units and the sum of absolute errors is 50 units. What is WAPE?

A. 5%

B. 10%

C. 50%

D. 550%

Correct Answer: B. 10%

Question 6

A MASE value below 1 generally means that the evaluated forecast:

A. Has zero bias

B. Outperforms the specified benchmark on MAE

C. Has lower volume than the benchmark

D. Has perfect service level

Correct Answer: B. Outperforms the specified benchmark on MAE

Question 7

Why should a final untouched test period be retained after model selection?

A. To increase training sample size

B. To remove the need for benchmarks

C. To provide an independent confirmation after selection

D. To guarantee zero forecast error

Correct Answer: C. To provide an independent confirmation after selection

Question 8

Which statement best distinguishes fill rate from cycle service level?

A. Fill rate concerns demand quantity, whereas cycle service level concerns stockout-free cycles

B. They are always mathematically identical

C. Cycle service level measures forecast bias only

D. Fill rate applies only to transport forecasts

Correct Answer: A. Fill rate concerns demand quantity, whereas cycle service level concerns stockout-free cycles

Question 9

If baseline loss is 100 and process loss is 80, what is FVA using FVA\%=100×(L(B)-L(P))/L(B)?

A. −20%

B. 0.2%

C. 20%

D. 180%

Correct Answer: C. 20%

Question 10

Why can a model with slightly worse WAPE still be preferred operationally?

A. WAPE is always irrelevant

B. It may deliver better service or lower asymmetric shortage cost

C. Higher error always improves inventory

D. It eliminates the need for validation

Correct Answer: B. It may deliver better service or lower asymmetric shortage cost