Forecast Accuracy, Validation and Uncertainty

Introduction

A forecast is not valuable merely because it is numerically close to historical demand. It must be evaluated against an appropriate benchmark, validated on data that were not used to construct it, communicated with suitable uncertainty, and connected to operational and financial decisions. Forecast accuracy is therefore a measurement and governance problem as much as a modelling problem.

In operations management, forecast errors influence inventory, capacity, labour, procurement, transportation, cash requirements and customer service. A model with excellent average accuracy may still be unsuitable if it is systematically biased for important products, fails during promotions, or produces prediction intervals that are too narrow. This chapter develops an integrated framework for assessing point forecasts, validating forecasting systems, quantifying uncertainty and evaluating business impact.

Learning objectives

By the end of this chapter, the reader should be able to:

1. Distinguish training, validation and test data in time-dependent forecasting.
2. Calculate and interpret bias, MAE, RMSE, MAPE, WAPE and scaled errors.
3. Select metrics appropriate to the demand pattern and managerial decision.
4. Construct and assess prediction intervals.
5. Evaluate forecast accuracy, calibration, stability and business impact.
6. Design a fair model-comparison and forecast-governance process.

1. The object being evaluated: forecasts and errors

Let y_t denote actual demand in period t, and let \hat{y}_t|t-h denote the forecast made using information available at the forecast origin t-h. The forecast error is conventionally defined as actual minus forecast:

A positive error means that demand was underestimated; a negative error means that it was overestimated. The sign convention must be stated because some organisations define error as forecast minus actual.

Forecast evaluation must preserve the forecasting situation. A one-period-ahead forecast should be compared with one-period-ahead outcomes, and a twelve-period-ahead forecast with twelve-period-ahead outcomes. Mixing horizons can conceal deterioration at longer lead times. Evaluation should also respect information availability: a model must not use future prices, promotions, stock-outs or revised data that would not have been available when the forecast was issued.

Accuracy has several dimensions:

• Bias: systematic over- or under-forecasting.
• Magnitude: how large errors are, regardless of direction.
• Relative performance: how the model compares with a benchmark.
• Uncertainty quality: whether stated intervals are reliable.
• Stability: whether performance persists across time, products and regimes.
• Business impact: whether forecast improvements create operational value.

No single metric measures all six dimensions.

2. Training, validation and test designs

2.1 The time-order principle

Randomly splitting observations, as is common in independent-data machine learning, is often inappropriate for time series. Random splitting can allow information from the future to influence model selection and can produce unrealistically optimistic results. A time-series split preserves chronological order.

A basic design is:

Figure: A chronological design separates model fitting, model selection and final evaluation.

The training set estimates model parameters. The validation set selects models, hyperparameters, transformations or forecast combinations. The test set is used once, after decisions are complete, to estimate expected performance on genuinely unseen future data. If the test set is repeatedly examined and used to change the model, it is no longer a valid test set.

When data are limited, a single split can be unstable. Rolling-origin evaluation repeatedly creates forecasts from expanding or moving training windows. For origin o_j, the model is fitted using data through o_j and evaluated on subsequent observations. This design estimates performance over multiple historical forecasting situations and is especially useful for selecting hyperparameters.

An expanding window retains all available history. A moving window retains only the most recent w observations and may be preferable when structural change makes distant history less relevant. The window rule must be specified before evaluating competing models.

2.2 Leakage and comparability

Leakage occurs when information unavailable at the forecast origin enters model estimation or feature construction. Examples include using end-of-month sales adjustments in a mid-month forecast, calculating a promotion variable from future campaign information, or imputing missing demand with a statistic calculated from the full dataset. Leakage produces invalidly favourable results.

Comparisons should use identical forecast origins, horizons, actuals, exclusions and aggregation levels. If one model receives promotional calendars and another does not, the comparison is a comparison of information sets as well as algorithms.

3. Accuracy measures

Suppose evaluation covers n forecast errors e₁,\ldots,e_n. The formulas below assume that actuals and forecasts are measured in the same units, such as units per week.

3.1 Mean error and bias

Mean error (ME) measures signed bias. Its units are demand units. Under the convention e_t=y_t−\hat{y}_t, positive ME indicates systematic under-forecasting. Positive and negative errors can cancel, so ME should not be used as a measure of error magnitude.

A relative bias measure may be written as:

This is meaningful when total actual demand is positive. It is not appropriate when the denominator is zero or when net demand can be negative.

3.2 MAE and RMSE

Mean absolute error (MAE) treats all absolute deviations proportionally and is expressed in demand units. It is relatively interpretable for operational planning.

Root mean squared error (RMSE) also has demand units, but squaring gives larger errors disproportionate influence. RMSE is useful when large misses are especially costly. It is more sensitive to outliers than MAE.

For nonnegative errors, RMSE is at least as large as MAE. Equality occurs when all absolute errors are equal; otherwise, variation in error magnitude makes RMSE larger.

3.3 Percentage and aggregate-scaled measures

MAPE is undefined when any y_i=0 and becomes highly unstable when actual demand is close to zero. It also weights a 10-unit error on a small item more heavily than the same error on a large item. Thus, MAPE should not be the default metric for intermittent or low-volume demand.

Weighted absolute percentage error is:

For ordinary nonnegative demand, the denominator is total demand. WAPE is an aggregate measure that gives greater influence to high-volume observations. It is defined when the denominator is nonzero, but it can conceal poor performance on low-volume items.

A scaled error compares a forecast error with the error of a naïve benchmark. For seasonal period m, the mean absolute scaled error (MASE) is commonly defined as:

Here T is the number of observations in the training data used to calculate the scaling denominator. The denominator is the mean absolute error of a seasonal naïve forecast on the training sample. MASE below 1 indicates lower MAE than that benchmark, subject to comparable data and horizon definitions. If the scaling denominator is zero, MASE is undefined and another benchmark or scale must be used.

3.4 Worked metric example

Consider four weekly observations and forecasts:

The ME is (10−10−20+10)/4=−2.5 units, indicating slight over-forecasting. MAE is (10+10+20+10)/4=12.5 units. RMSE is \sqrt{(100+100+400+100)/4}=\sqrt{175}≈13.23 units.

MAPE is 100×(0.10+0.0833+0.25+0.10)/4≈13.33\%. WAPE is 100×50/400=12.5\%. The difference arises because MAPE gives the 20-unit error on demand of 80 a larger relative weight, whereas WAPE aggregates absolute error against total volume.

4. Choosing and interpreting metrics

Metric selection should follow the decision, not convenience. A distribution centre concerned with total network volume may use WAPE, while a service-critical component may require MAE, quantile loss or a cost-weighted measure. A replenishment system that penalises stock-outs more than excess inventory should not select a model solely by symmetric RMSE.

Useful practice is to report a metric panel:

Metrics should be segmented by forecast horizon, product family, region, customer, demand state and event type. A single portfolio average is a useful summary but not a sufficient diagnostic.

5. Prediction intervals and uncertainty

A point forecast estimates a central outcome; a prediction interval describes plausible future outcomes. A 90% prediction interval is not a statement that there is a 90% probability that a fixed unknown parameter lies inside the interval. It is a procedure intended, under repeated comparable forecasting situations, to contain the future observation approximately 90% of the time.

Write an interval as [L_t+h,U_t+h], where L and U are lower and upper limits. A common model-based form is:

Here α is the nominal outside probability, z_1−α/2 is the corresponding standard-normal quantile, and σ_h is the forecast-error standard deviation at horizon h. This symmetric formula is appropriate only when the predictive error distribution and variance estimate justify it. For nonnegative, skewed or intermittent demand, quantile forecasts or transformed-scale intervals may be more appropriate.

If errors are independent with one-step standard deviation σ, the uncertainty of a sum over h periods is proportional to \sqrt{h}σ. In time-series models, serial correlation, parameter uncertainty, seasonality and changing variance alter this relationship; therefore, using \sqrt{h} mechanically can be misleading.

5.1 Interval evaluation

Coverage at nominal level 1−α is:

where I(·) equals 1 when the condition is true and 0 otherwise. Empirical coverage should be close to the nominal level over a sufficiently large and relevant evaluation sample. Coverage alone is insufficient: an interval extending from negative infinity to positive infinity has excellent coverage but little decision value.

Interval width is:

A useful interval is both calibrated and reasonably sharp. Sharpness means concentration of the predictive distribution, conditional on being calibrated. Calibration should also be assessed by horizon, product and demand regime. Repeated misses in one direction indicate miscalibration even if aggregate coverage appears satisfactory.

For probabilistic forecasts, the continuous ranked probability score (CRPS) evaluates the whole predictive distribution rather than only an interval. In applied governance, interval coverage, width, quantile loss and service outcomes are often easier to explain and operationalise.

6. Stability, drift and statistical comparison

A model’s average historical score can hide temporal instability. Plot rolling MAE or WAPE, compare pre- and post-event periods, and inspect error distributions rather than only means. A sudden increase may reflect data quality, a changed assortment, a new promotion policy or genuine concept drift.

Stability can be assessed across:

• time windows and forecast horizons;
• products, locations and customer segments;
• ordinary, promotional and disruption periods;
• high- and low-volume observations;
• model retraining cycles.

When models have similar scores, the difference may not be practically or statistically meaningful. For paired forecasts, compare errors on the same observations. A forecast comparison test such as the Diebold–Mariano approach can examine whether mean loss differs, but dependence, multiple horizons and nonstationarity require careful specification. Statistical significance is not the same as economic significance.

Benchmark models are essential. Examples include last-value naïve, seasonal naïve, drift and simple moving-average forecasts. A complex model that cannot consistently outperform an appropriate benchmark should not be adopted merely because it is technologically sophisticated.

7. From forecast accuracy to business impact

Forecast metrics are intermediate measures. Operations decisions respond to inventory and capacity consequences. Let c_u represent the unit cost of under-forecasting and c_o the unit cost of over-forecasting. A simplified expected error cost is:

This formula is a diagnostic cost model, not a complete inventory model. It assumes linear, separable costs and does not explicitly represent lead times, order quantities, substitutions, fixed costs or service constraints.

For a single-period newsvendor decision with continuous demand and underage and overage costs, the target demand quantile satisfies:

where F is the demand distribution function and q* is the order quantity. This result explains why a mean forecast is not always the appropriate operational forecast: asymmetric costs require an appropriate quantile.

Business evaluation should therefore include inventory holding cost, stock-out or lost-margin cost, expedite cost, capacity utilisation, obsolescence, working capital and customer service. Simulation or controlled back-testing can translate alternative forecasts into these outcomes. Causal interpretation requires caution: an observed service improvement may result from increased safety stock rather than improved forecasts.

8. Illustrative case: forecasting a seasonal consumer product

Background

The following is a fictional illustrative case. A regional retailer forecasts weekly demand for a seasonal household product across 180 stores. The current system uses a seasonal naïve forecast. A data science team proposes a gradient-boosted model using calendar variables, price, promotions, weather and store characteristics.

Problem

The proposed model reports WAPE of 14%, compared with 17% for the incumbent. However, the evaluation used random cross-validation, included revised promotion records, and combined low-volume stores with high-volume stores. Managers also report frequent stock-outs during holiday promotions.

Analysis

The retailer redesigns the evaluation using rolling origins. Each forecast is generated with information available at the time, with one- to four-week horizons. The final eight weeks are retained as a test period. The team reports MAE, RMSE, WAPE, bias, seasonal MASE, interval coverage and average interval width by store group and horizon.

The boosted model achieves lower WAPE for high-volume stores but is biased low during promotions. Its nominal 90% intervals have only 72% coverage during promotion weeks, although overall coverage is 88%. The seasonal naïve model has slightly worse average WAPE but more stable performance and better promotional coverage.

Approach and solution

The retailer combines a promotion-aware model with a calibrated residual distribution. It separately evaluates ordinary and promotion periods, recalibrates interval limits using a validation period, and retains the seasonal naïve forecast as a fallback when promotion data are incomplete. Model selection uses a cost-weighted score that reflects stock-out consequences, while the dashboard continues to display conventional metrics for transparency.

Outcome

In back-testing, the selected system reduces stock-out cost without increasing total inventory cost. The improvement is not attributed to WAPE alone: it arises from better event representation, improved interval calibration, and a decision rule that values under-forecasting appropriately. The model is monitored monthly, and a human review is required when data completeness or forecast bias breaches predefined thresholds.

Learning points

1. Random validation can overstate performance in temporal data.
2. Portfolio averages can hide segment-specific failure.
3. Prediction intervals require calibration checks, not merely visual plausibility.
4. A benchmark and fallback model are components of responsible deployment.
5. Forecast improvement should be evaluated through operational consequences.

9. Governance, ethics and contemporary practice

Forecasting systems can affect product availability, labour allocation and supplier relationships. Governance should document data sources, forecast origins, transformations, exclusions, model versions, metric definitions and approval rights. Reproducible evaluation requires a frozen test set and an auditable record of changes.

Bias can arise from systematic underrepresentation, stock-out-censored sales, historical allocation rules or features that act as proxies for protected characteristics. Demand observed during a stock-out is not necessarily unconstrained demand. Treating it as ordinary low demand can create a self-reinforcing cycle of under-forecasting. Sensitive applications require appropriate privacy, fairness and security controls.

Machine-learning systems can improve nonlinear feature representation, but they do not remove the need for temporal validation, benchmarks, uncertainty estimates and human accountability. Contemporary practice increasingly uses probabilistic forecasting, forecast combinations, hierarchical reconciliation, conformal prediction and monitoring for data and concept drift. These methods are valuable only when their assumptions and calibration are tested in the relevant operational environment.

Best-practice checklist

1. Define the forecast origin, horizon and information set.
2. Preserve time order and prevent leakage.
3. Use rolling-origin validation when data permit.
4. Keep a final test set untouched until model selection is complete.
5. Compare with strong naïve and seasonal benchmarks.
6. Report bias, magnitude, relative performance and uncertainty quality.
7. Avoid uncritical use of MAPE for zero or near-zero demand.
8. Segment results by horizon, volume, product and event regime.
9. Assess interval coverage and width together.
10. Translate forecast differences into operational and financial outcomes.
11. Monitor drift, data quality and calibration after deployment.
12. Maintain documentation, version control and escalation rules.

Chapter Summary

Forecast evaluation is a structured process rather than a search for one superior metric. Chronological training, validation and test designs protect against leakage and optimistic performance estimates; rolling-origin evaluation provides evidence across multiple forecasting situations. Bias measures direction, MAE measures typical absolute error, RMSE emphasises large misses, MAPE measures percentage error but fails at zero demand, WAPE measures aggregate relative error, and scaled errors compare performance with a benchmark.

Point forecasts should be complemented by prediction intervals or full predictive distributions. Coverage measures calibration, while width and sharpness address usefulness. Accuracy must be examined across horizons, segments and demand regimes because averages can hide instability. Finally, forecast quality should be judged by its effect on inventory, service, capacity and financial outcomes, with governance procedures ensuring transparency, fairness, reproducibility and accountability.

Key Terms

Bias: Systematic tendency to over- or under-forecast.

Calibration: Agreement between stated predictive probabilities or interval levels and observed frequencies.

Concept drift: Change in the underlying demand-generating relationship over time.

Coverage: Proportion of future observations contained within a prediction interval.

Forecast error: Difference between actual demand and forecast demand.

Forecast horizon: Number of periods between the forecast origin and the predicted period.

Forecast origin: Time at which a forecast is produced.

MAE: Mean absolute error, the average absolute forecast error.

MAPE: Mean absolute percentage error.

MASE: Mean absolute scaled error relative to a naïve benchmark.

ME: Mean error, a signed measure of average forecast error.

Prediction interval: Range intended to contain a future observation at a stated nominal level.

RMSE: Root mean squared error, which penalises large errors more strongly.

Rolling-origin evaluation: Repeated chronological forecasting and evaluation using successive origins.

Sharpness: Concentration or narrowness of a predictive distribution, considered alongside calibration.

Test set: Chronological holdout used for final performance estimation.

Training set: Data used to estimate model parameters.

Validation set: Data used to select models, features or hyperparameters.

WAPE: Weighted absolute percentage error based on aggregate absolute error divided by aggregate demand.

Review Questions

Short Answer Questions

1. Why is random train–test splitting often inappropriate for time-series forecasting?
2. What does a positive mean error indicate under the convention e_t=y_t−\hat{y}_t?
3. State one important limitation of MAPE.
4. What is the difference between interval coverage and interval width?
5. Why should a forecasting system be compared with a naïve benchmark?

Long Answer Questions

1. Explain how training, validation and test sets should be designed for a multi-horizon forecasting problem.
2. Compare MAE, RMSE, MAPE, WAPE and MASE, including suitable and unsuitable applications.
3. Explain why a forecast with good point accuracy may still have poor uncertainty quality.
4. Discuss how forecast errors can be translated into inventory and service consequences.
5. Develop a governance framework for monitoring a forecasting model after deployment.

Analytical / Application Questions

1. A product has actual demand of 200, 250 and 150 units and forecasts of 180, 275 and 120 units. Calculate ME, MAE, RMSE and WAPE, and interpret the signs and units.
2. A model reports 92% coverage for nominal 90% intervals, but its intervals are twice as wide as a competing model with 89% coverage. Which additional evidence would you require before selecting a model?
3. A retailer’s model performs well in ordinary weeks but under-forecasts every promotion. Design a validation, metric and monitoring approach that would reveal and address this problem.

Multiple Choice Questions with Answers

Question 1

Which data split is most appropriate for evaluating a time-series forecasting model?

A. A random split that mixes past and future observations

B. A chronological split that preserves the forecasting order

C. A split based only on product names

D. A split using only observations with nonzero demand

Correct Answer: B. A chronological split that preserves the forecasting order

Question 2

Under the convention e_t=y_t−ŷ_t, what does a positive mean error indicate?

A. Systematic over-forecasting

B. Systematic under-forecasting

C. Perfect calibration

D. Zero absolute error

Correct Answer: B. Systematic under-forecasting

Question 3

For actuals 100 and 200, with forecasts 90 and 180, what is the WAPE?

A. 5%

B. 8.33%

C. 10%

D. 15%

Correct Answer: B. 8.33%

Question 4

Which metric is undefined when an actual observation equals zero?

A. MAE

B. RMSE

C. MAPE

D. WAPE, provided total demand is positive

Correct Answer: C. MAPE

Question 5

Which metric gives disproportionate influence to unusually large errors because errors are squared?

A. MAE

B. RMSE

C. ME

D. WAPE

Correct Answer: B. RMSE

Question 6

A MASE value below 1 generally indicates that the evaluated forecast has what property?

A. Higher MAE than the scaling benchmark

B. Lower MAE than the scaling benchmark

C. Exactly zero bias

D. Nominal 90% interval coverage

Correct Answer: B. Lower MAE than the scaling benchmark

Question 7

What does 90% prediction-interval coverage measure?

A. The average width of the interval

B. The proportion of actual outcomes inside the interval

C. The percentage of forecasts equal to actuals

D. The cost of under-forecasting

Correct Answer: B. The proportion of actual outcomes inside the interval

Question 8

Why can excellent aggregate WAPE conceal an operational problem?

A. WAPE always ignores actual demand

B. WAPE gives greater aggregate influence to high-volume observations

C. WAPE is always undefined for positive demand

D. WAPE measures only forecast bias

Correct Answer: B. WAPE gives greater aggregate influence to high-volume observations

Question 9

Which practice most directly reduces leakage in validation?

A. Using future revised promotion records

B. Randomly shuffling all observations

C. Constructing features only from information available at the forecast origin

D. Selecting the model after repeatedly inspecting the test set

Correct Answer: C. Constructing features only from information available at the forecast origin

Question 10

If underage cost is greater than overage cost in a single-period decision, the desired demand quantile should generally be:

A. Lower than 0.5

B. Exactly 0.5 regardless of costs

C. Greater than 0.5

D. Equal to zero

Correct Answer: C. Greater than 0.5