Forecast Evaluation, Uncertainty and Decision Risk

Introduction

A demand forecast is not a promise about the future. It is an estimate produced from information available at a particular time, under particular assumptions. Marketing managers use forecasts to set prices, allocate media budgets, plan inventory, schedule capacity, negotiate with channels, and evaluate commercial investments. Because the future is uncertain, forecast quality must be assessed systematically and uncertainty must be translated into decisions.

Forecast evaluation has two distinct dimensions. Accuracy evaluation asks how close forecasts were to observed outcomes. Decision evaluation asks whether the forecast led to a good managerial action given the costs of being wrong. A model may have excellent average accuracy but be unsuitable for a high-risk launch; conversely, a slightly less accurate model may be preferable if it produces more reliable downside estimates.

This chapter develops a practical framework using error metrics, bias diagnostics, cross-validation, benchmark comparisons, prediction intervals, probabilistic forecasts, stress tests, scenarios, and decision rules.

Learning Objectives

By the end of this chapter, you should be able to:

1. Calculate and interpret MAE, RMSE, MAPE, WAPE, bias, and tracking signals.
2. Select metrics appropriate to the scale, volatility, and purpose of a forecast.
3. Evaluate models using time-series cross-validation and meaningful benchmarks.
4. Construct and interpret prediction intervals and probabilistic forecasts.
5. Design stress tests and scenarios for pricing and demand decisions.
6. Translate forecast uncertainty into inventory, pricing, promotion, and capacity decisions.
7. Identify governance, ethical, and operational risks in forecast use.

1. The Forecast Error and Its Meaning

Let y_t denote actual demand in period t, and \hat{y}_t denote the forecast made for that period. The forecast error is:

A positive error means demand exceeded the forecast (under-forecasting); a negative error means the forecast was too high. This sign convention must be stated because some systems define error as forecast minus actual.

A forecast error contains several possible components: random noise, systematic bias, omitted demand drivers, structural change, measurement error, and errors introduced by promotional or pricing assumptions. A low error in one period does not establish that a forecasting process is reliable. Evaluation should therefore use a relevant historical sample and preserve the information chronology of the business.

Scale-dependent metrics

Mean absolute error (MAE) measures the average absolute error in the original unit of demand:

Here, n is the number of evaluated periods. If demand is measured in units, MAE is measured in units. MAE is easy to explain and gives equal weight to each period’s absolute miss.

Root mean squared error (RMSE) is:

RMSE is also expressed in demand units, but squares errors before averaging. Consequently, it penalizes large misses more heavily than MAE. It is appropriate when occasional large errors have disproportionate commercial consequences, such as stockouts during a major launch.

RMSE is never smaller than MAE for the same observations when both are calculated consistently, because the root-mean-square is at least the arithmetic mean for non-negative values. The difference between them is informative: a large gap suggests outliers or highly uneven error magnitudes.

Percentage and aggregate metrics

Mean absolute percentage error (MAPE) is commonly written:

MAPE is interpretable as an average percentage miss, but it is undefined when actual demand is zero and becomes unstable when actual demand is close to zero. It also treats proportional errors asymmetrically: over- and under-forecasting can have different percentage behavior around the same values.

Weighted absolute percentage error (WAPE) is:

For non-negative demand, the absolute-value signs in the denominator are unnecessary. WAPE weights periods by their demand volume and is often more stable than MAPE for portfolios containing low-volume items. It should not be confused with an average of item-level percentage errors: high-volume items dominate the aggregate.

When actual demand is zero, MAPE should be reported as not applicable, replaced by a suitable metric, or calculated only under a transparently stated rule. Imputing an arbitrary small denominator can create misleading performance claims.

Bias and tracking signal

The cumulative forecast error, or mean error, indicates directional tendency:

Some organizations use the cumulative error instead. Therefore, reports should label whether “bias” means mean error, cumulative error, or a percentage version.

A common tracking signal is:

where MAD is the mean absolute deviation, usually equivalent to MAE over the same evaluation window. A positive tracking signal indicates repeated under-forecasting; a negative value indicates repeated over-forecasting. Control limits such as ±4 or ±5 are organizational conventions, not universal statistical laws. They should be calibrated to the business, data volume, and consequences of intervention.

Bias is different from accuracy. Forecasts may have zero average bias while making large positive and negative errors. Conversely, a consistently biased forecast may have a modest MAE but systematically mislead capacity or pricing decisions.

2. A Worked Evaluation Example

Suppose weekly demand and forecasts for five weeks are as follows.

Thus, MAE = 55/5 = 11 units and Bias = -5/5 = -1 unit per week. The forecast is slightly high on average because actual minus forecast is negative.

The squared errors sum to 100 + 25 + 100 + 400 + 100 = 725, so:

RMSE = √(725/5) = √145 ≈ 12.04 units.

For WAPE:

WAPE = 100 × 55/500 = 11%.

MAPE is calculated as:

(10% + 4.17% + 12.5% + 13.33% + 20%)/5 ≈ 12.0%.

The MAPE result is strongly affected by the low-volume fifth week, where a 10-unit miss equals 20 percent. Management should therefore avoid declaring one model superior using MAPE alone.

3. Choosing Metrics for Marketing Decisions

No single metric is universally best. Metric selection should reflect the decision loss function.

A forecast for pricing should often be evaluated in economic terms. If contribution margin per unit is m and the business cost of a stockout or excess unit differs, the preferred model is the one minimizing expected economic loss, not necessarily MAE. This requires explicit assumptions about salvage value, replenishment, service levels, and customer response.

4. Benchmarks and Time-Series Cross-Validation

A complex model should earn its complexity by outperforming credible benchmarks. Useful benchmarks include:

• Naïve forecast: the next period equals the latest observed value, \hat{y}_t+1=y_t.
• Seasonal naïve forecast: the next value equals the observation from the corresponding prior season, \hat{y}_t+h=y_t+h-s, where s is the seasonal period.
• Moving-average forecast: the forecast is the average of a specified recent window.
• Management or consensus forecast: useful as a governance benchmark, even when not statistically constructed.

A model that cannot outperform a seasonal naïve benchmark consistently may be adding complexity without value. However, a benchmark comparison must use identical forecast horizons, information sets, data cleaning rules, and evaluation periods.

Random k-fold cross-validation is generally inappropriate for ordinary time series because it can train on observations that occur after the validation observation. Rolling-origin evaluation preserves temporal order. For example, a model may be trained through month 12 and tested on month 13, then trained through month 13 and tested on month 14, and so forth. For a fixed-origin expanding-window design:

1. Select an initial training window.
2. Fit the model using only that window.
3. Forecast the next horizon.
4. Record the error.
5. Expand the training window and repeat.

A sliding-window design keeps the training window length fixed and may be preferable when relationships change rapidly. Cross-validation should match the actual decision horizon: a model used for a 12-week procurement decision should not be selected only on one-step-ahead errors.

5. Uncertainty: From Point Forecasts to Distributions

A point forecast gives one central estimate. A prediction interval gives a range intended to contain a future observation with a stated coverage probability, conditional on model assumptions. A 90 percent prediction interval does not mean that there is a 90 percent probability that a fixed, already-realized outcome is random in a philosophical sense; operationally, it means that the forecasting procedure is designed to contain future outcomes about 90 percent of the time under comparable conditions.

For a simple forecast with approximately independent, normally distributed residuals, a prediction interval can be represented as:

Here, h is the forecast horizon, α is the probability outside the interval, z_1-α/2 is the corresponding standard-normal critical value, and SE_pred,h is the standard error of prediction at horizon h. The standard error must include uncertainty in the future observation, not merely uncertainty in estimated model parameters. For time series with autocorrelation, seasonality, changing variance, promotions, or non-normal residuals, a simple normal interval may be poorly calibrated.

A probabilistic forecast assigns probabilities to multiple demand outcomes. It may be represented by quantiles, such as the 10th, 50th, and 90th percentiles, or by a full predictive distribution. Quantile forecasts are especially useful for inventory and capacity because decisions often depend on a particular percentile rather than the mean.

Intervals should be tested for calibration: among forecasts labeled 90 percent, approximately 90 percent of comparable outcomes should fall inside the intervals. They should also be assessed for sharpness: narrower intervals are desirable only when calibration is maintained. A narrow but poorly calibrated interval is dangerous false precision.

6. Stress Tests and Scenarios

Historical error evaluation cannot fully represent structural breaks, competitor entry, regulatory change, viral demand, or unprecedented supply disruption. Managers should therefore supplement statistical validation with structured uncertainty analysis.

A sensitivity analysis changes one input at a time, such as price elasticity, media response, or conversion rate. A stress test examines an adverse but plausible condition, such as a 25 percent demand decline after a competitor launch. A scenario combines coherent assumptions—for example, recession, aggressive competitor discounting, and reduced consumer traffic.

A practical scenario structure is:

Scenarios should not be presented as precise probabilities unless a defensible probability model exists. If probabilities are assigned, expected demand is:

where D_j is demand under scenario j, p_j is its probability, and ∑p_j = 1. Expected demand is not necessarily the correct stocking quantity because asymmetric underage and overage costs may make a higher or lower quantile optimal.

7. Translating Uncertainty into Decisions

The decision-maker should connect the predictive distribution to consequences. Consider an inventory decision with underage cost C_u and overage cost C_o. Under the standard single-period logic, the critical fractile is:

where Q^* is the target quantity and D is uncertain demand. This result assumes a single ordering decision, known costs, and a suitable demand distribution. If underage cost is high relative to overage cost, the target percentile rises.

Pricing decisions require additional care. A higher price may improve margin per unit but reduce volume. Forecast uncertainty should be carried through the demand-response model into revenue and contribution-margin distributions. Managers should ask:

• What is the expected contribution margin?
• What is the downside margin under adverse elasticity or competitor response?
• What capacity or inventory commitment is irreversible?
• Can the decision be delayed until information improves?
• Is there a trigger for changing price, promotion, or replenishment?

A useful governance device is a decision trigger: for example, revise the promotion if the rolling tracking signal exceeds a calibrated limit, or increase replenishment when observed demand exceeds the 80th percentile for two consecutive periods. Triggers reduce ad hoc reactions but must be monitored to prevent automatic overcorrection.

8. Illustrative Case: NovaBrew’s Seasonal Launch

Background

NovaBrew, a fictional premium beverage company, is launching a limited seasonal flavor. The marketing team must choose an initial production quantity and introductory price. Historical products show strong seasonality and occasional promotional spikes. The team’s machine-learning forecast predicts 100,000 units for the launch month, while a seasonal naïve benchmark predicts 92,000.

Problem

The point forecast conceals substantial uncertainty. A competitor may launch a substitute, but social-media attention could also create unusually high demand. Excess inventory has a disposal cost, whereas a stockout damages retailer relationships and forfeits contribution margin.

Analysis

Rolling-origin validation over comparable launches shows the machine-learning model has lower MAE than the benchmark, but its RMSE is higher because it made two very large misses. WAPE favors the machine-learning model by a small margin. Bias is positive on the actual-minus-forecast convention, indicating under-forecasting during promotional weeks. The team therefore investigates whether the model omits media intensity and retailer feature placement.

The revised forecast is expressed through quantiles: the 20th percentile is 78,000 units, the median is 100,000, and the 80th percentile is 128,000. Scenario analysis adds a downside case of 70,000 units and an upside case of 145,000 units. These are not treated as guaranteed bounds; they are planning cases.

Approach and solution

Because overage cost is substantial but stockout cost is also material, NovaBrew selects a quantity near the 70th percentile rather than automatically using the mean. It negotiates a staged production contract: 85,000 units initially, with an option for rapid replenishment if early sales exceed a predefined threshold. The introductory price is tested in a controlled regional rollout rather than changed nationally. A dashboard tracks actual demand, forecast error, interval coverage, competitor price, and retailer availability.

Outcome and learning points

The staged commitment limits downside exposure while preserving upside capacity. The case demonstrates that forecast evaluation is not simply a contest for the lowest average error. The relevant process combines benchmark comparison, bias diagnosis, probabilistic forecasts, operational flexibility, and explicit loss trade-offs.

9. Limitations, Governance and Ethical Issues

Forecast metrics can be manipulated through selective time windows, excluded stockout periods, favorable denominators, or retrospective data cleaning. Organizations should document data vintages, forecast creation dates, overrides, excluded observations, and metric definitions.

Stockouts create a particularly serious problem: observed sales may be lower than unconstrained demand. Evaluating a forecast against sales without recognizing availability can incorrectly reward under-forecasting. Promotions also require careful attribution because forecast error may reflect an incorrect promotional plan rather than a failure of the baseline demand model.

AI and machine-learning forecasts introduce additional governance concerns. Models may learn historical discrimination, amplify unequal access patterns, or produce opaque recommendations. Managers should assess data provenance, explainability, drift, privacy, and human accountability. Automated pricing must comply with competition law and should not facilitate collusion or exploit vulnerable customers. Forecasts should support judgment, not conceal responsibility behind a model score.

Contemporary practice increasingly uses forecast reconciliation across product, region, channel, and total-company levels; conformal or distribution-free methods for interval construction; probabilistic machine learning; and monitoring systems for calibration and drift. These methods are valuable, but their performance still depends on representative data and realistic back-testing.

Best-Practice Checklist

1. Define error sign conventions and metric formulas before evaluation.
2. Report at least one scale-dependent metric, one relative metric, and a bias measure.
3. Handle zero and near-zero demand explicitly.
4. Compare sophisticated models with naïve and seasonal benchmarks.
5. Use rolling-origin validation that matches the business forecast horizon.
6. Evaluate forecast intervals for calibration, not merely narrowness.
7. Include promotions, stockouts, price changes, and structural breaks in interpretation.
8. Translate demand uncertainty into revenue, margin, service, and risk outcomes.
9. Use scenarios and stress tests for plausible structural changes.
10. Preserve an audit trail for data, overrides, assumptions, and decisions.

Chapter Summary

Forecast evaluation requires more than selecting the model with the smallest average error. MAE provides an interpretable absolute error; RMSE emphasizes large misses; MAPE expresses proportional error but fails with zero or very small actuals; WAPE aggregates absolute error relative to total volume. Bias and tracking signals diagnose persistent directional error. Benchmarks establish whether complexity adds value, while rolling-origin cross-validation respects time order and forecast horizon.

Point forecasts should be supplemented with prediction intervals, quantiles, or full predictive distributions. Calibration and sharpness are central to judging uncertainty estimates. Stress tests and scenarios address risks that historical data cannot represent. Finally, managers should translate predictive uncertainty into economic decisions using costs, flexibility, service requirements, and decision triggers. Ethical governance requires transparency, accountability, careful treatment of constrained demand, and oversight of automated pricing and forecasting systems.

Key Terms

Forecast error: Difference between actual demand and forecast demand.

MAE: Mean absolute error measured in the original demand units.

RMSE: Root mean squared error, which gives greater weight to large errors.

MAPE: Mean absolute percentage error.

WAPE: Weighted absolute percentage error based on aggregate absolute error.

Bias: A systematic tendency to over-forecast or under-forecast.

Tracking signal: Cumulative error scaled by mean absolute deviation.

Benchmark: A simple or established forecast used for comparison.

Naïve forecast: Forecast based on the latest observation.

Seasonal naïve forecast: Forecast based on the corresponding prior-season observation.

Rolling-origin validation: Time-ordered repeated forecasting evaluation.

Prediction interval: Range intended to contain a future observation at stated coverage.

Probabilistic forecast: Forecast represented by probabilities, quantiles, or a distribution.

Calibration: Agreement between stated predictive coverage and observed coverage.

Sharpness: Concentration or narrowness of predictive distributions, conditional on calibration.

Stress test: Evaluation under an adverse but plausible condition.

Scenario: A coherent set of assumptions describing a possible future.

Critical fractile: Target cumulative probability determined by underage and overage costs.

Forecast override: Human adjustment to a statistical forecast.

Forecast drift: Deterioration caused by changing data patterns or relationships.

Review Questions

Short Answer Questions

1. Why can RMSE be substantially higher than MAE for the same forecast set?
2. Identify two limitations of MAPE.
3. What does a positive tracking signal indicate under the convention e_t=y_t-\hat{y}_t?
4. Why is random k-fold cross-validation usually unsuitable for time-series forecasting?
5. Distinguish a prediction interval from a point forecast.

Long Answer Questions

1. Compare MAE, RMSE, MAPE, and WAPE, including their units, strengths, and limitations.
2. Explain how rolling-origin cross-validation and benchmark comparisons should be designed for a seasonal product.
3. Discuss how forecast bias can remain hidden when managers examine accuracy metrics alone.
4. Explain calibration and sharpness in the evaluation of probabilistic forecasts.
5. Discuss the ethical and governance risks associated with automated forecasting and pricing.

Analytical / Application Questions

1. A product has actual demand of 200, 250, and 150 units, with forecasts of 220, 230, and 180. Calculate MAE, RMSE, bias, and WAPE.
2. A retailer faces an underage cost of $12 per unit and an overage cost of $8 per unit. What demand percentile follows from the critical-fractile rule, and what assumptions must hold for that rule to be useful?
3. Design a rolling-origin evaluation and scenario plan for a product whose demand is affected by price promotions, stockouts, and a newly entered competitor.

Multiple Choice Questions with Answers

Question 1

Which metric penalizes large forecast errors most heavily because it squares errors before averaging?

A. MAE

B. RMSE

C. WAPE

D. Bias

Correct Answer: B. RMSE

Question 2

Using the convention e_t=y_t-\hat{y}_t, a positive tracking signal generally indicates:

A. Persistent over-forecasting

B. Persistent under-forecasting

C. Perfect calibration

D. Zero forecast variance

Correct Answer: B. Persistent under-forecasting

Question 3

Which metric is undefined when actual demand is zero?

A. MAE

B. RMSE

C. MAPE

D. Bias

Correct Answer: C. MAPE

Question 4

Why is random k-fold cross-validation problematic for ordinary time series?

A. It always produces biased parameter estimates

B. It can use future observations to predict earlier observations

C. It cannot calculate absolute errors

D. It requires probabilistic forecasts

Correct Answer: B. It can use future observations to predict earlier observations

Question 5

A seasonal naïve forecast for a monthly series most commonly uses:

A. The overall mean of all months

B. The immediately previous day

C. The corresponding month from the prior season

D. The largest historical observation

Correct Answer: C. The corresponding month from the prior season

Question 6

A 90 percent prediction interval is intended to:

A. Contain every future observation

B. Contain future observations approximately 90 percent of the time under comparable conditions

C. Provide the exact probability of a fixed outcome

D. Eliminate model uncertainty

Correct Answer: B. Contain future observations approximately 90 percent of the time under comparable conditions

Question 7

If absolute errors are 4, 6, and 10 units, the MAE is:

A. 5 units

B. 6⅔ units

C. 20 units

D. 8 units

Correct Answer: B. 6⅔ units

Question 8

If underage cost is $12 and overage cost is $8, the critical fractile is:

A. 0.40

B. 0.60

C. 0.67

D. 1.50

Correct Answer: C. 0.67

Question 9

Which statement best describes WAPE?

A. It weights errors by squared demand

B. It divides total absolute error by total actual volume

C. It is always identical to MAPE

D. It is defined only when every actual is zero

Correct Answer: B. It divides total absolute error by total actual volume

Question 10

A stress test is best described as:

A. A random deletion of historical observations

B. An adverse but plausible condition used to examine decision resilience

C. A replacement for all historical validation

D. A guarantee of the worst possible outcome

Correct Answer: B. An adverse but plausible condition used to examine decision resilience