Forecasting Workflow, Baselines, and Evaluation

Introduction

A forecasting model is useful only when it supports a decision. A highly sophisticated algorithm can be less valuable than a simple benchmark if it is difficult to maintain, poorly validated, or systematically biased. Professional forecasting therefore requires a disciplined workflow: define the decision and forecast target, prepare time-ordered data, establish credible baselines, validate without looking into the future, diagnose errors, select suitable accuracy measures, and monitor performance after deployment.

This chapter develops that workflow for business analytics. It focuses on naive and seasonal-naive benchmarks, rolling-origin validation, time-series cross-validation, residual diagnostics, and evaluation measures including MAE, RMSE, MAPE, MASE, bias, and business-weighted loss.

Learning Objectives

By the end of this chapter, you should be able to:

1. Design a leakage-resistant forecasting workflow.
2. Construct and interpret naive and seasonal-naive forecasts.
3. Apply rolling-origin and time-series cross-validation.
4. Calculate and interpret common forecast-error measures.
5. Diagnose residual dependence, bias, outliers, and changing variance.
6. Select models using statistical and business criteria.
7. Communicate forecast uncertainty and governance implications.

1. Forecasting as a Decision Process

Let y_t denote the observed value of a variable at time t, and let \hat{y}_t|t-1 denote a forecast made using information available no later than time t−1. The forecast error is:

A positive error means that actual demand exceeded the forecast; a negative error means that the forecast was too high. The sign convention must be stated because some organizations define error as forecast minus actual.

A practical forecasting project usually contains these stages:

1. Decision definition: identify who uses the forecast, when, and for what action.
2. Target definition: specify the variable, unit, frequency, horizon, and aggregation level.
3. Data audit: check timestamps, missing values, duplicates, outliers, revisions, and structural breaks.
4. Temporal partitioning: separate training, validation, and test periods chronologically.
5. Baseline construction: establish naive and seasonal-naive benchmarks.
6. Model development: fit candidate methods using training information only.
7. Rolling evaluation: simulate repeated historical forecasting decisions.
8. Residual diagnosis: investigate patterns that indicate model inadequacy.
9. Business evaluation: incorporate costs, service levels, capacity, and risk.
10. Deployment and monitoring: record forecast versions, inputs, errors, and interventions.

The workflow is iterative. A residual problem may require revising the data definition, adding explanatory variables, changing the seasonal period, or selecting a different model.

Forecast horizon and information sets

A one-step-ahead forecast predicts y_t from information available at t−1. A four-week-ahead forecast made at time t predicts y_t+4. The information set must exclude data that would not have been available when the forecast was supposedly issued. This principle prevents look-ahead bias, one of the most serious errors in forecasting evaluation.

For business use, define at least:

2. Data Preparation and Leakage Control

Time-series observations must be ordered by time. Randomly shuffling observations before splitting usually allows future patterns to influence model training and produces over-optimistic results. A chronological split might use early observations for training, later observations for validation, and the most recent observations for final testing.

Data preparation should address:

• Missing periods: distinguish zero demand from an unrecorded observation.
• Stockouts: observed sales may understate true demand when inventory was unavailable.
• Calendar effects: holidays, trading days, school terms, and month length can alter demand.
• Changing definitions: a product, customer, or accounting rule may change over time.
• Data revisions: historical values may be updated after the original forecast date.
• Exogenous variables: prices, promotions, weather, and economic indicators must be time-aligned.

A common leakage error is to calculate a feature using the entire dataset. For example, an average demand value used to predict January sales must not include February or later observations. Similarly, an imputation rule must be fitted using only information available at the forecast origin.

3. Baselines: The Essential First Comparison

A baseline is a transparent reference forecast. It answers a fundamental question: does a proposed model outperform a reasonable simple alternative? Baselines also reveal whether apparent predictive power comes from genuine structure or from leakage.

3.1 Naive forecast

The one-step naive forecast assumes that the next value equals the latest observed value:

For a multi-step forecast from origin T, the level forecast remains constant:

Here h is the forecast horizon and H is the maximum horizon. The method is appropriate as a benchmark when recent level is informative and no strong seasonal pattern is present. It is not necessarily appropriate for intermittent demand, rapidly changing systems, or series with pronounced seasonality.

3.2 Seasonal-naive forecast

If a series has a stable seasonal period m, the seasonal-naive forecast repeats the observation from the corresponding previous season:

For example, with weekly data and annual seasonality, m=52 may be used when 52 weeks is a reasonable operational approximation. With monthly data and yearly seasonality, m=12. For a forecast made at time T:

provided the historical seasonal counterpart exists. Seasonal-naive forecasting is particularly powerful when seasonal repetition dominates short-term changes.

3.3 Baseline example

Suppose weekly demand for the last four corresponding weeks was 120, 135, 128, and 142 units. The latest observed demand is 142 units, while demand in the same week last year was 150 units.

• A one-step naive forecast is 142 units.
• A seasonal-naive forecast with annual seasonality is 150 units.

Neither forecast claims to explain demand. Their value is diagnostic: a complex model should beat the relevant baseline on the evaluation period, preferably across the horizons that matter operationally.

4. Rolling-Origin and Time-Series Cross-Validation

4.1 Why ordinary cross-validation is unsuitable

Standard random k-fold cross-validation assumes that observations can be rearranged without violating the information structure. Time series do not satisfy this assumption. A model trained on randomly selected observations may indirectly use future data to predict the past.

Time-series cross-validation preserves order. It evaluates forecasts at several historical origins, each time training only on data available at that origin.

4.2 Rolling-origin evaluation

Let the origins be T₁, T₂, …, T_K. At origin T_i, fit the model using observations through T_i, then forecast the next H periods. The resulting errors are:

A growing-window design expands the training sample after each origin. A fixed-window or sliding design retains only the most recent w observations. Growing windows are useful when older data remain relevant; fixed windows may be preferable after structural change.

A valid evaluation must respect the forecast horizon. If the business needs eight-week forecasts, evaluating only one-step forecasts can select the wrong model.

4.3 Aggregating validation results

Suppose errors are collected over origins and horizons. A metric may be computed over all errors, by horizon, or by origin. Pooling all errors gives an overall view, while horizon-specific results reveal whether a model deteriorates at longer horizons. When origins have unequal numbers of observations, define the aggregation rule explicitly so that one period does not receive unintended weight.

Validation must also reflect the deployment frequency. A model used every week should normally be evaluated at historical weekly origins, including the same data-latency and forecast-update rules used operationally.

5. Accuracy Measures

No single metric captures every forecasting objective. Metrics must be interpreted with the scale, decision, and error costs in mind.

5.1 Mean absolute error

For n forecast errors:

MAE is expressed in the units of the target, such as units, dollars, or customers. It gives equal linear weight to each absolute error and is easy to communicate. It is often suitable when under- and over-forecasting have similar costs.

5.2 Root mean squared error

RMSE has the same units as the target but penalizes large errors more heavily than MAE. It is useful when occasional major misses are especially costly. Because squaring magnifies outliers, RMSE should be accompanied by diagnostic investigation.

5.3 Mean absolute percentage error

For nonzero actual values:

MAPE is unit-free and often intuitive as a percentage. It is undefined when y_i=0 and unstable when actual values are close to zero. Consequently, it is unsuitable without modification for intermittent or low-volume demand. Removing zero observations without reporting the change can also distort conclusions.

5.4 Mean absolute scaled error

MASE compares the model's MAE with the MAE of a benchmark, commonly the in-sample one-step naive forecast. For a nonseasonal benchmark:

Here N is the number of observations in the training sample used to calculate the scale. A MASE below 1 indicates that the evaluated forecasts have lower MAE than the benchmark, assuming the numerator and denominator are defined on compatible forecasting contexts. For seasonal data, a seasonal scaling denominator can be used:

The scaling choice must be documented. MASE is especially useful when comparing series measured on different scales.

5.5 Bias and mean error

Mean error, often called forecast bias under the convention used here, is:

A positive ME indicates average under-forecasting; a negative ME indicates average over-forecasting. Bias can be hidden by MAE or RMSE because positive and negative errors may cancel in ME while still being large individually. Report bias alongside an absolute-error measure.

Some organizations define bias as ∑(\hat{y}_i−y_i)/∑y_i, which reverses the sign and scales the result. Definitions must therefore be stated explicitly.

5.6 Business-weighted loss

Statistical error is not identical to business cost. Let c_U be the cost per unit of under-forecasting and c_O the cost per unit of over-forecasting. With the convention e_i=y_i−\hat{y}_i, a simple asymmetric loss is:

Average business loss is:

This formulation assumes linear costs and that the magnitude of shortage or excess is measured in target units. Real businesses may require nonlinear costs, capacity constraints, service-level penalties, salvage value, or inventory simulation. In those cases, evaluate forecasts through the relevant decision model rather than relying only on generic metrics.

5.7 Worked metric example

Actual demand is [100, 120, 80, 100], and forecasts are [90, 130, 100, 90]. Errors are [10, −10, −20, 10].

Absolute errors sum to 50, so:

Squared errors sum to 100 + 100 + 400 + 100 = 700, so:

The mean error is:

Thus the forecasts overpredict by 2.5 units on average under this convention, although the absolute errors are substantially larger.

For nonzero actuals, the percentage errors are 10%, 8.33%, 25%, and 10%, giving:

If under-forecasting costs 3 per unit and over-forecasting costs 1 per unit, under-forecasting occurs when e_i>0. Business losses are 30, 10, 0, and 30; average loss is 17.5 cost units per observation.

6. Residual Diagnostics

A residual is an observed forecast error. A good forecasting model does not require residuals to be exactly zero, but residuals should contain little predictable structure after accounting for the information available to the model.

6.1 Residual plots

Plot residuals against time and fitted values. Look for:

• Runs above or below zero: possible bias or level change.
• Trend: omitted trend or model drift.
• Repeating waves: unmodelled seasonality.
• Increasing spread: changing variance or scale effects.
• Isolated extremes: data errors, unusual events, or genuine shocks.

A histogram or density plot can reveal skewness, but normality is not a universal requirement for useful point forecasts. It becomes more relevant when constructing particular prediction intervals or applying statistical tests.

6.2 Autocorrelation

Residual autocorrelation means that errors are related across time. The lag-k autocorrelation is commonly estimated as:

where \bar{e} is the mean residual. Significant residual autocorrelation suggests that information remains unexploited, such as trend, seasonality, calendar structure, or dynamic dependence. A residual autocorrelation plot is useful, but statistical significance should be interpreted cautiously when many lags are examined.

6.3 Residuals versus forecast errors

Residuals from a fitted model may be calculated in-sample, whereas forecast errors are generated on unseen observations. Out-of-sample errors provide stronger evidence of operational performance. A model can fit historical data closely yet produce poor future forecasts because of overfitting or regime change.

6.4 Prediction intervals and calibration

A point forecast gives a central estimate; a prediction interval expresses uncertainty about a future observation. Interval quality should be assessed by coverage and sharpness. For example, a nominal 80% interval should contain approximately 80% of comparable future observations over a sufficiently large evaluation sample, subject to sampling variation. An interval that is always extremely wide may achieve coverage but be operationally unhelpful.

7. Selecting a Model for Practice

Model selection should consider:

1. Out-of-sample error at relevant horizons.
2. Performance relative to naive and seasonal-naive benchmarks.
3. Bias and error distribution, not only one average metric.
4. Business-weighted cost.
5. Stability across time and segments.
6. Interpretability and maintainability.
7. Data and computational requirements.
8. Forecast uncertainty and decision risk.

A small improvement in RMSE may not justify a model that is difficult to explain or that fails during data outages. Conversely, a model with slightly worse MAE may be preferable if it substantially reduces costly stockouts.

Forecast combinations can be useful when different models capture different aspects of the series. However, combinations must also be evaluated using rolling-origin validation; combining models does not eliminate leakage or structural bias.

8. Illustrative Case: Regional Grocery Replenishment

Background

A grocery retailer forecasts weekly demand for a chilled product across 40 stores. Forecasts are issued every Monday for the next four weeks. Excess inventory has a disposal cost, while shortages reduce sales and customer satisfaction.

Problem

The analytics team proposes a machine-learning model using promotions, price, weather, and lagged demand. Initial results show a MAPE of 7%, but the evaluation randomly shuffled the data. The team has not compared the model with seasonal-naive forecasting and has not distinguished stockout weeks from low-demand weeks.

Analysis

The retailer first defines the target as unconstrained demand where possible, records the four-week horizon, and reconstructs the information available at each historical Monday. Data are split chronologically. Weeks affected by stockouts are flagged rather than automatically treated as true zero demand.

The team evaluates three methods:

• Naive: latest weekly demand.
• Seasonal-naive: demand in the corresponding week of the previous year.
• Candidate model: lagged demand plus price, promotion, and calendar variables.

A rolling-origin design uses 20 historical forecast origins. At each origin, the model is refitted using only data then available. Results are reported by horizon and by store group. MAE, RMSE, ME, MASE, and business-weighted loss are calculated.

Approach and solution

The candidate model has lower overall MAE than the naive benchmark, but seasonal-naive performs better during holiday periods. The candidate model also has a negative ME, indicating over-forecasting. Because excess chilled inventory is costly, this is not automatically undesirable; however, the business-weighted loss shows that the model is too conservative in ordinary weeks and still misses promotional spikes.

The retailer adopts a hybrid policy: use the candidate model during promotion periods, seasonal-naive as a benchmark and fallback during data outages, and a calibrated adjustment for stores with persistent bias. The forecast dashboard displays point forecasts, prediction intervals, horizon-specific MAE, and stockout-adjusted diagnostics.

Outcome

The final evaluation shows improved business-weighted loss rather than merely improved statistical accuracy. Governance procedures require logging each forecast version, input data snapshot, manual override, reason for override, and subsequent outcome.

Learning points

1. A randomly shuffled evaluation can exaggerate performance.
2. Baselines are necessary for judging incremental value.
3. Metric choice must reflect inventory economics.
4. Stockouts can contaminate the target.
5. Forecast monitoring must continue after deployment.

9. Governance, Ethics, and Contemporary Practice

Forecasts influence staffing, credit, inventory, pricing, and public services. Poor forecasts can distribute resources unfairly or create feedback loops. Governance should include data lineage, reproducibility, access control, versioning, and documented assumptions.

When forecasts affect people, analysts should examine performance across relevant groups and locations. A model may have acceptable aggregate MAE while producing systematically larger errors for smaller regions or vulnerable populations. Ethical review should therefore consider consequence, not only statistical fit.

Artificial intelligence and machine learning can model nonlinear relationships and large numbers of predictors, but they do not remove the need for temporal validation. Automated feature generation can create leakage, and retraining pipelines can silently change results. Human oversight is particularly important when structural breaks, policy changes, or extraordinary events occur.

Contemporary forecasting practice increasingly emphasizes probabilistic forecasts, hierarchical reconciliation, forecast combinations, automated monitoring, and decision-focused evaluation. These developments strengthen—not replace—the foundational workflow presented here.

Best-Practice Checklist

Before approving a forecasting system, confirm that:

• The target, horizon, frequency, and decision are explicit.
• Data are time ordered and leakage has been tested.
• Naive and seasonal-naive baselines are reported.
• Validation simulates the real forecast origin and horizon.
• Metrics include at least one scale-dependent measure and bias.
• Zero and near-zero actuals are handled appropriately.
• Residuals are examined for dependence, trend, seasonality, and outliers.
• Business costs and service consequences are considered.
• Prediction intervals are evaluated where uncertainty matters.
• Deployment includes monitoring, versioning, and escalation rules.

Chapter Summary

A professional forecasting process begins with a clearly defined decision and information set. Chronological data preparation and leakage control are essential because future information must not influence historical forecasts. Naive and seasonal-naive methods provide transparent benchmarks against which more complex models must be judged.

Rolling-origin and time-series cross-validation simulate repeated forecasting decisions while preserving temporal order. MAE measures average absolute error, RMSE emphasizes large misses, MAPE expresses relative error but fails with zero or near-zero actuals, MASE scales error against a benchmark, and mean error reveals directional bias. Business-weighted loss connects forecast accuracy to the unequal costs of shortages and surpluses.

Residual diagnostics help identify remaining trend, seasonality, autocorrelation, changing variance, and unusual observations. Model selection should combine statistical accuracy, robustness, interpretability, operational cost, and governance. Forecasting is therefore not simply a model-fitting exercise; it is a monitored decision system.

Key Terms

Forecast origin: The time at which a forecast is generated.

Forecast horizon: The number of future periods being predicted.

Forecast error: The difference between actual and forecast value under a stated sign convention.

Naive forecast: A forecast that carries the latest observed value forward.

Seasonal-naive forecast: A forecast that repeats the value from the corresponding previous season.

Baseline: A simple reference method used to judge whether a model adds value.

Rolling-origin validation: Repeated evaluation in which training ends at successive historical forecast origins.

Growing window: Validation design in which all earlier observations remain in the training set.

Sliding window: Validation design using only a fixed number of recent observations.

Look-ahead bias: Evaluation bias caused by using information unavailable at the forecast date.

MAE: Mean absolute error, the average absolute forecast error.

RMSE: Root mean squared error, which penalizes large errors more strongly.

MAPE: Mean absolute percentage error, an average relative error measure.

MASE: Mean absolute scaled error, which compares error with a benchmark scale.

Bias: Average signed error under a stated convention.

Residual: An error remaining after a model has produced a fitted or forecast value.

Autocorrelation: Dependence between observations or residuals separated by a time lag.

Prediction interval: An interval intended to contain a future observation with stated probability.

Business-weighted loss: A loss function that assigns costs according to business consequences.

Data leakage: Use of information in model development that would not have been available operationally.

Review Questions

Short Answer Questions

1. Why should a forecasting project begin with the decision rather than the algorithm?
2. State the one-step naive forecasting equation and explain its interpretation.
3. Why is random k-fold cross-validation generally inappropriate for time series?
4. What does a positive mean error indicate under the convention e_t=y_t−\hat{y}_t?
5. Give one reason why MAPE can be unsuitable for intermittent demand.

Long Answer Questions

1. Explain a complete leakage-resistant forecasting workflow from target definition through deployment monitoring.
2. Compare growing-window and sliding-window rolling-origin validation, including situations in which each is appropriate.
3. Explain the differences among MAE, RMSE, MAPE, and MASE and discuss the strengths and limitations of each.
4. Describe how residual plots and autocorrelation analysis can reveal model inadequacy.
5. Discuss why business-weighted loss may lead to a different model choice from RMSE.

Analytical / Application Questions

1. A retailer has strong annual seasonality and monthly demand data. Design a validation experiment comparing naive, seasonal-naive, and a candidate model for a six-month horizon. Specify the origins, training windows, metrics, and decision rule.
2. A model has lower MAE than its benchmark but persistent negative mean error. Explain two possible business consequences and recommend diagnostic or corrective actions.
3. Actual values are [50, 0, 50] and forecasts are [45, 10, 60]. Explain why ordinary MAPE is problematic, calculate MAE, and recommend an alternative evaluation approach.

Multiple Choice Questions with Answers

Question 1

Which forecast is produced by the one-step naive method?

A. The average of all historical observations

B. The most recent observed value

C. The value from the same season two years earlier

D. The median of the validation sample

Correct Answer: B. The most recent observed value

Question 2

For monthly data with stable yearly seasonality, which seasonal period is usually the starting benchmark?

A. 2

B. 4

C. 12

D. 52

Correct Answer: C. 12

Question 3

What is the principal purpose of rolling-origin validation?

A. To randomly mix observations across time

B. To simulate repeated historical forecasting decisions

C. To guarantee normally distributed residuals

D. To remove all outliers automatically

Correct Answer: B. To simulate repeated historical forecasting decisions

Question 4

Which metric penalizes large errors more strongly because errors are squared before averaging?

A. MAE

B. MAPE

C. RMSE

D. ME

Correct Answer: C. RMSE

Question 5

Under the convention e_t=y_t−\hat{y}_t, what does a negative mean error indicate?

A. Average under-forecasting

B. Average over-forecasting

C. No forecast errors

D. Seasonality has been removed

Correct Answer: B. Average over-forecasting

Question 6

Why is MAPE undefined for an observation whose actual value is zero?

A. The forecast must also be zero

B. The error is always positive

C. The error is divided by zero

D. RMSE becomes negative

Correct Answer: C. The error is divided by zero

Question 7

A MASE below 1 generally indicates that the evaluated forecast:

A. Has lower scaled MAE than its stated benchmark

B. Has zero bias in every period

C. Has a MAPE of less than 1 percent

D. Always beats the benchmark at every horizon

Correct Answer: A. Has lower scaled MAE than its stated benchmark

Question 8

What pattern most directly suggests unmodelled seasonality in residuals?

A. A constant residual mean of zero

B. Repeating peaks and troughs at a regular lag

C. One isolated residual outlier

D. A smaller MAE than RMSE

Correct Answer: B. Repeating peaks and troughs at a regular lag

Question 9

If under-forecasting costs more than over-forecasting, which evaluation method is most directly aligned with that asymmetry?

A. Business-weighted loss

B. Unscaled residual histogram only

C. Random train-test splitting

D. Seasonal decomposition alone

Correct Answer: A. Business-weighted loss

Question 10

Which practice is an example of look-ahead bias?

A. Using only data available at each forecast origin

B. Evaluating forecasts at the operational horizon

C. Calculating a feature using future observations before making historical forecasts

D. Comparing a model with a naive benchmark

Correct Answer: C. Calculating a feature using future observations before making historical forecasts