Model Selection, Validation, and Ensembles

Introduction

Forecasting quality depends not only on the forecasting algorithm but also on how models are selected, evaluated, governed, and deployed. A model that fits historical demand exceptionally well may perform poorly when exposed to a promotion, stockout, competitor action, supply disruption, or structural change that was absent from its training data. Conversely, a simple seasonal-naïve forecast may outperform a sophisticated machine-learning system when the demand signal is stable, the data set is small, or the operational environment changes frequently.

Model selection is therefore a decision problem under uncertainty. The analyst must determine which forecasting method is sufficiently accurate, stable, interpretable, maintainable, and suitable for the decision being supported. This chapter develops a rigorous framework based on holdout samples, rolling-origin validation, benchmarks, residual diagnostics, champion–challenger testing, leakage prevention, and ensemble forecasting.

The central principle is that a model should be judged by its performance on data that reproduce the information available at the time the forecast would actually have been made. This principle is easy to state but difficult to implement in supply chains, where data are revised, inventory is censored by stockouts, promotions are planned in advance, and forecast horizons differ across decisions.

Learning Objectives

After completing this chapter, the reader should be able to:

1. Explain the difference between training, validation, and test data in time-series forecasting.
2. Design holdout and rolling-origin validation schemes that respect temporal order.
3. Use naïve and seasonal-naïve benchmarks to assess whether model complexity is justified.
4. Select and interpret forecast-error measures, including scale-dependent, percentage, and scaled metrics.
5. Diagnose residuals, bias, instability, leakage, and overfitting.
6. Design champion–challenger experiments for operational forecasting systems.
7. Explain when and how forecasts should be combined.
8. Evaluate model performance in relation to inventory, service, financial, and managerial consequences.

1. Forecasting as an Out-of-Sample Decision Problem

Suppose demand observed at period t is denoted by yₜ. A forecasting model uses information available at the forecast origin t to produce a forecast for a future period t+h:

Here, ŷₜ₊ₕ|ₜ is the forecast h periods ahead, Mₜ represents historical demand information available at time t, Xₜ represents available explanatory variables, and θ denotes estimated model parameters. The notation emphasizes that a forecast is conditional on an information set. A model must not use information that would not have been available at the time of forecasting.

The forecast error is:

The error has the same units as demand, such as units, kilograms, pallets, or dollars. Positive error indicates underforecasting, while negative error indicates overforecasting. For a set of n forecast errors, common summary measures include:

Mean absolute error (MAE) treats all absolute errors proportionally. Root mean squared error (RMSE) gives greater influence to large errors because errors are squared before averaging. Both require the same units as the dependent variable and are meaningful when observations are measured on a common scale.

Mean error, also called forecast bias in some operational settings, is:

A positive ME indicates systematic underforecasting under the error definition above. Mean error can be close to zero even when individual errors are large, because positive and negative errors cancel. It should therefore be reported alongside MAE or RMSE.

Percentage measures require care. Mean absolute percentage error is often written as:

MAPE is undefined when yᵢ = 0 and can become disproportionately large when actual demand is close to zero. For intermittent demand, it is often inappropriate. Weighted absolute percentage error can be written as:

WAPE is defined when total actual demand is positive, but it can be dominated by high-volume items and may conceal poor performance on low-volume items. No single metric is universally best; the metric should reflect the decision and the cost of error.

2. The Three-Way Data Logic: Training, Validation, and Test

A training sample is used to estimate model parameters. A validation sample is used to compare candidate models, tune hyperparameters, select features, or determine forecast settings. A final test sample is reserved for an unbiased assessment after model choices have been completed.

For time-series data, observations must normally remain in chronological order. Randomly shuffling observations can allow future conditions to influence the training sample and can produce an unrealistically optimistic estimate of performance.

A basic chronological split is:

The test set must not be repeatedly consulted during model development. If analysts examine test performance, modify the model, and examine the same test period again, the test set has effectively become a validation set. A new untouched test period is then needed.

In supply-chain applications, the split should also reflect the operational horizon. A monthly planning process might require a 12-month validation horizon, whereas a daily replenishment process may require several weeks or months. If the business experiences annual seasonality, a validation period shorter than one full seasonal cycle may give misleading results.

3. Holdout Samples and Their Limitations

A holdout evaluation reserves the most recent observations for validation or testing. For example, a firm may train on January 2019 through December 2023 and evaluate forecasts for January through December 2024. This approach is intuitive and preserves temporal order.

A holdout is useful when:

• the historical sample is sufficiently long;
• the recent period is representative of expected future conditions;
• model development will not repeatedly consume the holdout;
• the forecast horizon is clearly defined.

However, a single holdout can be fragile. A period containing an unusual port closure, pandemic shock, promotion, or stockout may make one model appear better or worse by chance. A model may also perform well in the selected holdout but fail in another season.

Holdout evaluation is especially problematic when the number of time observations is small. Reserving too much data reduces the amount available for parameter estimation, while reserving too little produces an imprecise evaluation. There is no universal percentage rule. The split should be determined by forecast horizon, seasonal cycle, structural changes, and the amount of data required by the candidate models.

4. Rolling-Origin Validation

Rolling-origin validation, also called time-series cross-validation or walk-forward validation, evaluates forecasts at multiple historical origins. At each origin, the model uses only information available up to that date and forecasts a future horizon.

Consider origins t₁, t₂, …, tⱼ and a fixed horizon h. The set of errors is:

The model can be re-estimated after each origin, as it would be in practice, or parameters can be held fixed if that matches the intended deployment process.

Two common designs are expanding-window and sliding-window validation.

In an expanding-window design, the training set grows over time. In a sliding-window design, old observations are discarded as new observations enter. The correct choice is empirical and operational: it should reflect how the forecasting system will be maintained and how quickly demand relationships change.

Rolling-origin validation can evaluate multiple horizons. For example, a distribution centre may assess h = 1, 2, …, 12 weeks separately. A model that is strong one week ahead may be weak at twelve weeks. Aggregating all horizons into a single score can hide this distinction.

A horizon-specific MAE is:

If different horizons have different business importance, a weighted score may be used:

where Lₕ is the loss at horizon h, wₕ ≥ 0 is the business weight assigned to that horizon, and the weights sum to 1. The weighting scheme must be documented because it embeds managerial priorities into model selection.

5. Benchmarks and the Value of Simplicity

A candidate model should be compared with a benchmark. Without a benchmark, an error value has little meaning. A complex model with a monthly MAE of 1,000 units is not useful if a seasonal-naïve model achieves an MAE of 700 units.

Important benchmarks include:

1. Naïve forecast: the next forecast equals the latest observed value.

1. Seasonal-naïve forecast: the forecast equals the observation from the corresponding previous season. For seasonal period m:

when the required historical observation exists.

1. Mean forecast: future demand equals the historical mean, appropriate mainly for stable series without meaningful trend or seasonality.
2. Drift forecast: a linear extrapolation between the first and latest observations, used in some forecasting contexts when a persistent trend is plausible.
3. Business-rule forecast: an existing forecast generated by planners, spreadsheets, or current software.

A relative performance measure compares a candidate loss Lₘ with benchmark loss L_b:

A value below 1 indicates that the candidate has lower loss than the benchmark under the selected metric and evaluation design. Relative measures are useful across products with different scales, but they are not automatically comparable when the benchmark or metric changes.

Simple models often outperform complex models because they have lower estimation variance, fewer parameters, lower maintenance requirements, and greater robustness to noisy predictors. Complexity is justified only when it produces consistent, decision-relevant improvement after validation and operational costs are considered.

6. Preventing Overfitting and Leakage

Overfitting

Overfitting occurs when a model captures random variation or historical idiosyncrasies rather than relationships that persist into the future. It is more likely when the model has many parameters relative to the amount of information available, when features are selected using the test set, or when repeated experimentation is not controlled.

A model can have low training error and high validation error. This is a classic generalization problem. Regularization, feature reduction, pooling across related items, simpler model structures, and genuinely independent validation can reduce overfitting.

Leakage

Data leakage occurs when information unavailable at forecast creation enters model training or evaluation. Common supply-chain examples include:

• using end-of-month sales totals to forecast earlier days in that month;
• using a revised inventory record that was corrected after the forecast date;
• including realized future promotion performance rather than the promotion plan available at the time;
• using sales during a stockout as if they represented unconstrained demand;
• calculating a rolling average that accidentally includes future observations;
• selecting features based on correlations calculated over the full data set, including the test period;
• using a contemporaneous variable whose publication is delayed beyond the forecast deadline.

Leakage prevention requires point-in-time data reconstruction. Every predictor should have an availability timestamp, not merely an event timestamp. Feature engineering, scaling, imputation, outlier treatment, and model fitting should be performed within each training fold. A transformation estimated on the full data set can leak information even if the transformation appears harmless.

7. Model Diagnostics

Accuracy scores answer how large errors are; diagnostics help explain why errors occur and whether the model is trustworthy.

Residual bias

Plot or summarize residuals by time, product, location, customer segment, promotion status, and forecast horizon. Persistent positive errors indicate underforecasting under the convention eₜ = yₜ − ŷₜ. Bias may be acceptable in a cost-asymmetric system only if it is deliberate and economically justified; otherwise it signals a correctable problem.

Residual autocorrelation

If residuals are correlated over time, the model may have omitted trend, seasonality, lagged effects, or calendar structure. A common diagnostic is the autocorrelation at lag k:

Here, ē is the mean residual and n is the number of residuals. The statistic is dimensionless. Residual autocorrelation does not by itself prove that a particular model is wrong, but it indicates remaining temporal structure.

Heteroscedasticity and scale effects

Error variance may increase with demand volume. A model can appear inaccurate for high-volume items simply because the units are larger. Conversely, percentage metrics can exaggerate errors for low-volume items. Evaluation should therefore consider both absolute and relative performance, with segmentation by volume and value.

Stability and operational robustness

A model should be evaluated across time periods, products, and scenarios. A small average improvement accompanied by extreme failures during promotions may be unacceptable. Diagnostic review should include error distributions, worst-period performance, forecast revision behaviour, and sensitivity to missing or delayed inputs.

8. Champion–Challenger Testing

The champion is the model currently used in production. Challengers are alternative models evaluated against it under a controlled protocol. The purpose is not to replace the champion whenever a challenger wins one metric once; it is to determine whether the challenger provides reliable and material improvement.

A sound champion–challenger process includes:

1. a fixed evaluation period or rolling-origin protocol;
2. identical data cut-off rules;
3. predefined primary and secondary metrics;
4. segmentation by product, location, horizon, and demand type;
5. minimum improvement thresholds;
6. operational checks such as latency, explainability, data availability, and failure handling;
7. a documented decision and monitoring plan.

Suppose the champion has loss Lc and the challenger has loss Ld. The relative improvement is:

This expression is meaningful when Lc > 0 and the same loss definition is used for both models. Statistical significance may be assessed using paired forecast errors because both models forecast the same observations. However, statistical significance is not equivalent to commercial significance. A small improvement may be statistically detectable but too small to justify system migration.

A challenger should often be deployed in shadow mode. It generates forecasts without influencing decisions, allowing the organization to test data pipelines, run time, monitoring, and exception handling before full adoption.

9. Forecast Combination and Ensembles

Different models may capture different aspects of demand. One model may represent seasonality well, another may respond to promotions, and a third may be robust during structural change. Combining their forecasts can reduce model-specific risk.

The simplest combination is an average of K forecasts:

A weighted combination is:

with constraints:

The weights may be equal, estimated from validation performance, or determined by constrained optimization. Weight estimation must occur using training and validation information only. If weights are optimized on the final test period, the ensemble is overfit to the test data.

The benefit of combination depends on forecast errors. For two forecasts with errors e₁ and e₂, an equal-weight combined error is (e₁ + e₂)/2. Its variance is:

If errors are not perfectly correlated, the combination may have lower variance than either forecast. The covariance term is crucial: combining highly similar models may provide little benefit, while combining models with complementary errors can improve robustness. A combination does not guarantee improvement; biased or unstable component forecasts can harm performance.

Useful ensemble structures include:

• Simple average: transparent and often difficult to beat consistently.
• Weighted average: assigns greater influence to historically stronger models.
• Clustered or segmented ensemble: uses different combinations for product families or demand regimes.
• Forecast reconciliation ensemble: combines forecasts across hierarchical levels while preserving aggregation coherence.
• Stacking: trains a second-level model to map component forecasts to the outcome. Stacking requires especially careful out-of-fold predictions to prevent leakage.

Ensembles should be evaluated against the best individual model and benchmarks. Their additional complexity, monitoring burden, and explainability requirements must be justified.

10. Decision-Centered Model Selection

Forecast accuracy is a means rather than the final objective. A forecast supports decisions about inventory, capacity, labour, transportation, procurement, and customer commitments. Model selection should therefore reflect the cost of forecast errors.

If underforecasting and overforecasting have different consequences, symmetric MAE may be insufficient. A simple asymmetric loss is:

where e = y − ŷ, cᵤ is the cost per unit of underforecasting, and cₒ is the cost per unit of overforecasting. This formula assumes nonnegative error costs and treats the two directions differently. It does not itself model inventory dynamics, service-level constraints, or nonlinear shortage costs, but it can align evaluation more closely with managerial priorities.

For inventory decisions, the forecast distribution may matter more than the point forecast. A model that produces a slightly less accurate mean forecast but well-calibrated uncertainty estimates may support better safety-stock decisions. Calibration should therefore be assessed where probabilistic forecasts are used. For an interval forecast [Lₜ, Uₜ], coverage is the proportion of future observations falling within the interval. A nominal 90% interval should achieve approximately 90% coverage over a stable evaluation sample, although coverage can vary by segment and horizon.

11. Illustrative Case: Selecting a Forecasting Process for a Regional Distributor

Background

The fictional NorthRiver Distribution Company supplies packaged beverages to supermarkets. It forecasts weekly demand for 1,200 stock-keeping units across 18 distribution centres. Forecasts are generated eight weeks ahead for procurement and one week ahead for replenishment. Demand is seasonal, promotions are frequent, and stockouts occur for a subset of products.

Problem

NorthRiver’s analytics team proposes a gradient-boosting model using price, promotion, weather, holidays, inventory, and competitor variables. The current system uses seasonal-naïve forecasts adjusted by planners. A preliminary report claims that the new model reduces historical MAPE by 28%. Senior managers question the result because the model was tested on a random sample of rows and because MAPE is very high for low-volume products.

Analysis

The governance team first reconstructs the information available at each historical forecast date. It discovers that the promotion field contained the realized discount in some records, although only the planned discount would have been available operationally. It also discovers that sales during stockouts were treated as true demand, causing the model to learn that constrained sales were low demand.

The team defines three benchmarks: naïve, seasonal-naïve with a 52-week seasonal period, and the existing planner forecast. It then establishes an expanding-window rolling-origin design. Origins are selected monthly, and forecasts are evaluated for horizons of 1, 4, and 8 weeks. The final six months are preserved as an untouched test period.

The evaluation uses MAE and WAPE for scale-based reporting, bias by product class, and an asymmetric shortage-sensitive loss for priority items. The team also reports performance separately for regular demand, promotion periods, and stockout-affected periods.

The corrected results are summarized below.

The original claimed improvement disappears once leakage is removed. The gradient-boosting model remains useful, but its advantage is small at the eight-week horizon. The equal ensemble improves both horizons and reduces extreme errors during unusual promotions. However, the team rejects a fully automated replacement because promotion data are sometimes late and the model has weak performance for new products.

Approach and solution

NorthRiver adopts a segmented champion–challenger process. The planner forecast remains the champion for new products and for centres with unreliable promotion data. The ensemble becomes the challenger for established products with adequate history. It runs in shadow mode for three months, with automated monitoring of data freshness, bias, WAPE, stockout flags, and forecast revisions.

The company also introduces a point-in-time feature store, records the forecast creation timestamp, and requires that every model version be reproducible. Forecasts are reviewed using decision-relevant measures rather than MAPE alone.

Outcome

After shadow testing, the ensemble is deployed for the established-product segment. Procurement reports fewer severe underforecasts, while planners retain override authority for known commercial events. The improvement is not attributed solely to algorithmic sophistication; it results from leakage controls, appropriate validation, segmentation, and combining forecasts with complementary strengths.

Learning points

1. Random row-level splitting is inappropriate for most time-series forecasting problems.
2. Leakage can create impressive but fictitious model improvements.
3. Benchmarks reveal whether complexity adds value.
4. Performance should be examined by horizon, demand regime, and operational segment.
5. An ensemble can improve robustness without requiring a single universally superior model.
6. Governance and data availability are part of forecasting performance.

12. Technology, AI, and Contemporary Practice

Modern forecasting systems may use automated model selection, gradient boosting, deep neural networks, probabilistic forecasting, foundation models, and cloud-based feature stores. These tools can be valuable when there is sufficient data, stable infrastructure, and a clear evaluation protocol. They do not eliminate the need for temporal validation or leakage prevention.

Automated machine learning systems should expose the data split, feature-availability logic, candidate models, hyperparameter search space, and evaluation metrics. Black-box performance claims are insufficient for high-consequence supply-chain decisions. Generative AI can assist with documentation, code review, and diagnostic interpretation, but generated code and explanations require independent testing. AI systems may also introduce privacy, security, intellectual-property, and accountability risks.

Contemporary practice increasingly emphasizes probabilistic and hierarchical forecasting. A distribution centre may need forecasts that reconcile from SKU to category and region, while inventory decisions require uncertainty estimates. Model selection should consequently consider coherence, calibration, revision stability, and decision value in addition to point accuracy.

13. Best-Practice Checklist

Before approving a forecasting model, an organization should be able to answer yes to the following questions:

• Is the forecast origin and horizon explicitly defined?
• Are training, validation, and test periods chronologically separated?
• Does rolling-origin validation represent the deployment process?
• Are naïve, seasonal-naïve, and current-process benchmarks included?
• Have all features been checked for point-in-time availability?
• Are stockouts, returns, cancellations, and data revisions handled explicitly?
• Are metrics appropriate for volume, intermittency, and business cost?
• Are residual bias, autocorrelation, stability, and extreme errors monitored?
• Has the challenger been compared with the champion under identical conditions?
• Were ensemble weights estimated without using the final test period?
• Are model performance and data quality monitored after deployment?
• Is there a documented rollback and override process?

Challenges and Limitations

No validation design perfectly predicts the future. Structural breaks, policy changes, new competitors, climate events, and supply disruptions can invalidate historical relationships. Rolling-origin validation can also be computationally expensive, especially for large product portfolios and models requiring extensive retraining.

Metrics can conflict. A model may reduce MAE while increasing bias, improve average performance while worsening high-priority items, or improve statistical accuracy without reducing inventory cost. Aggregated metrics can conceal segment-level failure. Human overrides may improve performance in some circumstances but make evaluation difficult if override decisions are not recorded.

Ensembles introduce additional governance and monitoring requirements. Weighted ensembles may become stale when model performance changes. Stacking can leak information if its training data are not generated out of fold. Finally, model selection is vulnerable to repeated experimentation: even a correctly held-out test set can be overused when many alternatives are tried and only the best result is reported.

Chapter Summary

Model selection in demand forecasting is fundamentally an out-of-sample evaluation problem. Chronological holdouts preserve temporal order but may be sensitive to one unusual period; rolling-origin validation provides a broader assessment by simulating repeated historical forecast decisions. Training, validation, and test data must remain distinct, and every transformation and feature must respect point-in-time availability.

Naïve and seasonal-naïve methods are essential benchmarks. Complexity is justified only when it produces stable, decision-relevant improvement over appropriate benchmarks. MAE, RMSE, bias, WAPE, percentage measures, and asymmetric losses answer different questions and should be selected according to demand characteristics and business consequences.

Residual diagnostics reveal bias, autocorrelation, scale effects, and instability that aggregate accuracy scores can hide. Champion–challenger testing provides a controlled path for improving production systems, while shadow deployment reduces operational risk. Forecast combinations can reduce variance when component-model errors are not perfectly correlated, but ensemble weights must be learned without test-set leakage.

Effective forecasting governance connects statistical validation with supply-chain decisions, data quality, monitoring, explainability, and accountability. In many applications, the best model is not the most complex model; it is the model or ensemble that performs reliably under realistic information constraints and supports better decisions.

Key Terms

Benchmark: A reference forecasting method used to judge whether a candidate model adds value.

Champion model: The model currently deployed in operational use.

Challenger model: An alternative model evaluated against the champion.

Data leakage: Use of information in training or evaluation that would not have been available at forecast creation.

Expanding window: A rolling-origin design in which the training sample grows over time.

Forecast bias: Systematic tendency to overforecast or underforecast.

Forecast origin: The date or period at which a forecast is created.

Holdout sample: Chronologically reserved observations used for validation or testing.

MAE: Mean absolute error, the average absolute forecast error.

MAPE: Mean absolute percentage error; problematic when actual demand is zero or near zero.

Model ensemble: A forecast formed by combining outputs from multiple models.

Overfitting: Learning historical noise rather than relationships that generalize.

Point-in-time availability: The condition that a predictor is available in the form known at the forecast origin.

Residual: The difference between actual demand and forecast demand.

Rolling-origin validation: Repeated evaluation using multiple historical forecast origins.

Seasonal-naïve forecast: A forecast that repeats demand from the corresponding previous seasonal period.

Sliding window: A validation design using a fixed-length, moving training sample.

Test set: Data reserved for final evaluation after model selection.

WAPE: Weighted absolute percentage error based on total absolute error divided by total actual demand.

Review Questions

Short Answer Questions

1. Why is random shuffling usually inappropriate when validating a time-series forecasting model?
2. Distinguish between a validation set and a final test set.
3. What is the principal difference between an expanding-window and a sliding-window design?
4. Why should seasonal-naïve forecasts be included as benchmarks?
5. Give two examples of data leakage in supply-chain forecasting.

Long Answer Questions

1. Explain how rolling-origin validation can be designed for a business requiring forecasts at multiple horizons. Discuss training-window choice, horizon-specific metrics, and computational considerations.
2. Compare MAE, RMSE, MAPE, WAPE, and bias. Explain when each may be useful and identify important limitations.
3. Describe a complete champion–challenger governance process, from candidate definition through shadow deployment and post-deployment monitoring.
4. Explain the statistical and managerial reasons that simple forecasting methods can outperform complex machine-learning models.
5. Derive the logic behind forecast combination and discuss how error covariance affects the potential value of an ensemble.

Analytical / Application Questions

1. A company reports that a new model reduces MAPE by 35% using randomly sampled historical rows. Design an investigation to determine whether the claimed improvement is valid, including leakage checks, benchmarks, validation design, and metrics.
2. A seasonal-naïve model has WAPE of 20%, while a machine-learning model has WAPE of 18% but is unavailable for 15% of forecast runs because promotion data arrive late. Recommend a deployment approach and justify it using accuracy, reliability, and operational risk.
3. Two component forecasts have equal bias, but their errors are only weakly correlated. Explain why an equal-weight ensemble may outperform either component and identify the validation evidence needed before deployment.

Multiple Choice Questions with Answers

Question 1

Which validation design best preserves the information constraints of time-series forecasting?

A. Randomly shuffle all observations before splitting

B. Use chronological rolling-origin validation

C. Use only the training error

D. Select the model with the most parameters

Correct Answer: B. Use chronological rolling-origin validation

Question 2

What is the primary purpose of a benchmark forecast?

A. To guarantee zero forecast error

B. To provide a reference for judging model value

C. To replace all diagnostic analysis

D. To increase the number of model parameters

Correct Answer: B. To provide a reference for judging model value

Question 3

Under the convention eₜ = yₜ − ŷₜ, a positive mean error generally indicates:

A. Systematic underforecasting

B. Systematic overforecasting

C. Perfect calibration

D. No forecast error

Correct Answer: A. Systematic underforecasting

Question 4

Which situation is an example of data leakage?

A. Using last year's demand to forecast this year's seasonal period

B. Using a promotion plan known before the forecast date

C. Using realized future promotion performance unavailable at forecast creation

D. Using a seasonal-naïve benchmark

Correct Answer: C. Using realized future promotion performance unavailable at forecast creation

Question 5

Why can RMSE be substantially larger than MAE for a model?

A. RMSE ignores large errors

B. RMSE squares errors before averaging

C. MAE uses squared errors

D. RMSE is always measured as a percentage

Correct Answer: B. RMSE squares errors before averaging

Question 6

Which statement about MAPE is correct?

A. It is always defined when demand is zero

B. It is unaffected by very small actual values

C. It can be problematic for zero or near-zero demand

D. It has the same units as demand

Correct Answer: C. It can be problematic for zero or near-zero demand

Question 7

In an expanding-window validation scheme, the training sample:

A. Always contains only the latest observation

B. Grows as later forecast origins are evaluated

C. Is randomly resampled at every origin

D. Contains future observations beyond the origin

Correct Answer: B. Grows as later forecast origins are evaluated

Question 8

Why can combining forecasts reduce forecast variance?

A. Component errors must be perfectly correlated

B. The combination removes the need for validation

C. Component errors may not be perfectly correlated

D. An ensemble always eliminates bias

Correct Answer: C. Component errors may not be perfectly correlated

Question 9

What is the role of a challenger in a champion–challenger process?

A. To act as an alternative evaluated under controlled conditions

B. To overwrite the champion immediately

C. To provide only training data

D. To eliminate operational monitoring

Correct Answer: A. To act as an alternative evaluated under controlled conditions

Question 10

Which practice best prevents test-set overuse?

A. Choose the model with the best test result after every experiment

B. Reserve the test set for final assessment after model choices are complete

C. Use the test set to estimate ensemble weights repeatedly

D. Publish only the most favorable test metric

Correct Answer: B. Reserve the test set for final assessment after model choices are complete