Ensemble Learning

Introduction and Context

An ensemble is a collection of predictive models whose outputs are combined to produce a single prediction. Ensemble learning is based on a practical observation: several models that make different errors can collectively outperform any one of them. The central challenge is therefore not merely to create many models, but to create models that are both reasonably accurate and sufficiently diverse.

Ensembles are particularly important when individual learners are unstable, meaning that small changes in the training data produce substantially different models. Decision trees are a classic example. A single deep tree can fit complex relationships but may have high variance and overfit. Ensembles reduce this weakness by averaging, weighting, or sequentially correcting multiple learners.

Ensemble methods are widely used in credit-risk assessment, fraud detection, medical prediction, customer retention, remote sensing, manufacturing quality control, and search and recommendation systems. They are also important in machine-learning competitions and production systems because they often provide strong predictive performance on tabular data.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain why combining learners can improve generalization.

2. Distinguish bagging, random forests, boosting, voting, and stacking.

3. Describe and apply the main training procedures for these methods.

4. Interpret the trade-offs among performance, complexity, robustness, and interpretability.

5. Select suitable ensemble methods for practical machine-learning problems.

6. Evaluate ensembles using appropriate validation procedures and performance measures.

7. Recognize risks involving leakage, bias, fairness, reproducibility, and explainability.

The Core Idea: Accuracy and Diversity

Suppose an ensemble contains predictions from models $f_1, f_2, \ldots, f_M$. For regression, a simple average is

$$

\hat{y}(x)=\frac{1}{M}\sum_{m=1}^{M}f_m(x),

$$

where $x$ is an input and $M$ is the number of models. For classification, voting or probability averaging is commonly used. The average is useful when model errors are not perfectly correlated.

To see why, consider regression errors $e_m=f_m(x)-y$. If every model has error variance $\sigma^2$ and every pair has correlation $\rho$, the variance of the average is

$$

\mathrm{Var}\left(\frac{1}{M}\sum_{m=1}^{M}e_m\right)

=\sigma^2\left[\rho+\frac{1-\rho}{M}\right].

$$

This expression assumes equal error variances and equal pairwise error correlations. When $\rho<1$, increasing $M$ reduces the second term; when $\rho=1$, averaging gives no variance reduction. Thus, an ensemble benefits from diversity, but diversity alone is not enough: models must also be better than random or systematically useful.

A useful bias–variance interpretation is that bagging primarily reduces variance, while boosting can reduce bias as well as variance, depending on its configuration. These are tendencies rather than absolute rules. A sufficiently complex boosting model can overfit, and a random forest can sometimes reduce both variance and effective bias compared with a single tree.

Important terminology includes:

• Base learner: An individual model used by an ensemble.
• Weak learner: A model that is only modestly predictive but performs better than a relevant baseline.
• Aggregation: Combining predictions from multiple learners.
• Homogeneous ensemble: An ensemble using the same model family, such as many trees.
• Heterogeneous ensemble: An ensemble combining different model families.
• Diversity: The extent to which learners make different errors.
• Out-of-bag observation: A training example not selected for a particular bootstrap sample.

Bagging: Bootstrap Aggregation

Bagging, short for bootstrap aggregation, trains multiple base learners on different bootstrap samples of the training set. A bootstrap sample of size $n$ is produced by sampling $n$ observations with replacement from a dataset containing $n$ observations. Some observations occur several times, while others are omitted.

For $M$ bootstrap samples, the procedure is:

1. Draw a bootstrap sample from the training data.

2. Train a base learner on that sample.

3. Repeat for $M$ learners.

4. Average regression predictions or use majority voting for classification.

Bagging is especially effective for high-variance learners, such as unpruned decision trees. The individual trees may differ because they receive different samples, and averaging reduces the influence of unusual training observations.

Bootstrap inclusion and out-of-bag data

For a dataset of $n$ observations, the probability that a particular observation is not selected in one draw is $1-1/n$. The probability that it is not selected in all $n$ draws is

$$

\left(1-\frac{1}{n}\right)^n.

$$

As $n$ becomes large, this approaches $e^{-1}\approx0.368$. Therefore, a bootstrap sample contains approximately $63.2\%$ unique observations and leaves approximately $36.8\%$ out of bag. The exact percentage varies for finite samples.

Out-of-bag observations can be used to estimate prediction performance. For each training observation, predictions are aggregated only from trees for which that observation was out of bag. This provides an internal estimate without requiring a separate validation set, although a separate test set remains valuable for final unbiased evaluation.

Strengths and limitations of bagging

Bagging is simple, parallelizable, and relatively robust to noise. It often improves a single tree substantially. Its limitations include increased memory and inference cost, reduced direct interpretability, and limited improvement when base learners are already stable or highly correlated. Bagging does not automatically solve class imbalance, label noise, or data leakage.

Random Forests

A random forest combines bootstrap aggregation with random feature selection. At each split of each tree, the algorithm considers only a randomly selected subset of the available predictors rather than all predictors.

A typical random-forest procedure is:

1. Draw a bootstrap sample for each tree.

2. Grow a decision tree, usually using recursive binary splits.

3. At each node, randomly select $m$ candidate features from the total of $p$ features.

4. Choose the best split among those candidates according to an impurity or loss criterion.

5. Aggregate predictions across trees.

For classification, common split criteria include Gini impurity and entropy. For a node containing class proportions $p_1,\ldots,p_K$, Gini impurity is

$$

G=1-\sum_{k=1}^{K}p_k^2.

$$

It is zero when all observations belong to one class and is larger when classes are mixed. Entropy is

$$

H=-\sum_{k=1}^{K}p_k\log_2 p_k,

$$

where terms with $p_k=0$ are treated as zero. Both criteria measure node impurity, but they can rank candidate splits somewhat differently.

Random feature selection reduces correlation among trees. This is important when a few strong predictors would otherwise cause most trees to make similar splits. Random forests can model nonlinearities, interactions, and mixed-scale predictors without feature standardization. They are often strong default models for structured tabular data.

Hyperparameters

Important random-forest hyperparameters include:

• Number of trees: more trees usually stabilize predictions, with diminishing returns.
• Maximum tree depth: limits complexity and computational cost.
• Minimum observations per leaf: larger values smooth predictions.
• Number of candidate features per split: controls the accuracy–diversity trade-off.
• Bootstrap usage and sample size: affect the relationship among trees.
• Class weights or sampling strategy: useful for imbalanced classification.

Increasing the number of trees generally does not cause the same type of overfitting associated with repeatedly increasing the depth of a single tree, although computational cost rises and performance may eventually plateau. Excessively deep trees can still be expensive and may capture noisy structure.

Feature importance and interpretation

Random forests commonly report impurity-based importance, calculated from the reduction in split impurity attributable to a feature. This measure can be biased toward continuous variables or categorical variables with many levels. Permutation importance measures the performance decrease after randomly permuting a feature in validation data. It is often more informative, but correlated predictors can divide or obscure importance.

Partial dependence plots and accumulated local effects can describe average relationships, while local explanation methods can describe individual predictions. These tools are aids to interpretation, not guarantees of causal explanation.

Boosting

Boosting constructs an ensemble sequentially. Each new learner is trained to improve the current ensemble, often by emphasizing poorly predicted observations or approximating the gradient of a loss function.

In contrast to bagging, where learners can be trained independently, boosting creates dependence among learners. This can produce highly accurate models but makes the method more sensitive to hyperparameters, noise, and implementation details.

AdaBoost

For binary classification, AdaBoost begins with equal observation weights. At each iteration, a weak classifier is trained. Misclassified observations receive greater weight, so the next learner focuses more strongly on difficult cases. If learner $m$ has weighted error $\epsilon_m$, its contribution is commonly set to

$$

\alpha_m=\frac{1}{2}\ln\left(\frac{1-\epsilon_m}{\epsilon_m}\right),

$$

provided $0<\epsilon_m<0.5$. The final classifier is the sign of the weighted sum of learner outputs. The formula assumes the standard binary AdaBoost setting and a learner better than random guessing under the current weights. If the error is at least $0.5$, the learner does not provide a positive contribution under this formulation.

AdaBoost can perform well with shallow trees, but it may be sensitive to mislabeled observations because repeatedly emphasizing difficult examples can cause the ensemble to focus on noise.

Gradient Boosting and Gradient-Boosted Trees

Gradient boosting generalizes the idea of sequential correction by fitting each new learner to the negative gradient of a differentiable loss function. Let the current model after $m-1$ stages be $F_{m-1}(x)$. For observation $i$, the pseudo-residual is

$$

r_{im}=-\left[\frac{\partial L(y_i,F(x_i))}{\partial F(x_i)}\right]_{F=F_{m-1}},

$$

where $L$ is the chosen loss function. A weak learner $h_m(x)$ is fitted to these pseudo-residuals, and the model is updated as

$$

F_m(x)=F_{m-1}(x)+\nu\gamma_m h_m(x).

$$

Here, $\nu$ is the learning rate, $\gamma_m$ is a step size selected by minimizing the loss, and $h_m$ is the new learner. In some implementations, the step size is incorporated into the learner or learning-rate update.

For squared-error regression, $L(y,F)=(y-F)^2/2$, so the negative gradient is $y_i-F_{m-1}(x_i)$: the ordinary residual. For binary classification, a logistic loss is commonly used, and the pseudo-residual differs from the raw class error.

Gradient-boosted trees use shallow or moderately deep decision trees as learners. Important hyperparameters include the number of trees, learning rate, maximum depth or leaf count, minimum leaf size, row subsampling, column subsampling, and regularization terms.

Learning rate and number of trees

A smaller learning rate usually requires more trees. This can improve generalization because the model makes smaller, more controlled updates, but training and prediction become more expensive. The product of learning rate and number of trees is not a complete equivalence: tree depth, subsampling, regularization, and loss interactions also matter.

Modern implementations

Libraries such as XGBoost, LightGBM, and CatBoost introduce engineering and algorithmic improvements, including regularization, efficient split finding, histogram-based training, handling of missing values, row and feature subsampling, and specialized categorical-feature treatment. Their exact behavior differs, so results should not be assumed interchangeable. Hyperparameters and defaults must be documented for reproducibility.

Gradient-boosted trees are often among the strongest methods for small- and medium-sized tabular datasets. They can, however, overfit noisy data, require careful tuning, and be computationally demanding when large ensembles or extensive search are used.

Voting Ensembles

Voting combines predictions from different trained models. In hard voting, each classifier casts a class vote and the most frequent class is selected. In soft voting, class probabilities are averaged or weighted and the class with the largest combined probability is selected.

For models producing class probabilities $p_m(y=k\mid x)$, weighted soft voting can be written as

$$

\hat{y}=\arg\max_k\sum_{m=1}^{M}w_m p_m(y=k\mid x),

$$

where $w_m\geq0$. If weights are normalized to sum to one, the expression is a convex combination, but normalization is not necessary for the resulting argmax as long as all weights are nonnegative and scaled consistently.

Soft voting requires reasonably calibrated probabilities. A model may rank cases well but produce probabilities that are too extreme or too conservative. Calibration methods such as Platt scaling or isotonic regression should be fitted using validation data without leakage.

Voting is attractive because it is transparent at the system level and can combine complementary model families, such as logistic regression, a tree ensemble, and a neural network. Its weakness is that poorly chosen or strongly correlated models may add little value.

Stacking

Stacking, or stacked generalization, trains a meta-model to combine predictions from base models. The critical requirement is that the meta-model must be trained on out-of-fold predictions rather than predictions made on the same data used to fit the base models.

A typical procedure is:

1. Divide the training set into folds.

2. For each base model, train on all but one fold and predict the held-out fold.

3. Combine the held-out predictions across folds to create an out-of-fold prediction for every training observation.

4. Train a meta-model on these predictions, optionally including original features.

5. Retrain base models on the full training data.

6. Apply base models to new data and pass their predictions to the meta-model.

If base predictions for the meta-model are generated in-sample, the meta-model sees unrealistically accurate outputs and may learn to exploit overfitting. This is a form of target leakage.

A simple stacking system might use a regularized logistic regression as the meta-model for classification. Regularization is useful because base predictions can be correlated. Stacking can improve performance when models have complementary strengths, but it increases complexity, maintenance requirements, and opportunities for leakage.

A Worked Comparison: Selecting an Ensemble

Consider a binary credit-default problem with 50,000 observations, 40 predictors, missing values, nonlinear relationships, and an imbalanced target in which 8% of customers default. A sensible experimental design is:

• Hold out a final test set before model development.
• Use stratified cross-validation on the remaining training data.
• Compare a regularized logistic regression, a single decision tree, a random forest, gradient-boosted trees, soft voting, and stacking.
• Report area under the ROC curve, precision–recall area, recall at a specified false-positive constraint, calibration, and expected cost.

Accuracy alone is unsuitable because a classifier predicting “no default” for every customer would achieve 92% accuracy while detecting no defaults. Suppose the institution assigns a cost of $5,000 to a missed default and $500 to an unnecessary intervention. A model should be evaluated according to expected operational cost, not merely a threshold-free metric.

A plausible interpretation of results might be:

• Logistic regression is fastest and easiest to explain, but may miss nonlinear effects.
• A single deep tree is interpretable locally but unstable and comparatively inaccurate.
• A random forest is robust and strong with limited tuning, but probability calibration may need attention.
• Gradient boosting gives the highest validation discrimination but requires careful control of depth, learning rate, and early stopping.
• Voting may improve stability if the models make complementary errors.
• Stacking may produce a small additional gain, but the gain must justify higher governance and deployment complexity.

These conclusions cannot be asserted from algorithm names alone; they must be established using leakage-resistant validation and an appropriate business metric.

Illustrative Case Study: Predictive Maintenance in Manufacturing

Background

A manufacturer operates hundreds of industrial pumps. Sensors record vibration, temperature, pressure, operating hours, and maintenance history. The goal is to predict whether a pump will require a major repair within the next 30 days.

Problem

Failures are rare, sensor readings are correlated, and operating conditions differ between production lines. Missing sensor values occur during communication interruptions. A false negative can cause costly downtime, whereas a false positive causes an inspection and modest service expense.

Analysis

A single decision tree provides an accessible baseline but changes considerably when a few failure examples are added. Bagging and random forests reduce this instability. Gradient-boosted trees can capture subtle interactions, such as a combination of rising vibration and high temperature. However, random splitting of time-indexed records could leak future information into training, producing unrealistically strong results.

Approach and solution

The team uses a chronological training, validation, and test split. Features are constructed only from sensor readings available before each prediction time. A random forest and gradient-boosted tree model are tuned using time-aware validation. Class weighting is considered because failures are rare. Threshold selection is based on the cost of missed failures and inspections rather than on a default probability threshold of 0.5.

A soft-voting model combining the calibrated forest, boosted trees, and a regularized linear model is evaluated. Stacking is considered only after out-of-fold predictions are generated correctly. Feature importance is supplemented with partial dependence and case-level review by maintenance engineers.

Outcome

The ensemble identifies substantially more upcoming failures than the baseline at the same inspection capacity. Engineers discover that some apparent predictive power came from a maintenance-code field entered after a failure had already been diagnosed. Removing that field reduces validation performance but produces a trustworthy estimate. The final system is deployed with monitoring for sensor drift, alert volume, calibration, and subgroup performance across production lines.

Learning points

1. Validation design can matter more than the choice between two strong algorithms.

2. Ensembles do not eliminate leakage or biased labels.

3. Operational costs should determine thresholds and evaluation metrics.

4. Explanations must be reviewed with domain experts.

5. Deployment requires monitoring because data distributions and equipment behavior change.

Comparing Ensemble Methods

| Method | Main mechanism | Typical strengths | Main limitations |

|---|---|---|---|

| Bagging | Bootstrap samples and aggregation | Variance reduction, parallel training, robustness | Limited bias reduction; larger model |

| Random forest | Bagging plus random feature subsets | Strong tabular baseline, nonlinear interactions, relatively low tuning burden | Less compact and less transparent than one tree |

| AdaBoost | Sequential emphasis on errors | Effective with weak learners | Sensitive to noisy labels and outliers |

| Gradient boosting | Sequential loss-gradient correction | Excellent predictive accuracy and flexible losses | Tuning, runtime, and overfitting risk |

| Voting | Direct combination of model outputs | Simple heterogeneous integration | Depends on model quality and calibration |

| Stacking | Learned meta-level combination | Can exploit complementary models | Leakage risk and high complexity |

Performance should be compared across several dimensions:

• Predictive quality: discrimination, error, calibration, and decision utility.
• Complexity: number of learners, depth, features, training time, and inference latency.
• Robustness: sensitivity to noise, missingness, distribution shift, and random seeds.
• Interpretability: transparency of rules, global relationships, local explanations, and probability reliability.
• Maintainability: ease of retraining, monitoring, debugging, and version control.

There is no universally best ensemble. A small, well-calibrated model may be preferable to a marginally more accurate but opaque stack in a regulated application.

Practical Workflow and Best Practices

1. Establish a meaningful baseline, including a simple statistical or business rule.

2. Define the target, prediction time, and permissible information before feature engineering.

3. Split data according to its structure: stratified for ordinary classification, grouped when entities repeat, and chronological for forecasting or operational prediction.

4. Tune hyperparameters inside cross-validation, not on the final test set.

5. Use pipelines so preprocessing is fitted only on the relevant training folds.

6. Address class imbalance using class weights, resampling, or threshold selection, while preserving realistic validation data.

7. Calibrate probabilities when decisions depend on risk estimates.

8. Use early stopping with a validation set that reflects deployment conditions.

9. Compare ensembles against simpler models on performance, cost, and governance.

10. Record data versions, random seeds, software versions, features, hyperparameters, and evaluation procedures.

11. Monitor drift, calibration, error rates, subgroup performance, and changes in the input pipeline after deployment.

Challenges, Ethics, and Governance

Ensembles can encode historical discrimination because they learn from labels and features shaped by institutional decisions. High predictive performance does not establish fairness, legality, or causality. A model used for lending, employment, insurance, or healthcare should be assessed for disparate error rates, accessibility, privacy, and applicable regulation.

Feature importance must not be interpreted as proof that a feature causes an outcome. Correlated variables and proxy variables can produce misleading explanations. Removing a sensitive feature does not necessarily remove its influence if correlated proxies remain.

Data leakage is a major governance risk. Leakage may result from future variables, duplicate records across splits, target-derived features, or incorrectly fitted preprocessing. Stacking introduces an additional leakage risk through in-sample base predictions.

Security and reliability issues also matter. Attackers may manipulate input features, exploit model thresholds, or infer sensitive information from predictions. Production systems should include access controls, audit logs, model documentation, fallback procedures, and human review where consequences are significant.

Contemporary Developments

Current ensemble practice increasingly combines highly optimized gradient-boosting libraries with automated hyperparameter search, calibrated decision systems, and explainability tools. Distributed training and histogram-based algorithms make large-scale boosting practical. Categorical-feature handling and missing-value strategies continue to improve.

Ensembles are also used beyond conventional tabular prediction. In computer vision and language systems, model averaging, snapshot ensembles, and mixture-of-experts architectures combine specialized predictors. However, the fundamental principles remain the same: complementary errors, controlled complexity, valid evaluation, and responsible deployment.

Chapter Summary

Ensemble learning combines multiple predictive models to improve generalization. Its success depends on balancing individual accuracy with diversity of errors. Bagging trains models independently on bootstrap samples and primarily reduces variance. Random forests add random feature selection, reducing correlation among decision trees. Boosting trains models sequentially; AdaBoost emphasizes difficult observations, while gradient boosting fits learners to loss-function gradients. Gradient-boosted trees are highly effective for structured data but require careful tuning and validation.

Voting combines predictions directly, whereas stacking learns how to combine base-model outputs. Stacking must use out-of-fold predictions to avoid leakage. Ensemble performance should be evaluated using metrics appropriate to the task, including calibration and decision cost for risk-sensitive applications. Random forests are often robust and convenient; boosting may achieve higher accuracy; voting and stacking can exploit complementary models but increase system complexity.

No ensemble is automatically reliable. Data leakage, class imbalance, label noise, distribution shift, unfairness, poor calibration, and weak monitoring can undermine even a technically sophisticated model. Professional practice requires leakage-resistant validation, comparison with simple baselines, reproducible pipelines, interpretability appropriate to the context, and post-deployment governance.

Key Terms

Ensemble: A collection of models whose predictions are combined.

Base learner: An individual model used within an ensemble.

Bagging: Bootstrap aggregation of independently trained learners.

Bootstrap sample: A sample drawn with replacement from a dataset.

Out-of-bag observation: An observation omitted from a particular bootstrap sample.

Random forest: An ensemble of randomized decision trees using bootstrap samples and feature subsampling.

Feature subsampling: Selecting a random subset of features for candidate splits.

Boosting: Sequential construction of learners that improve an existing ensemble.

AdaBoost: A boosting method that increases emphasis on misclassified observations.

Gradient boosting: Sequential fitting to negative loss gradients.

Learning rate: A factor controlling the size of each boosting update.

Weak learner: A modestly predictive learner used as a component of an ensemble.

Voting: Combining predictions directly through class votes or probability aggregation.

Soft voting: Combining predicted probabilities, often with weights.

Stacking: Training a meta-model to combine base-model predictions.

Out-of-fold prediction: A prediction for an observation made by a model that did not train on that observation.

Calibration: Agreement between predicted probabilities and observed frequencies.

Diversity: Difference in predictions or errors among ensemble members.

Data leakage: Use of information unavailable at prediction time or improperly shared across evaluation splits.

Feature importance: A measure intended to indicate the contribution of features to predictions, not necessarily causation.

Review Questions

Short Answer Questions

1. Why does diversity among ensemble learners matter?

2. What is an out-of-bag observation, and how can it be used?

3. How does random feature selection improve a random forest?

4. Distinguish hard voting from soft voting.

5. Why are out-of-fold predictions required in stacking?

Long Answer Questions

1. Explain bagging and random forests, including their effects on variance, correlation, interpretability, and computational cost.

2. Compare AdaBoost with gradient boosting in terms of training process, loss correction, and sensitivity to noise.

3. Discuss how learning rate, tree depth, number of trees, and subsampling influence gradient-boosted trees.

4. Develop a validation and model-selection plan for an imbalanced classification problem using at least three ensemble methods.

5. Evaluate the ethical and governance challenges of deploying an ensemble in lending, healthcare, or employment.

Analytical / Application Questions

1. A company has a highly accurate boosted-tree model and a less accurate but well-calibrated logistic model. Design a procedure to determine whether soft voting improves operational decisions.

2. A stacking system reports exceptional validation performance, but its base models were trained on all training observations before generating meta-model inputs. Diagnose the problem and propose a corrected procedure.

3. In a predictive-maintenance task with time-dependent records and rare failures, specify the data split, evaluation metrics, threshold-selection method, and monitoring plan you would use before deployment.

Multiple Choice Questions with Answers

Question 1

What is the primary purpose of bootstrap sampling in bagging?

A. To create different training sets so that model errors can be averaged

B. To guarantee that every training observation appears exactly once

C. To remove all noisy observations before training

D. To force every base learner to use the same decision tree structure

Correct Answer: A. To create different training sets so that model errors can be averaged

Question 2

Approximately what proportion of unique observations is expected in a bootstrap sample of size n drawn from n observations?

A. 36.8%

B. 50.0%

C. 63.2%

D. 100%

Correct Answer: C. 63.2%

Question 3

How does a random forest differ from ordinary bagging of decision trees?

A. It uses no bootstrap samples

B. It randomly restricts candidate features at tree splits

C. It trains only one shallow tree

D. It requires all trees to have identical predictions

Correct Answer: B. It randomly restricts candidate features at tree splits

Question 4

For class proportions p1 through pK in a node, which expression is the Gini impurity?

A. Σ pk

B. −Σ pk log2(pk)

C. 1 + Σ pk²

D. 1 − Σ pk²

Correct Answer: D. 1 − Σ pk²

Question 5

In standard binary AdaBoost, what happens when a learner has weighted error below 0.5?

A. It receives a positive ensemble weight

B. It is discarded automatically

C. Its weight must be negative

D. It receives zero weight

Correct Answer: A. It receives a positive ensemble weight

Question 6

What does gradient boosting fit at each stage?

A. A random label permutation

B. The negative gradient of the loss with respect to current predictions

C. Only the observations predicted correctly

D. A model using features excluded by all previous learners

Correct Answer: B. The negative gradient of the loss with respect to current predictions

Question 7

Which voting method combines class-probability outputs?

A. Hard voting

B. Majority-label voting

C. Soft voting

D. Bootstrap voting

Correct Answer: C. Soft voting

Question 8

Why must stacking use out-of-fold predictions for training its meta-model?

A. To ensure base models have access to the test set

B. To prevent the meta-model from learning from overly optimistic in-sample predictions

C. To eliminate the need for base models

D. To make all base models identical

Correct Answer: B. To prevent the meta-model from learning from overly optimistic in-sample predictions

Question 9

Which issue is most directly associated with using impurity-based feature importance in tree ensembles?

A. It cannot be calculated for classification

B. It is always causal

C. It can favor continuous variables or categorical variables with many levels

D. It requires probability calibration first

Correct Answer: C. It can favor continuous variables or categorical variables with many levels

Question 10

A dataset contains repeated records over time, and future sensor values would not be available at prediction time. Which validation design is most appropriate?

A. Randomly split all records without restrictions

B. Use a chronological split that respects prediction time

C. Evaluate only on the training data

D. Use the same observations for preprocessing and final testing

Correct Answer: B. Use a chronological split that respects prediction time