Decision Trees and Ensemble Learning

Introduction

Business analytics frequently involves prediction problems in which managers need not only accurate forecasts but also understandable reasons for those forecasts. A bank may need to decide whether a loan applicant is likely to default, a retailer may predict whether a customer will respond to an offer, and a manufacturer may estimate whether a machine will fail soon. Decision trees and ensemble methods are especially useful in these settings because they can model nonlinear relationships, interactions among variables, and mixtures of numerical and categorical data.

A decision tree represents a sequence of questions that divides observations into increasingly homogeneous groups. A single tree is easy to explain, but it can be unstable and may overfit. Ensemble learning addresses these weaknesses by combining many models. Random forests reduce variance by averaging diverse trees, whereas boosting constructs trees sequentially so that later trees focus on earlier errors. The resulting models often provide strong predictive performance, although usually with less direct interpretability than a single tree.

This chapter develops the theory and practice of decision trees, random forests, and boosting methods for business prediction. It also explains model tuning, feature importance, partial dependence, and SHAP-based interpretation. The central professional challenge is to balance predictive accuracy, interpretability, operational cost, fairness, and governance.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain how decision trees partition a feature space and make predictions.
2. Calculate and interpret impurity measures used for classification and regression trees.
3. Compare pruning, regularization, random forests, and boosting.
4. Select appropriate evaluation metrics and validation procedures for business problems.
5. Tune tree-based models while controlling overfitting and leakage.
6. Interpret models using feature importance, partial dependence, and SHAP-based methods.
7. Evaluate tree-based systems for fairness, stability, transparency, and managerial usefulness.

1. Decision Trees as Predictive Models

A decision tree consists of internal decision nodes, branches, and terminal nodes or leaves. At an internal node, the model applies a rule such as “income ≤ $50,000” or “customer segment = business.” Each observation follows a path through the tree until it reaches a leaf. The prediction is then based on the observations assigned to that leaf.

For classification, a leaf commonly predicts the most frequent class or the estimated class probability. For regression, a leaf commonly predicts the mean target value of observations in that leaf.

A tree partitions the input space into regions. If the features are represented by x = (x₁, x₂, …, x_p), a tree creates regions R₁, R₂, …, R_J and predicts a constant value within each region. The partition is usually formed through binary splits, although some implementations support other forms of split.

Classification impurity

A split is desirable when it produces child nodes that are more homogeneous in their target classes than the parent node. Two widely used impurity measures are the Gini impurity and entropy.

Here, t denotes a node, K is the number of classes, and p_k|t is the proportion of observations in node t belonging to class k. Gini impurity is zero when every observation belongs to one class and is larger when classes are mixed.

Entropy is defined as:

The convention 0 log₂ 0 = 0 is used. Entropy is measured in bits when the logarithm is base 2. It is zero for a pure node. For a binary classification problem with equal class proportions, entropy is 1 bit and Gini impurity is 0.5.

Suppose a parent node contains 100 customers: 60 renew a subscription and 40 do not. Its Gini impurity is:

A candidate split produces a left child containing 50 customers, of whom 40 renew, and a right child containing 50 customers, of whom 20 renew. The left-child impurity is 1 − (0.8² + 0.2²) = 0.32, and the right-child impurity is also 0.32. The weighted impurity after the split is:

The impurity reduction, or Gini gain, is 0.48 − 0.32 = 0.16. The split improves class separation under the Gini criterion.

More generally, the weighted impurity of a split is:

where n is the number of observations at the parent node, n_L and n_R are the numbers in the left and right children, and I is the selected impurity measure. The best split minimizes I_split, equivalently maximizing impurity reduction.

Regression impurity

For regression, a common splitting criterion is the reduction in squared error. Within a leaf, the prediction minimizing squared error is the sample mean:

where n_R is the number of observations in region R, y_i is the observed target, and ŷ_R is the regional prediction. The residual sum of squares in a region is:

A candidate split is useful when the sum of the children’s RSS values is smaller than the parent’s RSS. Squared-error trees are sensitive to extreme target values, so alternative criteria such as absolute error may be useful when robustness is important.

2. Constructing and Controlling a Tree

A typical tree-growing procedure is greedy. At the current node, the algorithm examines candidate features and thresholds, selects the split producing the greatest impurity reduction, and repeats this process recursively. Greedy means that the algorithm optimizes the next split rather than searching all possible complete trees. This makes training practical but does not guarantee a globally optimal tree.

Important hyperparameters include:

An unrestricted tree can memorize training observations. This is overfitting: training error is low, but performance on new data deteriorates. Regularization for trees is usually structural rather than based on a penalty added to a smooth coefficient model. Limiting depth, increasing minimum leaf size, and pruning are common approaches.

Cost-complexity pruning selects a subtree by balancing training error and tree size:

Here, T is a tree, R(T) is its empirical error, |T| is the number of terminal nodes, and α ≥ 0 controls the penalty for complexity. The formula is a selection criterion, not a universal guarantee of generalization. The value of α should be chosen using validation data or cross-validation.

A single tree has several strengths: it handles nonlinear effects, can model interactions automatically, needs little feature scaling, and produces rules that business users can inspect. Its weaknesses include instability, sensitivity to small data changes, poor extrapolation in regression, and a tendency to favor dominant or high-cardinality variables under some split-selection procedures.

3. Evaluation for Business Prediction

Model evaluation must reflect the business decision, not merely a statistical preference. For classification, accuracy is:

where TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives. Accuracy can be misleading when the positive class is rare.

Precision and recall are:

Precision measures the reliability of positive predictions; recall measures the proportion of actual positives detected. The F1 score is their harmonic mean:

These metrics assume that a positive prediction has been defined using a classification threshold. Changing the threshold changes the confusion matrix and can alter the business trade-off.

For probabilistic classifiers, log loss evaluates the quality of predicted probabilities:

For binary outcomes, y_i ∈ {0,1} and p_i is the predicted probability of class 1. Lower log loss is better. It penalizes confident incorrect predictions strongly.

For regression, mean absolute error is:

and root mean squared error is:

MAE is in the units of the target and treats errors linearly. RMSE is also in target units but penalizes large errors more heavily. Neither metric is automatically superior; the choice depends on the cost of errors.

Random train-test splitting is inappropriate when observations are time-dependent, repeated by customer, or clustered by store. A credit model should generally be tested on later applications, not randomly mixed applications from the same period. Customer-level grouping may be required to prevent the same customer appearing in both training and validation data.

4. Random Forests

A random forest is an ensemble of decision trees trained with two principal sources of randomness:

1. Each tree is trained on a bootstrap sample of the training data.
2. At each split, the tree considers only a random subset of available features.

For regression, the forest prediction is commonly the average of tree predictions:

where B is the number of trees and f_b(x) is the prediction of tree b. For classification, the ensemble may use majority voting or average class probabilities, depending on the implementation.

Averaging reduces variance when individual trees are imperfect but not perfectly correlated. If each tree prediction has variance σ² and pairwise correlation ρ, the variance of an average of B trees is approximately:

This expression assumes equal variances and a common pairwise correlation. As B increases, the second term decreases, but the correlation term remains. Thus, increasing the number of trees cannot eliminate error caused by highly correlated trees. Random feature selection helps reduce correlation.

Random forests are usually less interpretable than one tree but often more accurate and stable. They can estimate out-of-bag performance because each bootstrap sample leaves some observations unused for a particular tree. These unused observations can provide an internal validation estimate, although a separate time-based or external test set remains important for realistic business assessment.

A known caution concerns impurity-based feature importance. A feature may appear important because it has many possible split points or categories, even if its substantive predictive value is limited. Permutation importance offers an alternative: after fitting the model, randomly permute one feature in validation data and measure the deterioration in performance. It should be computed on held-out data and interpreted cautiously when features are correlated.

5. Boosting Methods

Boosting builds an ensemble sequentially. Each new weak learner attempts to improve the current ensemble. Unlike random forests, where trees are trained largely independently and then averaged, boosting emphasizes systematic correction of prior errors.

AdaBoost

In binary classification, AdaBoost assigns greater weight to observations misclassified by earlier learners. A new learner receives a weight related to its weighted error. Conceptually, the final model gives more influence to learners that perform better than random guessing. AdaBoost can be effective but may be sensitive to noisy labels and extreme observations.

Gradient boosting

Gradient boosting treats prediction as an additive model:

Here, F₀(x) is the initial prediction, h_m(x) is the tree added at iteration m, M is the number of boosting iterations, and η is the learning rate. The trees are usually shallow and are fitted to an approximation of the negative gradient of the chosen loss function.

For squared-error regression, the negative gradient is the residual:

Thus, a new tree approximates the remaining errors. For classification, the working targets depend on the selected loss, often logistic loss rather than raw residuals.

The principal boosting hyperparameters are learning rate, number of trees, tree depth or leaf complexity, subsampling fraction, and regularization parameters. A smaller learning rate often requires more trees. Early stopping evaluates performance on validation data and stops training when additional trees no longer improve the selected metric.

Modern gradient-boosting implementations, including histogram-based and categorical-feature-aware systems, can be highly competitive for tabular business data. However, the convenience of a powerful implementation does not remove the need for leakage control, calibration checks, fairness analysis, and monitoring.

6. Comparing Tree-Based Methods

A single tree may be preferable when a policy requires transparent rules, when the dataset is small, or when a human decision-maker must verify each branch. A random forest is useful when stability and a strong baseline are priorities. Boosting is attractive when predictive performance is central and the team can support careful tuning, validation, and explanation.

Performance differences should be evaluated statistically and operationally. A small improvement in AUC may not justify substantially greater implementation complexity, latency, or compliance risk. Conversely, a modestly more accurate model may create substantial value in high-volume decisions.

7. Model Tuning and Reliable Workflow

A defensible workflow begins with a clearly defined target, prediction time, decision horizon, and cost of errors. The analyst should document which information was available when the prediction would have been made. Variables created after the outcome, or variables that directly encode the outcome, create target leakage.

A practical workflow is:

1. Define the business decision and target.
2. Establish a realistic training, validation, and test design.
3. Prepare features using transformations fitted only on training data.
4. Train a simple baseline and a single tree.
5. Tune random forest and boosting hyperparameters.
6. Select a threshold or operating point based on business costs.
7. Evaluate discrimination, calibration, stability, fairness, and operational performance.
8. Explain predictions and document limitations.
9. Deploy with monitoring for drift and performance deterioration.

Grid search tests a predefined set of hyperparameter combinations. Random search samples combinations and may be more efficient when only a few hyperparameters strongly affect performance. Bayesian optimization can use previous trials to choose promising configurations, but it requires careful validation design to avoid overfitting the validation process.

Cross-validation estimates generalization performance by repeatedly fitting a model on training folds and evaluating it on held-out folds. For time-dependent data, use forward-chaining or rolling-origin validation. Hyperparameter tuning must occur inside the training process; using the test set repeatedly for selection turns the test set into a de facto validation set.

Class imbalance requires more than changing an algorithm. Options include class weights, threshold adjustment, carefully designed resampling, and metrics such as recall, precision, PR-AUC, or expected financial value. Synthetic sampling should be applied only within training folds and should not create unrealistic records.

8. Explaining Tree Ensembles

Feature importance

Global feature importance summarizes how much a feature contributes across the model. Impurity importance sums the weighted impurity reductions attributable to splits involving that feature. It is fast but can be biased by feature scale structure, cardinality, and correlation.

Permutation importance measures the change in a validation metric after a feature is permuted. If performance declines substantially, the feature contains information used by the model. With correlated predictors, permuting one variable may understate importance because a correlated substitute remains available. Grouped permutation or domain-based feature sets can provide a more meaningful analysis.

Importance is not causality. A feature can be highly predictive because it is a proxy for another factor, because it records a process consequence, or because it reflects a historical policy.

Partial dependence

Partial dependence describes the average model prediction as a selected feature varies while the distribution of other features is averaged over. For feature x_j, the population partial dependence function is conceptually:

Here, f is the fitted model, z is a selected value of feature x_j, and X_−j denotes all other features. In a sample, the expectation is approximated by averaging predictions over observed values of the other features.

Partial dependence is useful for identifying nonlinear patterns, such as increasing predicted churn risk up to a service-delay threshold. However, it can be misleading when x_j is strongly correlated with other features, because it may evaluate unrealistic combinations of values. Individual conditional expectation curves show how predictions vary for individual observations and can reveal heterogeneity hidden by an average curve.

SHAP-based explanations

SHAP methods use cooperative game theory to allocate a prediction among features. For observation i, an additive explanation can be represented as:

Here, f(x_i) is the prediction, φ₀ is a baseline value, and φ_ij is the contribution assigned to feature j for observation i. A positive contribution raises the prediction relative to the baseline; a negative contribution lowers it.

SHAP values are conditional on the chosen background or reference distribution and the model. Different dependence assumptions can produce different attributions, especially with correlated variables. They explain model behavior, not why the real-world outcome occurred. A SHAP explanation that says “late payment history raised predicted default risk” does not establish that late payment caused default.

For governance, explanations should be combined with data lineage, policy documentation, counterfactual reasoning, and human review. Explanations must not be used to imply precision that the model does not possess.

9. Illustrative Case: Subscription Churn Prediction

Background

A fictional telecommunications company, Northstar Connect, wants to identify customers likely to cancel within the next 60 days. The company can offer retention incentives, but each offer has a cost. Unnecessary offers reduce margins, while missed high-risk customers reduce lifetime value.

Problem

The analytics team has 80,000 historical customer-month records. Features include tenure, monthly charge, service tickets, payment history, contract type, data usage, and recent service interruptions. The target is cancellation within 60 days. The team must avoid using information recorded after the prediction date.

The positive class represents 14% of records. Therefore, accuracy alone is inadequate. The business estimates that a missed churner costs $180 in expected contribution margin, an unnecessary incentive costs $25, and a correctly retained customer generates $120 in incremental expected value after the incentive.

Analysis and approach

The team uses the first 18 months for training, the following three months for validation, and the final three months for testing. Customer-level duplicates are managed so that records do not allow future information to leak backward. A logistic regression baseline, a pruned tree, a random forest, and a gradient-boosting model are compared.

The selected evaluation measures are PR-AUC, recall at the available campaign capacity, calibration, and expected value. The team uses class weights during training and chooses the final probability threshold on the validation period, not on the test period.

The single tree is easiest for campaign managers to inspect, but it is unstable across time windows. The random forest improves stability. Gradient boosting achieves the strongest validation PR-AUC, but its advantage over the random forest is modest on the final time-based test period. Calibration is improved using a separate calibration procedure because ranking quality and probability accuracy are different properties.

The final operating threshold is selected so that approximately 8,000 customers can be contacted. The team examines SHAP summaries and discovers that recent service interruptions, repeated service tickets, and month-to-month contracts are major contributors. Partial dependence suggests that risk rises sharply after several unresolved tickets, but the team checks this pattern against customer-service records before turning it into an operational rule.

Outcome

A pilot campaign compares model-selected customers with the existing broad campaign. The pilot tracks incremental retention using a randomized control group. Model performance alone is not treated as evidence that incentives cause retention. The pilot finds that customers with high predicted risk respond differently by contract type, leading to separate treatment policies rather than one universal offer.

Learning points

1. A predictive model must be aligned with the intervention and its capacity.
2. Time-based validation is essential when customer behavior and business processes change.
3. Strong ranking performance does not guarantee calibrated probabilities or profitable decisions.
4. Feature explanations identify model patterns but do not prove causal relationships.
5. A controlled experiment is required to assess incremental campaign impact.

10. Ethics, Governance, and Deployment

Tree ensembles can reproduce historical discrimination. If past lending or service decisions were biased, a model trained on those outcomes may learn and operationalize that bias. Sensitive attributes may be excluded, yet proxy variables such as location, occupation, language, or device type may still encode protected characteristics.

Fairness assessment should be defined for the decision context. Possible measures include group-specific selection rates, false-positive rates, false-negative rates, calibration, and error costs. These measures can conflict, particularly when base rates differ across groups. Analysts should not claim that a model is “fair” based on one statistic alone.

Governance should include:

• documented purpose, scope, and prohibited uses;
• data provenance and feature definitions;
• validation design and performance by relevant subgroup;
• threshold and intervention documentation;
• human escalation and appeal procedures;
• monitoring for drift, calibration loss, and changing error rates;
• version control for data, code, and model artifacts;
• review of privacy, security, and applicable regulation.

Feature importance and SHAP plots should be communicated with uncertainty and limitations. A model explanation is not a justification for an adverse decision if the organization cannot verify data quality or provide a meaningful review process.

11. Contemporary Developments and Best Practices

Gradient-boosting systems remain strong choices for structured tabular data, while automated machine-learning platforms increasingly search preprocessing, algorithms, and hyperparameters. Such systems can accelerate experimentation but may conceal leakage, optimize an unsuitable metric, or produce models that the organization cannot govern.

Monotonic constraints can enforce domain expectations, such as ensuring that risk does not decrease when a verified debt burden increases, although constraints must be substantively justified. Quantile or distributional forests can support prediction intervals rather than only point predictions. Conformal prediction can provide prediction sets or intervals under stated assumptions about data exchangeability, offering a useful complement to point forecasts.

Best practice is not simply to select the most accurate model. It is to build a reproducible decision system that is accurate enough, interpretable enough, stable under realistic changes, legally and ethically defensible, and economically valuable. Model cards, data sheets, experiment tracking, and post-deployment monitoring support this objective.

Chapter Summary

Decision trees recursively partition data into regions that are relatively homogeneous in their target values. Classification trees commonly use Gini impurity or entropy, while regression trees commonly minimize squared error or another loss. Trees are flexible and interpretable but can be unstable and prone to overfitting, making depth limits, minimum leaf sizes, and pruning important.

Random forests reduce variance by averaging diverse bootstrap-trained trees and randomly restricting candidate features at each split. Boosting builds trees sequentially to correct prior errors and can achieve excellent predictive performance, but it requires careful tuning and validation. Business evaluation should use metrics and thresholds connected to error costs, operational capacity, and decision value.

Feature importance, partial dependence, individual conditional expectation, and SHAP-based methods help explain tree models. None establishes causality, and correlated features can complicate interpretation. Reliable applications require leakage prevention, time-appropriate validation, calibration analysis, fairness assessment, governance, and monitoring after deployment.

Key Terms

Decision tree: A recursive model that partitions observations through decision rules.

Node: A point in a tree where observations are split or predicted.

Leaf: A terminal node that produces a prediction.

Gini impurity: A classification impurity measure based on squared class proportions.

Entropy: A classification impurity measure based on the logarithm of class probabilities.

Impurity reduction: The improvement in homogeneity produced by a candidate split.

Pruning: Removing tree branches to reduce complexity and overfitting.

Cost-complexity pruning: A method balancing empirical error against tree size.

Random forest: An ensemble of randomized, bootstrap-trained decision trees.

Bootstrap sample: A sample created by drawing observations with replacement.

Out-of-bag observation: An observation not included in a particular bootstrap sample.

Boosting: Sequential construction of learners that focus on improving previous predictions.

Gradient boosting: Boosting based on successive approximations to a loss-function gradient.

Learning rate: The scale applied to each new boosting learner.

Early stopping: Ending iterative training when validation performance stops improving.

Permutation importance: Importance estimated from performance loss after feature permutation.

Partial dependence: Average model prediction as a feature varies over a reference population.

SHAP value: An additive attribution of a model prediction to a feature.

Target leakage: Use of information unavailable at the time the prediction would be made.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Review Questions

Short Answer Questions

1. What is the purpose of a split in a classification decision tree?
2. Define Gini impurity and explain when it equals zero.
3. How does a random forest reduce the variance of a single tree?
4. What is the role of the learning rate in gradient boosting?
5. Why can permutation importance be preferable to impurity-based importance?

Long Answer Questions

1. Compare a single decision tree, random forest, and gradient-boosting model in terms of accuracy, stability, interpretability, and tuning requirements.
2. Explain how time-based validation differs from random cross-validation and why it matters in business forecasting.
3. Discuss overfitting in tree models and evaluate structural methods for controlling it.
4. Explain partial dependence and SHAP-based explanations, including their assumptions and limitations.
5. Develop a governance framework for deploying a churn or credit-risk model based on tree ensembles.

Analytical / Application Questions

1. A parent node contains 100 observations, with 60 in class 1 and 40 in class 0. A proposed split creates two equal children, each with class proportions 0.8 and 0.2. Calculate the parent Gini impurity, the weighted child impurity, and the impurity reduction.
2. A company can contact only 10% of customers for a retention campaign. Explain how you would choose a classification threshold and which evaluation metrics you would report.
3. A boosting model identifies location, device type, and service plan as its most important variables. Design an analysis to determine whether these features represent legitimate predictive information, leakage, or potentially unfair proxies.

Multiple Choice Questions with Answers

Question 1

Which condition corresponds to a classification node with Gini impurity equal to zero?

A. All classes occur in equal proportions

B. Every observation belongs to the same class

C. The node contains exactly two observations

D. The target variable is continuous

Correct Answer: B. Every observation belongs to the same class

Question 2

A random forest primarily reduces variance by:

A. Using one very deep tree

B. Averaging diverse trees

C. Removing all correlated features before training

D. Setting every leaf prediction to zero

Correct Answer: B. Averaging diverse trees

Question 3

Which metric is especially sensitive to large regression errors?

A. MAE

B. RMSE

C. Recall

D. Precision

Correct Answer: B. RMSE

Question 4

In gradient boosting, a smaller learning rate generally requires:

A. Fewer trees in every case

B. More boosting iterations for comparable learning

C. No validation data

D. A categorical target

Correct Answer: B. More boosting iterations for comparable learning

Question 5

What is target leakage?

A. Random variation in a feature

B. Using information unavailable when the prediction would be made

C. A model with too few features

D. A difference between MAE and RMSE

Correct Answer: B. Using information unavailable when the prediction would be made

Question 6

Permutation importance is calculated by:

A. Removing the target from the dataset

B. Permuting a feature and measuring performance deterioration

C. Increasing tree depth until training error is zero

D. Replacing every feature with its mean during training

Correct Answer: B. Permuting a feature and measuring performance deterioration

Question 7

What does a positive SHAP value for a feature usually indicate?

A. The feature lowers the prediction relative to the baseline

B. The feature raises the prediction relative to the baseline

C. The feature is necessarily causal

D. The feature is statistically insignificant

Correct Answer: B. The feature raises the prediction relative to the baseline

Question 8

Why can partial dependence be misleading with strongly correlated features?

A. It always produces a constant prediction

B. It may evaluate unrealistic combinations of feature values

C. It cannot be used with regression models

D. It requires a perfectly balanced target

Correct Answer: B. It may evaluate unrealistic combinations of feature values

Question 9

For a highly imbalanced classification problem, accuracy may be misleading because:

A. It cannot be calculated from a confusion matrix

B. A model can appear accurate by mostly predicting the majority class

C. It is always lower than recall

D. It measures only probability calibration

Correct Answer: B. A model can appear accurate by mostly predicting the majority class

Question 10

Which validation design is generally most appropriate for predicting future customer churn from historical monthly records?

A. Randomly mixing all months into every fold

B. Using only the training error

C. Time-based or forward-chaining validation

D. Evaluating only on duplicated customers

Correct Answer: C. Time-based or forward-chaining validation