Machine Learning for Insurance Prediction

Introduction

Insurance prediction concerns the estimation of future events, costs and behaviours from incomplete and uncertain information. Examples include claim frequency, claim severity, lapse, fraud, renewal, mortality, catastrophe loss and customer lifetime value. Traditional generalized linear models remain important because they are transparent, statistically disciplined and familiar to regulators. However, insurance data increasingly contain nonlinear relationships, high-order interactions, missing values, unstructured information and large numbers of predictors. Machine-learning methods can represent these structures more flexibly.

Flexibility is not automatically superiority. A model that performs well on historical data may fail when inflation, underwriting rules, distribution channels, regulation or customer behaviour changes. The central professional task is therefore not simply to maximize predictive accuracy. It is to select a model that provides an appropriate balance among predictive power, stability, calibration, interpretability, operational cost, fairness and governance.

This chapter examines decision trees, random forests, gradient boosting, support-vector methods, neural networks and ensemble models. It also explains nonlinearities, interactions, validation, hyperparameter tuning and the predictive power–stability–interpretability trade-off.

Learning objectives

By the end of this chapter, the reader should be able to:

1. Explain how major machine-learning algorithms produce insurance predictions.
2. Build and compare tree, kernel, neural-network and ensemble models.
3. Identify nonlinear effects and interactions in insurance data.
4. Design leakage-resistant validation and hyperparameter-tuning procedures.
5. Evaluate discrimination, calibration, stability, interpretability and fairness.
6. Recommend a model for an insurance decision while considering governance and implementation constraints.

1. Prediction problems in insurance

An insurance dataset usually contains an observation unit, predictors, a target and an observation or exposure period. A policy-year may be the unit for frequency modelling, while an individual claim may be the unit for severity modelling. The target may be:

• Binary: claim/no claim, lapse/no lapse, fraud/no fraud.
• Count: number of claims during an exposure period.
• Continuous and positive: claim amount, repair cost or loss ratio.
• Time-to-event: time until death, lapse or claim.
• Image, text or sequence: damage photographs, adjuster notes or telematics streams.

For a binary target, a model estimates a conditional probability p(x) = P(Y = 1 | X = x), where X is the predictor vector and Y is the outcome. For a continuous target, the objective may be to estimate E(Y | X = x). The appropriate loss function depends on the decision. A probability used for fraud investigation should be calibrated; a ranking model used to prioritize inspections may emphasize discrimination.

Insurance exposure must be handled carefully. If policy i is observed for exposure e_i, claim frequency is often expressed as claims per unit exposure. A Poisson model, for example, uses an exposure offset:

Here N_i is the claim count, e_i is exposure, x_i is the predictor vector and β is a coefficient vector. The offset assumes exposure is known and its coefficient is fixed at one. Machine-learning models should preserve this logic where the target is exposure-dependent; otherwise, a model may confuse longer observation time with higher underlying risk.

2. From statistical learning to insurance decisions

A supervised-learning model learns a function f(x) from labelled examples. Let the training sample be {(x_i,y_i)}_i=1ⁿ. A generic empirical-risk objective is:

L is the loss function, Ω(f) penalizes complexity and λ ≥ 0 controls the strength of regularization. The first term rewards fit to observed data; the second discourages excessive complexity. This is a general framework rather than a single algorithm.

For binary classification, common losses include log loss and hinge loss. For a predicted probability p_i, log loss is:

where y_i ∈ {0,1} and 0 < p_i < 1. Log loss strongly penalizes confident, incorrect predictions and is therefore useful when reliable probabilities are required. Mean squared error is common for continuous outcomes, while a Poisson deviance or count-specific loss may be more appropriate for claim counts.

A distinction is essential:

• Discrimination asks whether higher-risk cases receive higher scores.
• Calibration asks whether predicted probabilities agree with observed frequencies.
• Decision utility asks whether predictions improve the economic or operational decision.

A model can rank customers effectively but be poorly calibrated. Calibration can be assessed by grouping predictions into risk bands and comparing average predicted probability with observed event frequency, or with measures such as the Brier score:

The Brier score applies to binary outcomes and lies between 0 and 1, with lower values preferred. Its interpretation depends on event prevalence.

3. Decision trees

A decision tree recursively partitions the predictor space. A node applies a rule such as “vehicle age ≤ 8 years,” sending observations to child nodes. Each terminal node produces a prediction: a class probability for classification or an average target value for regression.

For a candidate split, impurity reduction can be used. Gini impurity for a node with class proportions p_k is:

For binary classification, a node containing only one class has Gini impurity zero. A split is attractive when the weighted impurity of its children is lower than that of the parent. Entropy is another criterion:

The logarithm base changes the unit but not the ranking of splits. These criteria are local heuristics; they do not guarantee optimal generalization.

For regression, a common split criterion minimizes within-node squared error. If node R contains observations with mean ȳ_R, its sum of squared errors is:

Strengths and limitations

Trees are easy to visualize, can represent nonlinearities and interactions automatically, and do not require predictors to be linearly related to the target. They can handle mixed predictor types and are relatively insensitive to monotonic transformations.

An unpruned tree is often unstable: a small change in the training sample can produce a different upper-level split. Deep trees also overfit. Controls include maximum depth, minimum observations per leaf, minimum split size, maximum number of leaves and cost-complexity pruning. A shallow tree may be valuable as a benchmark or as a transparent underwriting score, even when a more complex model is ultimately deployed.

4. Random forests and randomized tree ensembles

A random forest averages many decision trees. Each tree is trained on a bootstrap sample, and a random subset of predictors is considered at each split. For regression:

For classification, the forest may average class probabilities or use majority voting. Here B is the number of trees and T_b(x) is the prediction of tree b.

Averaging reduces variance when individual trees are imperfectly correlated. If each tree has variance σ² and pairwise correlation ρ, the approximate variance of the average is:

This expression assumes trees have comparable variance and correlation. It shows why increasing the number of trees reduces the independent component of variance, while correlation among trees limits the benefit.

Random forests are strong general-purpose models for tabular insurance data. They capture nonlinearities and interactions with limited preprocessing and can estimate out-of-bag performance because each bootstrap observation is omitted from some trees. Nevertheless, forests may be less accurate than boosting on structured tabular data, can be computationally heavy, and do not automatically provide well-calibrated probabilities. Variable-importance measures can also be misleading when predictors are correlated. Permutation importance should be interpreted as predictive reliance, not causation.

5. Gradient boosting

Boosting constructs an additive model sequentially. At iteration m, a new weak learner is fitted to reduce the current loss:

h_m(x) is the new tree or weak learner and η is the learning rate, usually between 0 and 1. For squared-error regression, the next learner is related to residuals r_i = y_i − F_m−1(x_i). For other losses, it is fitted to a negative gradient or pseudo-residual.

The model’s power comes from combining many small corrections. Important hyperparameters include number of boosting iterations, learning rate, tree depth, minimum leaf size, row subsampling and column subsampling. A lower learning rate often requires more trees. The number of iterations should be selected using validation or early stopping, not by inspecting test performance.

Modern implementations such as gradient-boosted decision-tree systems can be highly accurate on insurance tables. They may handle missing values, categorical encodings and nonlinear effects efficiently. Their risks include overfitting, sensitivity to tuning, probability miscalibration, unstable explanations under distribution shift and substantial computational requirements. Monotonic constraints can sometimes enforce domain knowledge, for example that risk should not decrease as a validated risk factor increases, but such constraints must be justified rather than imposed mechanically.

6. Support-vector methods

A support-vector classifier seeks a separating boundary with a large margin. For linearly separable data, the geometric margin is maximized subject to classification constraints. The soft-margin formulation permits violations:

Here w is the coefficient vector, b is the intercept, ξ_i are slack variables and C > 0 controls the trade-off between margin size and training violations. Predictors should generally be scaled because distances and inner products are central to the method.

The kernel trick replaces direct modelling in the original space with a similarity function. A radial basis function kernel is commonly written:

γ > 0 controls the scale of locality. A high γ creates highly local boundaries and may overfit; a low value produces smoother boundaries. The pair C, γ requires careful tuning.

Support-vector regression uses an ε-insensitive loss, treating errors within an ε tube as negligible. Support-vector methods can perform well in medium-sized, high-dimensional datasets, but training can become expensive for very large insurance portfolios. Their outputs are not automatically probabilities, and calibration is needed when probabilities are required. Kernel models are also difficult to explain to policyholders or underwriters compared with a short tree or additive model.

7. Neural networks

A feed-forward neural network transforms inputs through layers. A single hidden unit can be represented as:

where w_jk are weights, a_j is a bias and g is an activation function such as ReLU, g(t)=max(0,t). The output layer uses an activation appropriate to the target: sigmoid for binary probability, linear output for unconstrained regression, or other carefully selected functions for counts and positive amounts.

Weights are learned by minimizing a loss through backpropagation and an optimization method such as stochastic gradient descent or Adam. Regularization may include weight decay, dropout, early stopping and architectural restrictions. Numeric predictors usually require scaling; categorical predictors require suitable encoding or embeddings.

Neural networks are particularly useful for images, text, audio, telematics and other unstructured or sequential data. For ordinary, moderate-sized tabular insurance data, they may not outperform carefully tuned boosted trees. They require more design choices, are harder to audit and can be sensitive to preprocessing and random initialization. Their use should be justified by data structure and business value rather than by the label “deep learning.”

8. Nonlinearities and interactions

A linear predictor assumes that a one-unit change in a variable has a constant effect on the model scale. Insurance relationships rarely satisfy this universally. The effect of age may differ between young and older drivers; the relationship between mileage and breakdown risk may accelerate; and the effect of a deductible may vary by coverage type.

Trees represent nonlinearities through partitions. Boosting represents them through many local corrections. Neural networks and kernels represent smooth or complex transformations. A conventional alternative is to create splines, transformations or explicit interaction terms in a generalized linear or additive model.

An interaction exists when the effect of one predictor depends on another. In a regression model, the interaction term can be written:

The marginal effect of x₁ is β₁ + β₃x₂, so it varies with x₂. In machine learning, interactions are often learned automatically, but automatic discovery does not establish a causal relationship. Analysts should examine partial dependence, accumulated local effects, conditional plots or local explanations while recognizing that correlated features can make these interpretations unstable.

9. Validation and hyperparameter tuning

The data must be divided into training, validation and test roles. The training data fit model parameters; validation data select hyperparameters; the untouched test set estimates final out-of-sample performance. In small datasets, nested cross-validation can reduce optimistic tuning bias. The outer loop estimates performance, while the inner loop selects hyperparameters.

Random cross-validation is inappropriate when observations are dependent across time, policies or households. Insurance portfolios often require:

• Temporal validation: train on earlier periods and test on a later period.
• Group validation: keep policies, customers or claimants in one fold to prevent identity leakage.
• Geographic or channel validation: test portability across regions or distribution channels.
• Out-of-time monitoring: examine performance after deployment.

Data preprocessing, imputation, scaling, feature selection and target encoding must be fitted inside each training fold. Otherwise, information from the validation or test set leaks into the model. Leakage can also arise from post-event variables, claim settlement information, renewal decisions made after the prediction date and duplicated records.

A practical tuning workflow is:

1. Define the prediction date, target horizon and decision objective.
2. Establish a transparent benchmark, such as a GLM or shallow tree.
3. Create leakage-resistant temporal or grouped splits.
4. Build preprocessing and model fitting as one reproducible pipeline.
5. Use random or Bayesian search for important hyperparameters, with sensible bounds.
6. Apply early stopping only with an appropriate validation set.
7. Select using a primary metric and inspect secondary metrics.
8. Refit only after the selection rule has been finalized.
9. Evaluate once on the untouched test period.
10. Document data, code, versions, parameters and limitations.

10. Comparing models: a multi-criteria framework

A model comparison should not rely on a single score. Relevant measures include:

AUC measures ranking across thresholds and is insensitive to calibration. Precision–recall analysis is often more informative than ROC analysis for rare fraud events. RMSE gives greater weight to large errors than MAE. The chosen metric must reflect the cost of false positives, false negatives, underprediction and overprediction.

A model’s apparent improvement should be assessed for statistical and practical significance. Confidence intervals can be estimated by bootstrap resampling, but resampling must respect groups and time where necessary. A tiny gain in AUC may not justify a complex model if the model doubles implementation cost or loses regulatory acceptability.

11. Ensembles and model selection

An ensemble combines predictions from several models. Simple averaging is effective when component errors are not perfectly correlated. A weighted ensemble can be expressed as:

The weights may be selected using validation data. Stacking trains a meta-model on out-of-fold predictions from base models. Out-of-fold predictions are essential; using in-sample predictions allows the meta-model to learn overly optimistic signals.

Ensembles may improve accuracy and stability, but they reduce simplicity. A blended model should have a clear operating purpose, documented fallback behaviour and a monitoring plan. In regulated decisions, a transparent benchmark may be retained as a challenger, reason-code generator or safeguard even when an ensemble is the primary predictor.

12. Illustrative case: motor insurance claim-frequency prediction

Background

An insurer wants to improve its annual review of private motor policies. It has three years of policy exposure, driver and vehicle characteristics, prior claims, geographic indicators and telematics summaries. The outcome is the number of at-fault claims in the following policy year.

Problem

The existing Poisson GLM is stable and interpretable but underestimates risk for a subgroup with high annual mileage and recent speeding events. Management asks whether a more flexible model can improve ranking without creating unacceptable governance risk.

Analysis and approach

The analyst defines the prediction date as the renewal date and excludes all information recorded after that date. Policies are split by calendar time: the first two years are used for development and the final six months for validation; a later held-out period is reserved for final testing. Multiple policies belonging to one customer remain in the same partition.

The benchmark is an exposure-adjusted Poisson GLM. Candidate models are a pruned tree, random forest, gradient-boosted trees and a neural network. Claims are rare, so the analyst reports Poisson deviance, MAE, lift in the highest-risk decile, calibration by decile and out-of-time stability. Hyperparameters are selected using rolling validation rather than random folds.

The boosted model produces the strongest validation lift, while the random forest is slightly more stable across time slices. The neural network does not improve the held-out deviance and is harder to explain. The insurer therefore selects the boosted model for triage of proactive risk-management interventions, not as an automatic premium decision. Predictions are calibrated on a validation period, monitored monthly and accompanied by reason codes derived from a constrained explanation process.

Outcome

The model increases concentration of claims in the highest-risk decile while maintaining acceptable calibration. Governance approval is conditional on annual fairness testing, documentation of variables, human review of adverse actions and a fallback to the GLM if data quality or drift thresholds are breached.

Learning points

1. The prediction date is as important as the algorithm.
2. Exposure and time structure must be respected.
3. A model can be useful for intervention while unsuitable for automatic pricing.
4. Stable calibration and governance may outweigh a marginal accuracy gain.
5. Model monitoring and fallback design are part of the model, not postscript activities.

13. Governance, ethics and regulation

Insurance models influence access, price, claims handling and fraud scrutiny. Governance should address purpose limitation, data minimization, lawful data use, documentation, security, human oversight and contestability. Protected characteristics and proxy variables require particular care. Removing a protected attribute does not guarantee fairness if geographic, behavioural or socioeconomic variables act as proxies.

Fairness has no single universal definition. Possible assessments include group-specific calibration, selection rates, false-positive rates and error severity. These criteria can conflict, particularly when base event rates differ. The insurer must therefore define the relevant legal and ethical objective with expert and stakeholder input rather than treating a single metric as dispositive.

Explainability should match the decision. Global feature importance helps portfolio understanding; local explanations may support a claim or underwriting review; simplified reason codes may be needed for customer communication. Explanations are approximations and should not be presented as causal proofs. Model cards, data sheets, validation reports, change logs and adverse-impact analyses support accountable use.

14. Contemporary developments and best practices

Current practice increasingly combines structured policy data with text, images and telematics. Multimodal systems can improve claims triage and damage assessment, but they create additional risks involving privacy, image bias, adversarial manipulation, data drift and difficult-to-audit representations. Large pretrained models may assist document extraction or summarization, but generated content must not be treated as verified fact without controls.

Recommended practices are:

• Start with a strong, transparent benchmark.
• Align the target with the business decision and prediction horizon.
• Use time-aware, group-aware validation.
• Treat preprocessing and feature selection as part of the pipeline.
• Tune only within development data.
• Assess calibration, stability and subgroup performance.
• Prefer variables available reliably at decision time.
• Apply domain constraints where defensible.
• Maintain human review for consequential decisions.
• Monitor drift, missingness, latency, calibration and outcome quality.
• Establish challenger models, rollback procedures and documented approval gates.

Chapter Summary

Machine learning extends insurance prediction beyond fixed linear relationships. Decision trees provide transparent recursive partitions; random forests reduce tree variance through randomized averaging; gradient boosting builds sequential error corrections and is often highly effective for tabular data; support-vector methods use margins and kernels; neural networks learn layered representations and are especially useful for unstructured data; and ensembles combine diverse models to improve predictive performance.

The main analytical challenges are nonlinearities, interactions, leakage, temporal dependence, rare events, calibration and distribution shift. Hyperparameter tuning must be nested within a valid validation design, and performance must be assessed across discrimination, calibration, stability, fairness, interpretability and operational feasibility. In insurance, the best model is not necessarily the most complex or the one with the highest test score. It is the model whose benefits are material, reliable, explainable enough for its use, governable and sustainable in production.

Key Terms

Boosting: Sequentially adding weak learners to reduce a chosen loss.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Decision tree: A recursive partitioning model that assigns predictions to terminal nodes.

Discrimination: Ability to rank higher-risk observations above lower-risk observations.

Early stopping: Ending iterative training when validation performance ceases to improve.

Ensemble: A model combining predictions from multiple component models.

Exposure: Amount of time or risk basis over which an insurance outcome is observed.

Feature leakage: Use of information unavailable at the prediction date or derived from the target period.

Gradient boosting: Boosting in which learners follow loss gradients or pseudo-residuals.

Hyperparameter: A model-design setting selected outside the ordinary parameter-fitting process.

Interaction: A relationship in which the effect of one predictor depends on another predictor.

Kernel: A similarity function enabling algorithms to model relationships in transformed spaces.

Loss function: A numerical measure of prediction error used for fitting or evaluation.

Random forest: An ensemble of randomized bootstrap decision trees.

Regularization: Penalization or restriction used to control model complexity.

Support vector: An observation that influences the margin or regression boundary.

Temporal validation: Evaluation using future observations relative to the training period.

Out-of-fold prediction: A prediction generated for an observation by a model that did not train on that observation.

Variable importance: A measure of how predictive performance changes when a feature is used, removed or perturbed.

Drift: Change over time in data distributions, relationships or model performance.

Review Questions

5 Short Answer Questions

1. Distinguish discrimination from calibration.
2. Why can a deep decision tree be unstable?
3. What roles do C and γ play in a radial-basis support-vector classifier?
4. Why must exposure be considered in claim-frequency prediction?
5. What is feature leakage? Give one insurance example.

5 Long Answer Questions

1. Compare decision trees, random forests and gradient boosting in terms of bias, variance, interpretability and tuning.
2. Explain how nonlinearities and interactions can be represented by trees, kernels, neural networks and additive models.
3. Design a validation strategy for a lapse-prediction model using five years of policy data.
4. Discuss why a model with superior AUC may still be unsuitable for insurance deployment.
5. Explain the governance requirements for using machine learning in claims triage or pricing.

3 Analytical / Application Questions

1. An insurer has a rare-fraud problem with severe class imbalance. Propose appropriate metrics, validation splits and calibration checks, and justify each choice.
2. A gradient-boosted model improves held-out deviance by 1% but is materially less stable across regions than a random forest. Recommend a decision rule for model selection.
3. Design an ensemble for motor insurance claim prediction using a GLM, random forest and boosted trees. Explain how you would generate training data for the meta-model without leakage and how you would monitor the ensemble after deployment.

Multiple Choice Questions with Answers

Question 1

Which measure directly assesses whether predicted probabilities agree with observed event frequencies?

A. Calibration

B. Discrimination

C. Training speed

D. Tree depth

Correct Answer: A. Calibration

Question 2

In a random forest, why are bootstrap samples and random feature subsets used?

A. To force every tree to be identical

B. To reduce correlation among trees and reduce ensemble variance

C. To eliminate the need for validation

D. To guarantee causal interpretation

Correct Answer: B. To reduce correlation among trees and reduce ensemble variance

Question 3

In the boosting update F_m(x) = F_m−1(x) + ηh_m(x), what does η represent?

A. The exposure period

B. The number of classes

C. The learning rate

D. The Gini impurity

Correct Answer: C. The learning rate

Question 4

Which validation design is generally most appropriate when predicting future insurance outcomes from historical records?

A. Randomly mixing all dates before splitting

B. Training on future data and testing on past data

C. Temporal validation with future observations held out

D. Using only training error

Correct Answer: C. Temporal validation with future observations held out

Question 5

What is the main purpose of the penalty term λΩ(f) in a learning objective?

A. To increase model complexity without limit

B. To control complexity and reduce overfitting

C. To replace the target variable

D. To guarantee perfect calibration

Correct Answer: B. To control complexity and reduce overfitting

Question 6

Which support-vector hyperparameter controls the trade-off between margin size and training violations in the soft-margin formulation?

A. C

B. B

C. η

D. ρ

Correct Answer: A. C

Question 7

Why should out-of-fold predictions be used when training a stacking meta-model?

A. They prevent the meta-model from relying on in-sample optimistic predictions

B. They guarantee that all base models are linear

C. They remove the need for a test set

D. They make all component predictions identical

Correct Answer: A. They prevent the meta-model from relying on in-sample optimistic predictions

Question 8

Which metric gives greater weight to very large continuous prediction errors than MAE?

A. RMSE

B. Median

C. AUC

D. Accuracy

Correct Answer: A. RMSE

Question 9

A policy has claim count N_i observed over exposure e_i. In an exposure-adjusted Poisson model, what is the role of log(e_i)?

A. It is a fixed exposure offset

B. It is always the response variable

C. It is a fairness constraint

D. It is the tree-pruning parameter

Correct Answer: A. It is a fixed exposure offset

Question 10

Which statement best describes a neural network’s suitability for insurance data?

A. It is always superior to boosted trees on tabular data

B. It is especially useful when data contain complex unstructured inputs such as images or text

C. It requires no preprocessing or tuning

D. Its outputs are automatically calibrated probabilities

Correct Answer: B. It is especially useful when data contain complex unstructured inputs such as images or text