8. Supervised Machine Learning for Fraud Detection

Introduction and Context

Insurance fraud detection has moved from rule-based screening towards predictive systems that learn from historical claims, policies, transactions, customer interactions and investigation outcomes. Supervised machine learning is central to this transition because it uses labelled examples—claims previously classified as fraudulent or legitimate—to estimate the probability that a new claim belongs to either class.

The purpose of a supervised model is not merely to produce a classification. In insurance, its output may determine whether a claim is paid automatically, referred for investigation, subjected to enhanced verification, or escalated to a specialist team. Consequently, model development is both a statistical and managerial activity. A model with high predictive accuracy may still be unsuitable if it generates too many false accusations, ignores vulnerable customers, cannot be explained to regulators, or fails when fraud patterns change.

Fraud datasets are particularly difficult. Fraud is usually rare, labels may be delayed or incomplete, legitimate behaviour is highly diverse, and investigators may selectively examine cases already considered suspicious. Fraudsters also adapt to controls. These features make class imbalance, data leakage, concept drift, threshold selection and operational evaluation as important as algorithm choice.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Explain the role of supervised learning in insurance fraud detection.

2. Prepare labelled claims data while avoiding leakage and biased sampling.

3. Develop and compare decision trees, random forests, gradient boosting, support vector machines and neural models.

4. Select appropriate methods for imbalanced fraud data.

5. Interpret confusion matrices, precision, recall, F-scores, ROC curves, precision–recall curves, lift and cost-sensitive measures.

6. Recognise concept drift and design monitoring and retraining processes.

7. Assess models in relation to investigation capacity, fairness, explainability and governance.

8. Apply model outputs to practical insurance decisions.

From Claims Data to a Supervised Learning Problem

A supervised fraud model receives an input vector X containing information about a claim and a target variable y. In a binary formulation, y=1 represents confirmed or strongly supported fraud and y=0 represents a legitimate claim. The model estimates either a class label or a conditional probability:

P(y=1 | X).

The probability is valuable because operational decisions rarely have only two alternatives. An insurer may automatically settle claims with very low predicted risk, send medium-risk claims for additional documentation, and investigate high-risk claims. The classification threshold therefore reflects business costs and available capacity rather than a universal statistical rule.

Typical predictors include claim amount, policy tenure, time between policy inception and loss, frequency of prior claims, repair estimates, claimant and vehicle characteristics, geographical relationships, communication patterns, payment details and network links among claimants, brokers, garages or medical providers. Features must be available at the moment the prediction is made. Using a post-investigation outcome, later payment reversal, or information created after referral would produce target leakage and an unrealistically strong model.

Labels require careful definition. A claim labelled fraudulent because an investigator confirmed deliberate misrepresentation is different from a claim merely referred for suspicion. If referrals are used as positive labels, the model may learn historical investigator preferences rather than fraud itself. Unresolved claims should not automatically be treated as legitimate; they may require exclusion, delayed labelling or probabilistic treatment.

A sound development process separates data chronologically. Training data precede validation data, which precede a genuinely out-of-time test set. Random splitting can allow near-duplicate claims, repeated policyholders or future behavioural patterns to appear in both training and test samples. Group-based or time-based splitting is often safer.

Important Concepts and Terminology

A true positive is a fraudulent claim correctly identified. A false positive is a legitimate claim incorrectly flagged. A true negative is a legitimate claim correctly cleared, while a false negative is fraud not detected. These four outcomes form the confusion matrix.

The decision threshold converts a predicted probability into an action. Lowering it generally increases recall but also increases the number of false positives. Class imbalance occurs when one class, usually fraud, is much less frequent than the other. Data leakage occurs when information unavailable at prediction time enters model development. Concept drift is a change over time in the relationship between predictors and outcomes. Calibration describes whether predicted probabilities correspond to observed frequencies; among claims assigned a probability of 0.70, approximately 70 per cent should be positive in a well-calibrated group.

Feature engineering transforms raw data into informative variables, such as claim-to-premium ratio, time since last claim, distance between incident and repairer, or the number of linked entities. One-hot encoding represents categorical values as indicator variables. Regularisation penalises model complexity to reduce overfitting. Ensembling combines multiple models. Explainability concerns the ability to communicate how a model reached a prediction, while fairness concerns whether model performance and treatment are acceptable across relevant groups.

The Main Supervised Learning Models

Decision Trees

A decision tree recursively divides observations using questions such as whether the claim amount exceeds a threshold or whether a policy was recently issued. Each terminal node produces a class probability or decision. Splits are commonly selected using measures such as Gini impurity or entropy. Trees are attractive because their logic can be displayed as a sequence of rules and can accommodate nonlinear relationships and interactions.

Their weaknesses include instability, overfitting and a tendency to favour complex partitions when insufficiently constrained. Maximum depth, minimum observations per leaf, pruning and class weights are important controls. A small tree may be suitable for communicating a referral policy, but a single deep tree often performs less reliably than an ensemble.

Random Forests

A random forest builds many trees using bootstrap samples and randomly selected subsets of features. Predictions are aggregated, usually by averaging probabilities or voting. Randomisation reduces the variance of individual trees and allows the model to capture interactions without requiring a linear functional form.

Random forests are robust baseline models for mixed insurance data and can estimate variable importance. However, conventional importance measures may be biased when predictors have different scales or many categories. Forest probabilities may also require calibration. The model can be computationally heavier than a single tree and less transparent to explain claim by claim, although local explanation techniques can assist.

Gradient Boosting

Gradient boosting constructs trees sequentially. Each new tree focuses on errors made by the existing ensemble, and the combined model gradually improves predictive performance. Modern implementations can handle nonlinearities, missing values and complex interactions efficiently. Boosting often performs strongly on structured tabular insurance data.

Its flexibility creates risks. Excessive depth, too many iterations or a high learning rate can cause overfitting. Early stopping based on validation performance, shrinkage, subsampling and regularisation are therefore important. Boosting models may be sensitive to noisy labels and can appear highly confident without calibration. Their operational use should include probability validation and explanation mechanisms such as feature-attribution methods or simplified reason codes.

Support Vector Machines

A support vector machine seeks a boundary that separates classes with the widest possible margin. Kernel functions allow nonlinear boundaries by implicitly mapping observations into a higher-dimensional space. For fraud data, class-weighted SVMs can penalise errors on fraud more heavily than errors on legitimate claims.

SVMs can be effective in moderate-sized, well-prepared datasets, particularly where a clear separating structure exists. They require feature scaling, careful selection of the penalty parameter and, for nonlinear models, kernel parameters. Training and scoring can become expensive for very large datasets. Probability outputs are not intrinsic and may need calibration. Their decision boundaries are also less intuitive for claims staff than tree-based rules.

Neural Models

A neural network uses layers of weighted transformations and nonlinear activation functions. It can learn complex relationships from high-dimensional data and is especially useful when claims analysis combines text, images, voice, documents or transaction sequences with structured attributes. For example, a network may process accident descriptions and photographs alongside policy and payment variables.

Neural models demand substantial data, careful regularisation, validation and monitoring. They may be difficult to explain, vulnerable to changes in data quality, and prone to learning shortcuts such as document formatting or investigator identity. For ordinary tabular claims data, a well-tuned boosting model may be more practical than a deep network. Neural architectures should therefore be selected because their representation capability is needed, not because they are technologically fashionable.

Managing Rare Fraud and Unequal Error Costs

Accuracy is misleading when fraud is rare. If only 1 per cent of claims are fraudulent, a classifier that labels every claim legitimate achieves 99 per cent accuracy but detects nothing. Resampling can address this problem. Undersampling reduces the majority class, while oversampling increases the minority class. Synthetic methods such as SMOTE create interpolated minority observations, but synthetic examples must be generated only within the training folds; otherwise validation information leaks into training.

Class weighting changes the learning objective by assigning a larger penalty to misclassified fraud. It is often safer than indiscriminate oversampling, although it can produce poorly calibrated probabilities. A practical workflow may combine weighting during training with calibration and threshold selection on untouched validation data.

The best threshold depends on costs. Let C_FP be the cost of unnecessarily investigating a legitimate claim, C_FN the cost of missing fraud, C_TP the benefit or avoided loss from correctly detecting fraud, and C_TN the cost or benefit of clearing a legitimate claim. A simplified expected-cost expression is:

Expected cost = C_FP(FP) + C_FN(FN) - benefit of correctly detected fraud.

The insurer should estimate these values rather than assume that every error has equal importance. Investigation capacity also matters. If a team can review only 500 claims weekly, the model may be evaluated at the top 500 ranked claims, using precision at capacity and lift, rather than at an arbitrary probability threshold.

Evaluating Models for Fraud Operations

The confusion matrix supports several complementary measures. Precision is TP/(TP+FP), the proportion of referrals that are actually fraud. Recall, or sensitivity, is TP/(TP+FN), the proportion of fraud detected. The F1-score is the harmonic mean of precision and recall:

F1 = 2(precision × recall)/(precision + recall).

The F-beta score permits emphasis on recall or precision. When missed fraud is particularly costly, beta may exceed one; when investigation resources are scarce, precision may receive greater emphasis. Specificity measures the proportion of legitimate claims correctly cleared.

The receiver operating characteristic curve plots recall against the false-positive rate across thresholds. The area under this curve, ROC-AUC, measures ranking discrimination across all thresholds. It can look impressive in highly imbalanced settings because the false-positive rate may remain numerically small even when the absolute number of false referrals is operationally large.

The precision–recall curve is often more informative for rare fraud because it focuses on positive identification quality. Average precision summarises performance across thresholds, but it must be interpreted in relation to the fraud base rate. Precision at k, recall at k, and lift at k are especially useful where investigators review the top k cases. Lift at a selected population proportion compares the fraud rate in the selected group with the overall fraud rate. A lift of five means that the selected group contains five times the baseline fraud concentration.

Evaluation must be out-of-time and, where possible, out-of-entity. Confidence intervals or repeated temporal validation help reveal whether apparent differences are stable. A model should also be assessed for calibration, subgroup performance, processing latency, missing-data sensitivity and the financial value of decisions. No single metric is sufficient.

Concept Drift and the Learning Life Cycle

Fraud patterns change when economic conditions, products, regulations, repair practices and fraud controls change. This is concept drift. The distribution of predictors may change even when the fraud relationship remains stable; alternatively, the relationship between a predictor and fraud may itself change. New fraud schemes can make old features less useful, while a control introduced by the insurer may reduce the apparent incidence of a previously common pattern.

Monitoring should examine input distributions, missingness, prediction scores, calibration, referral rates, investigation outcomes and business losses. Drift indicators such as population stability measures or statistical distribution tests can provide alerts, but an alert is not proof that the model has become invalid. Analysts must distinguish genuine behavioural change from seasonal variation, data-pipeline failures or changes in labelling practice.

A governance process should define alert thresholds, investigation responsibilities, retraining frequency and emergency override procedures. Retraining may use rolling windows to emphasise recent behaviour, expanding windows to preserve historical diversity, or champion–challenger testing to compare a current production model with alternatives. Labels often arrive with delay, so monitoring must account for immature outcomes. Silent automatic retraining is risky; every replacement should pass validation, fairness review, documentation and approval.

From Prediction to Investigation

A fraud score is not a finding of fraud. It is a prioritisation signal. A mature workflow combines the score with reason codes, relevant evidence, investigator feedback and service-level rules. The system may recommend document verification, a telephone interview, specialist review or network analysis. Investigators should record outcomes using consistent categories so that future training labels are meaningful.

Human intervention can improve decisions but can also introduce feedback loops. If only high-scoring claims are investigated, the insurer learns little about low-scoring claims and may overestimate model effectiveness. Random samples of apparently legitimate claims can support unbiased performance estimation. Overrides should be monitored because systematic override patterns may indicate model weakness, poor usability or inappropriate policy.

Illustrative Case Study: Motor Claims Triage at Meridian General Insurance

Background

Meridian General Insurance, a fictional mid-sized insurer, receives approximately 40,000 motor claims each month. Its existing rules identify claims involving recent policies, large repair estimates and repeated claimants. The rules generate many referrals, but investigators report that a substantial proportion are legitimate. Management wishes to improve prioritisation without delaying ordinary customers.

Situation and Problem

Historical data contain investigator outcomes, payment records, policy information, repairer details and claim narratives. Confirmed fraud represents a small minority of claims. Outcomes are available only after several months, and some claims remain unresolved. Management initially proposes selecting the model with the highest accuracy.

Analysis

The analytics team rejects accuracy as the primary criterion because a majority-class model would appear successful. It defines the prediction point as the time of first claim registration and removes post-registration information. Data are divided chronologically into training, validation and out-of-time test periods. Claims from the same policyholder are grouped to reduce information sharing across partitions.

The team compares a pruned decision tree, class-weighted random forest, gradient-boosted trees and a class-weighted SVM. A neural model is not selected initially because the structured dataset is moderate in size and text quality is inconsistent. Features include claim-to-premium ratio, policy age, prior claim frequency, time and distance relationships, payment-account links and repairer concentration. Personally sensitive attributes are excluded from direct use, but subgroup performance is tested where legally and ethically appropriate.

The gradient-boosted model produces the strongest out-of-time ranking, while the random forest is slightly easier to communicate. The team calibrates the selected model and evaluates it at the actual weekly investigation capacity. At that operating point, precision and lift are more useful to management than ROC-AUC alone. A cost analysis shows that missing a high-value organised fraud case is more expensive than investigating a low-value false positive, but excessive referrals create customer-service and staffing costs.

Approach and Solution

Meridian deploys a two-stage process. Low-risk claims are processed through normal settlement controls. Medium- and high-risk claims receive a score, ranked queue position and a small number of reason codes. Investigators can request additional documents but cannot reject a claim solely because of the model score. A random sample of low-risk claims is audited to estimate false negatives and limit selection bias.

The model is monitored weekly for score distribution, missing values, referral volume and system failures. Confirmed outcomes are reviewed monthly, with formal model evaluation quarterly or sooner if drift alerts occur. A challenger model is developed using a rolling recent-data window. All changes are recorded in a model register.

Outcome

The organisation obtains a more concentrated investigation queue and reduces unnecessary referrals, while preserving manual review for ambiguous cases. The most important improvement is not simply a higher statistical score; it is the alignment of threshold, investigation capacity, documentation, feedback and governance.

Learning Points

The case illustrates five principles: the model must respect the prediction-time information boundary; rare-event metrics must replace raw accuracy; ranking performance must be connected to operational capacity; human review remains essential; and monitoring must continue after deployment.

Risks, Limitations and Professional Judgement

Historical labels may encode investigator bias, unequal access to evidence or settlement practices rather than objective fraud. Proxy variables can reproduce discriminatory effects even when protected characteristics are omitted. A model may also penalise new customers because they have little history, or flag customers from areas associated with particular repair networks without distinguishing legitimate market structure from misconduct.

Explainability has several levels. Global explanations describe overall feature influence; local explanations identify factors associated with an individual score; counterfactual explanations indicate what would need to differ for a changed outcome. None proves causation. Reason codes must be stable, comprehensible and faithful to the model. Overly technical explanations are not useful to customers or claims handlers.

Other common mistakes include random splitting of time-dependent claims, tuning on the test set, using post-outcome variables, ignoring probability calibration, optimising ROC-AUC while neglecting precision at capacity, applying SMOTE before cross-validation, and deploying without monitoring. Data security is also important because claims contain personal, financial and sometimes health information. Access controls, retention limits, encryption, audit trails and vendor oversight should be part of the model life cycle.

Governance, Ethics and Regulation

Fraud prevention must be proportionate. A suspicious score should trigger verification, not automatic condemnation. Customers should receive fair treatment, opportunities to provide evidence and appropriate explanations of adverse decisions. Human review is particularly important where a decision may delay payment, affect coverage or cause reputational harm.

Model governance should document purpose, data sources, label definitions, exclusions, features, assumptions, validation results, thresholds, limitations, approval authority and monitoring plans. Independent validation should challenge both technical performance and business justification. Privacy and insurance regulation vary by jurisdiction, but principles of lawful processing, data minimisation, security, transparency, accountability and non-discrimination are broadly relevant. Third-party models require contractual rights to audit, understand, monitor and discontinue the system.

Current Developments and Best Practice

Contemporary fraud programmes increasingly combine supervised learning with graph analytics, anomaly detection, natural-language processing, image analysis and human-in-the-loop review. Graph features can reveal links among claimants, addresses, devices, repairers and payment accounts. Synthetic data and privacy-preserving methods may support development where real claims cannot be widely shared. However, added technical complexity increases validation and governance requirements.

Best practice is to begin with a well-defined business decision, establish reliable labels and a time-aware evaluation design, create a transparent baseline, compare advanced models using operational metrics, calibrate probabilities, select thresholds through cost and capacity analysis, test subgroup performance, document limitations, and monitor both data and outcomes after deployment. Model sophistication should be justified by measurable value and acceptable risk.

Chapter Summary

Supervised machine learning estimates fraud risk from labelled historical claims and supports decisions such as automated settlement, verification and investigation prioritisation. Decision trees offer transparency; random forests reduce variance; gradient boosting often provides strong performance on structured data; support vector machines can model useful boundaries in appropriately scaled datasets; and neural models are valuable when complex, high-dimensional or multimodal information is available.

Fraud detection requires special treatment because fraud is rare, labels are imperfect and patterns change. Class weighting, careful resampling, calibration, threshold selection and cost-sensitive evaluation are essential. Precision, recall, F-scores, ROC and precision–recall curves, lift and capacity-based measures answer different questions. Time-aware validation, leakage prevention, concept-drift monitoring, explainability, fairness and human oversight determine whether a statistically effective model becomes a responsible insurance system.

Key Terms

Supervised learning: A machine-learning approach that learns from examples with known outcome labels.

Fraud label: The target classification indicating whether a claim has been established or assessed as fraudulent.

Feature engineering: Transforming raw data into variables useful for prediction.

Data leakage: Use of information in development that would not be available at prediction time.

Confusion matrix: A table of true and false positive and negative predictions.

Precision: The proportion of flagged cases that are actually positive.

Recall: The proportion of all positive cases detected by the model.

F1-score: The harmonic mean of precision and recall.

ROC curve: A plot of true-positive rate against false-positive rate across thresholds.

Precision–recall curve: A plot showing the trade-off between precision and recall.

Lift: The ratio of the positive rate in a selected group to the overall positive rate.

Class imbalance: A substantial difference in the frequencies of outcome classes.

Calibration: Agreement between predicted probabilities and observed frequencies.

Decision threshold: The score boundary used to convert probability into action.

Random forest: An ensemble of randomized decision trees whose predictions are aggregated.

Gradient boosting: A sequential ensemble method that improves predictions through error-focused trees.

Support vector machine: A margin-based classifier that may use kernels for nonlinear separation.

Concept drift: Change over time in data distributions or predictor–outcome relationships.

Model governance: Processes for approving, documenting, monitoring and controlling model use.

Explainability: The ability to communicate the basis and limitations of a model prediction.

Review Questions

Short Answer Questions

1. Why is accuracy an inadequate primary measure for rare insurance fraud?

2. Distinguish precision from recall in a claims investigation context.

3. What is target leakage? Give one insurance example.

4. Why are time-based data splits generally preferable to random splits for fraud models?

5. Define concept drift.

Long Answer Questions

1. Compare decision trees, random forests, gradient boosting, support vector machines and neural models for insurance fraud detection.

2. Explain how class weighting, resampling, calibration and threshold selection can be combined in an imbalanced classification workflow.

3. Discuss the use and limitations of ROC-AUC, precision–recall curves, F-scores and lift.

4. Design a monitoring and retraining framework for a deployed motor-fraud model.

5. Examine the ethical, governance and explainability requirements for using machine learning to prioritise claims investigations.

Analytical / Application Questions

1. An insurer has a 1 per cent fraud rate and an investigation team capable of reviewing the top 2 per cent of claims. Which performance measures should management examine, and why?

2. A model’s ROC-AUC remains stable, but precision at the investigation threshold falls sharply after a new claims product is introduced. Analyse possible causes and recommend actions.

3. Design an evaluation plan for the Meridian General Insurance case that addresses delayed labels, repeated policyholders, class imbalance, investigation capacity and false-positive costs.

22. Multiple Choice Questions with Answers

Question 1

Why can a model with 99 per cent accuracy still be useless for insurance fraud detection?

A. Accuracy cannot be calculated from a confusion matrix

B. Fraud may represent only a very small proportion of claims

C. Accuracy applies only to regression models

D. Accuracy always gives too many false negatives

Correct Answer: B. Fraud may represent only a very small proportion of claims

Question 2

Which practice most directly prevents target leakage?

A. Including post-investigation payment reversals as predictors

B. Generating synthetic fraud cases before splitting the data

C. Restricting predictors to information available at claim registration

D. Using more trees in the ensemble

Correct Answer: C. Restricting predictors to information available at claim registration

Question 3

An insurer can investigate only the 500 highest-ranked claims. Which measure is especially relevant?

A. Precision at 500

B. Training loss only

C. Unweighted accuracy on all historical claims

D. Number of neural-network layers

Correct Answer: A. Precision at 500

Question 4

What is the principal purpose of class weighting?

A. To remove all legitimate claims from training

B. To assign different error penalties to outcome classes

C. To guarantee calibrated probabilities

D. To convert a classifier into an unsupervised model

Correct Answer: B. To assign different error penalties to outcome classes

Question 5

Which model builds trees sequentially, with later trees concentrating on earlier errors?

A. Random forest

B. Support vector machine

C. Gradient boosting

D. Pruned single tree

Correct Answer: C. Gradient boosting

Question 6

Why may a precision–recall curve be more informative than ROC-AUC for rare fraud?

A. It focuses more directly on positive identification quality

B. It eliminates the need for labels

C. It guarantees a lower false-positive count

D. It measures only model calibration

Correct Answer: A. It focuses more directly on positive identification quality

Question 7

A fall in fraud prevalence will usually affect which metric most directly, even if ranking ability is unchanged?

A. Precision

B. Number of features

C. Maximum tree depth

D. Training sample size

Correct Answer: A. Precision

Question 8

Which situation is an example of concept drift?

A. A column is renamed in a database

B. The relationship between claim timing and fraud changes after a new fraud scheme emerges

C. A model uses a confusion matrix

D. A categorical variable is one-hot encoded

Correct Answer: B. The relationship between claim timing and fraud changes after a new fraud scheme emerges

Question 9

Why should a fraud score not automatically be treated as proof of fraud?

A. Scores are always random

B. A score is a prioritisation signal subject to error and possible bias

C. Models cannot process claims data

D. Fraud labels are never used in supervised learning

Correct Answer: B. A score is a prioritisation signal subject to error and possible bias

Question 10

Which evaluation design is most appropriate when claims outcomes are delayed and behaviour changes over time?

A. A single random split using all available outcomes

B. Chronological training, validation and out-of-time testing

C. Testing only on the training sample

D. Selecting the model with the highest raw accuracy

Correct Answer: B. Chronological training, validation and out-of-time testing