Machine Learning and AI for Fraud Detection

Introduction

Digital banking has transformed fraud from a predominantly manual and account-specific problem into a high-volume, real-time, adaptive threat. Payment platforms may process millions of transactions daily across cards, mobile applications, instant payments, branches, agents and open-banking interfaces. Fraudsters exploit stolen credentials, synthetic identities, mule accounts, social engineering, malware, collusion and weaknesses in customer journeys. Rule-based systems remain useful, but they struggle with changing behaviour, interactions among many variables and the need to distinguish unusual activity from legitimate customer variation.

Machine learning (ML) supports fraud detection by estimating risk from historical and streaming data. Artificial intelligence (AI) is the broader field that includes ML, knowledge-based reasoning, natural-language systems and generative models. In banking, the central task is not simply to maximise predictive accuracy. It is to allocate scarce investigative and customer-contact capacity while controlling financial loss, customer friction, regulatory risk and reputational damage.

A technically strong fraud programme therefore combines supervised learning, unsupervised detection, rules, network analysis, human judgement, governance and continuous monitoring. This chapter examines those components, with particular attention to imbalanced data, evaluation, drift, explainability, adversarial behaviour, responsible AI and generative-AI opportunities and risks.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Distinguish supervised, unsupervised and hybrid fraud-detection approaches.
2. Prepare fraud data without leakage and select features appropriate to banking operations.
3. Evaluate models using metrics suitable for highly imbalanced outcomes and operational constraints.
4. Explain threshold selection, calibration, cost-sensitive learning and human-in-the-loop decisions.
5. Diagnose model drift, adversarial manipulation and data-quality failure.
6. Apply explainability, validation, governance and responsible-AI principles.
7. Assess practical uses and risks of generative AI in fraud operations.

1. The Fraud-Detection Problem as a Decision System

A transaction record may contain amount, merchant, channel, device, location, authentication method, time, account age and behavioural history. The target variable may be y ∈ {0,1}, where 1 denotes confirmed fraud under a defined labelling policy. A model produces a score or probability p = P(y=1|x), where x is the feature vector.

The score is not itself a decision. The institution must choose an action: approve, step up authentication, queue for review, decline, freeze an instrument or contact the customer. The action depends on expected loss, intervention cost, service-level commitments and legal constraints.

A simplified expected-cost decision compares the cost of approving a fraudulent transaction with the cost of intervening on a legitimate one. If C_FN is the cost of a false negative and C_FP is the cost of a false positive, intervention is economically justified when:

Here, p is the estimated fraud probability. Under the narrow assumptions that probabilities are calibrated, these are the only two relevant outcomes, and action costs are constant, the equivalent threshold is:

The formula is a conceptual benchmark, not a universal operating threshold. In practice, costs vary by customer, product, recoverability, regulatory requirement and intervention type. A bank may deliberately choose a lower threshold for high-value instant payments or a higher threshold where customer interruption is especially harmful.

Fraud labels and the observation window

Fraud labels are imperfect. A transaction initially marked legitimate may later generate a chargeback; a customer dispute may be mistaken; investigation outcomes may be delayed; and confirmed cases represent only detected fraud. The label definition should specify the observation window, fraud taxonomy, source system and treatment of unresolved cases.

A common error is to train on information that became available only after the decision. For example, a chargeback outcome or an investigator’s conclusion must not be used as a feature for an earlier authorisation decision. This is target leakage and can produce excellent historical scores but operational failure.

2. Three Modelling Paradigms

2.1 Supervised learning

Supervised learning uses labelled examples. Classification methods include logistic regression, decision trees, random forests, gradient-boosted trees and neural networks. For a binary classifier, logistic regression models the log odds as a linear function of features:

Here, β₀ is the intercept, β_j are coefficients, x_j are features and e is the base of natural logarithms. The model assumes a linear relationship between predictors and log odds, although transformed variables and interactions can extend its flexibility. Logistic regression is often valuable where transparency, stability and governance are priorities.

Tree ensembles can represent nonlinear interactions, such as the combination of a new device, unusual geography and rapid beneficiary creation. Gradient boosting builds a sequence of weak learners, each correcting errors from earlier learners. It frequently performs strongly on tabular banking data, but requires careful validation, calibration and control of complexity.

Supervised models are strongest when labels are timely, representative and sufficiently comprehensive. They can nevertheless miss new fraud typologies, because an attack absent from the training labels is difficult to learn directly.

2.2 Unsupervised and semi-supervised learning

Unsupervised methods identify structure without relying on confirmed fraud labels. Examples include clustering, isolation-based methods, autoencoders and peer-group analysis. They are useful for discovery, especially where labels are sparse or delayed.

Anomaly detection should not be equated with fraud detection. A legitimate customer making an unusually large international purchase may be anomalous but not fraudulent. Conversely, organised fraud may imitate normal behaviour and avoid appearing anomalous. Unsupervised alerts therefore require contextual investigation and feedback.

Semi-supervised approaches use a small labelled set with a larger pool of unlabelled observations. Positive-unlabelled learning is relevant where confirmed fraud is observed but many apparently legitimate records may contain undiscovered fraud. Its assumptions must be made explicit; unlabelled does not necessarily mean negative.

2.3 Hybrid systems

A mature architecture commonly combines:

• deterministic rules for known controls and regulatory requirements;
• supervised scores for known fraud patterns;
• anomaly scores for novel behaviour;
• graph or network features for mule and collusive structures;
• case-management feedback from investigators; and
• human decisions for ambiguous or high-impact cases.

Scores can be combined through a calibrated meta-model, a policy layer or a decision table. The combination should remain auditable: analysts must know which signals contributed to an action and which component can override another.

3. Data, Features and Model Development

Feature design

Useful feature families include:

Velocity features count events over a specified window. For example, v_i,24h may denote the number of transactions for customer or instrument i during the previous 24 hours. The window must be defined relative to the scoring time, not calculated using future records.

Feature engineering should preserve provenance, reproducibility and privacy. Sensitive attributes may be legally restricted or ethically problematic. Removing a sensitive attribute does not guarantee fairness if proxy variables, such as location or language, encode similar information.

Splitting and validation

Random train-test splitting is unsafe when observations are temporal, linked or repeatedly generated by the same entity. A chronological split better represents deployment: train on earlier periods, validate on a later period and test on a still later period. Entity-aware splitting may be needed to prevent the same account or device appearing in both training and test data in a way that exaggerates generalisation.

The test set should remain untouched until model selection is complete. Hyperparameter tuning, feature selection and threshold optimisation belong in training and validation procedures. For production systems, backtesting should simulate delayed labels and the exact information available at decision time.

Imbalance and learning objectives

Fraud is often a small minority of transactions. Accuracy can therefore be misleading. A classifier that labels every transaction legitimate may achieve very high accuracy while detecting no fraud. Options for addressing imbalance include class weights, carefully designed undersampling, oversampling within training folds, focal-type objectives, anomaly methods and threshold optimisation. Resampling must not contaminate validation data or duplicate information across linked observations.

4. Evaluation for Imbalanced Data

Let TP be true positives, TN true negatives, FP false positives and FN false negatives. The confusion matrix should be reported at an explicitly stated threshold.

Precision is the proportion of alerts that are actually fraud. It is important for investigator workload and customer friction.

Recall, also called sensitivity or true-positive rate, is the proportion of fraud cases detected.

Specificity is the proportion of legitimate cases correctly not escalated. The false-positive rate is 1 − Specificity.

The F1 score is the harmonic mean of precision and recall:

It is useful when precision and recall are both important, but it ignores transaction value and may not reflect business costs. The Fβ score gives greater weight to recall when β > 1:

The area under the precision-recall curve (PR-AUC) is often more informative than ROC-AUC when the positive class is rare. ROC-AUC measures ranking across thresholds using true-positive and false-positive rates; it can appear strong even when precision is operationally low. Neither metric alone answers whether the model is profitable or acceptable for a particular queue.

Calibration measures whether predicted probabilities correspond to observed frequencies. Among cases assigned a probability near 0.20, approximately 20% should be positive in a well-calibrated population, subject to sampling variation. Calibration can be assessed with reliability plots, calibration error measures and proper scoring rules such as log loss or the Brier score:

Here, n is the number of observations, p_i is the predicted probability for observation i and y_i is its binary outcome. Lower values are better. The Brier score is sensitive both to calibration and discrimination.

Worked example: threshold performance

Suppose a validation sample contains 10,000 transactions, of which 100 are confirmed fraud. At a selected threshold, the system identifies 70 fraudulent transactions and incorrectly alerts 180 legitimate transactions. Therefore TP = 70, FN = 30, FP = 180 and TN = 9,720.

The alert queue contains 250 cases. Its 28% precision may be acceptable or unacceptable depending on investigator capacity and the cost of contacting legitimate customers. The example illustrates why a model should be evaluated at operational thresholds, not only by a single ranking metric.

5. Thresholds, Ranking and Capacity

A bank may have a fixed investigation capacity of 500 cases daily. In that setting, the practical question may be: how much fraud value is captured by the top 500 scores? This is a top-k or capacity-constrained evaluation. If transaction values differ substantially, value-weighted recall may be more meaningful than case-weighted recall:

This ratio assumes that fraud value is consistently defined and that detected value is not double-counted. It should be reported alongside case recall because a system can capture high-value fraud while missing many low-value cases.

Thresholds can be segment-specific, but segmentation creates governance obligations. The rationale, performance, fairness and stability of each segment should be documented. Thresholds should not be tuned repeatedly on the test set or changed informally in response to a single incident.

6. Explainability and Human Oversight

Explainability has several meanings. Global interpretability describes overall model behaviour; local explanation describes why a particular case received a score; procedural explainability describes the data, controls and decision process surrounding the model. Feature importance is not automatically causal explanation. A variable can be predictive because it is a proxy, a consequence of intervention or a source-system artefact.

Common tools include coefficient inspection, permutation importance, partial-dependence analysis, counterfactual explanations and additive attribution methods. Their outputs should be treated as diagnostic aids, not unquestionable facts. Explanations must be stable enough for analysts and customers, and should avoid revealing sensitive detection logic that would help evasion.

Human oversight is essential when decisions affect account access, essential payments or vulnerable customers. A sound operating model defines:

• which decisions are automated;
• which cases require review;
• reviewer authority and training;
• escalation and appeal paths;
• documentation requirements;
• override monitoring; and
• conditions for suspending the model.

Human review is not automatically a safeguard. Reviewers can suffer from automation bias, fatigue and inconsistent standards. Interfaces should display evidence, uncertainty, relevant history and policy guidance without encouraging blind acceptance of the model.

7. Drift, Feedback and Adversarial Manipulation

Fraud environments are non-stationary. Concept drift occurs when the relationship between features and fraud changes; data drift occurs when feature distributions change; label drift occurs when the prevalence or composition of fraud changes. A model may deteriorate because customer behaviour changes, a new payment channel is launched, a fraud ring adapts or a data pipeline breaks.

Monitoring should include:

• population distributions and missingness;
• score distributions and calibration;
• fraud rate by product, channel and segment;
• precision, recall and value capture after labels mature;
• latency and service availability;
• investigator overrides and queue ageing; and
• fairness and complaint indicators.

Distribution-shift statistics such as the population stability index can be useful diagnostics, but they do not prove performance degradation. A shift must be interpreted with business context and outcome data.

Adversarial manipulation includes probing thresholds, using legitimate credentials, poisoning training data, evading device signals and exploiting feedback loops. Defences include feature diversity, rate limits, secure data provenance, robust validation, out-of-time testing, access controls, adversarial simulations and ensemble or layered controls. Security of the ML pipeline is part of model risk management; the model file, feature store, labels and deployment process are all assets requiring protection.

8. A Detailed Illustrative Case: Instant-Payment Fraud

This is a fictional illustrative case based on common banking operating conditions.

Background and problem

MetroTrust Bank introduced instant account-to-account payments. Within three months, customer complaints increased. Fraudsters were using social engineering to persuade customers to add new beneficiaries, followed by rapid transfers from unfamiliar devices. The bank’s existing rules generated many alerts because legitimate customers also made urgent payments.

Analysis

A cross-functional team defined a fraud label as a payment confirmed through customer dispute and investigation within 30 days. It excluded transactions still unresolved at the evaluation cut-off. The team constructed only features available before authorisation: beneficiary age, time since device registration, recent failed authentication, payment velocity, deviation from historical amount, and links between device, phone number and accounts.

The team used a chronological training-validation-test design. A gradient-boosted model was compared with logistic regression, rules and an anomaly score. The model was calibrated on a validation period. Evaluation included PR-AUC, recall at fixed alert capacity, value recall, calibration, customer-friction rate and performance across customer segments.

Solution

The production design used three layers. Mandatory security rules blocked clearly prohibited activity. A supervised score ranked known-risk payments. A network feature identified clusters in which several new beneficiaries were connected to a small set of devices and receiving accounts. Medium-risk cases triggered a confirmation journey with carefully designed warnings; high-risk cases entered an analyst queue or temporary hold according to policy.

The bank introduced a champion-challenger process. The champion remained in production while a challenger was evaluated on a later time period. Analysts recorded reason codes and outcomes, but those records were not immediately fed into training without quality review. A model-risk committee approved the threshold and specified rollback criteria.

Outcome

After the observation window matured, the hybrid system detected more fraud value at the same investigation capacity and reduced unnecessary customer contacts compared with the original rule set. Performance was weaker for newly launched customer segments because behavioural history was limited. The bank therefore added a cold-start policy based on secure authentication and transaction context rather than simply assigning all new customers high risk.

Learning points

The case demonstrates that model quality depends on label governance, temporal validation, calibrated decisions, operational capacity and human processes. A high-performing algorithm cannot compensate for poor feature availability, delayed labels or an unsuitable customer intervention.

9. Responsible AI, Regulation and Governance

Fraud systems process personal, financial and behavioural data. Governance should address lawfulness, purpose limitation, data minimisation, retention, access, security, contestability and the rights applicable in the relevant jurisdiction. Requirements differ across countries, but banking supervisors generally expect effective controls, documented model risk management, auditability and accountability.

A governance framework should assign ownership across the first line (business and operations), second line (risk, compliance and model validation) and third line (internal audit). Model documentation should record purpose, population, features, labels, assumptions, exclusions, training data, validation, limitations, thresholds, approvals, monitoring and change history.

Fairness requires more than equal average accuracy. Institutions should examine error rates, access restrictions, intervention rates and outcomes across relevant groups, subject to legal and privacy constraints. A fairness metric is not universally optimal: equal opportunity focuses on comparable true-positive rates, while predictive parity concerns comparable precision. When base rates differ, some fairness criteria cannot generally be satisfied simultaneously without trade-offs. These trade-offs must be documented rather than hidden behind a single score.

Privacy-enhancing techniques may include tokenisation, restricted access, aggregation, federated learning or differential privacy. Each introduces limitations, such as reduced utility, implementation complexity or residual re-identification risk. Responsible AI is therefore a continuing risk-management process, not a one-time certification.

10. Generative AI in Fraud Management

Generative AI can assist investigators by summarising case histories, translating communications, drafting customer messages, extracting entities from documents and supporting analyst search. It can also generate synthetic scenarios for testing, although synthetic data must not be assumed to reproduce real fraud distributions. Retrieval-augmented systems can provide answers grounded in approved policies and case records.

Risks include hallucinated explanations, leakage of confidential information, prompt injection, insecure plugins, discriminatory language, overreliance and the generation of convincing phishing or social-engineering content by criminals. A language model should not independently close a case, freeze an account or communicate a legally consequential decision without appropriate controls.

Best practices include approved models and vendors, data classification, redaction, access controls, prompt and output logging, retrieval from authoritative sources, citation checks, human approval, red-team testing and fallback procedures. Generative AI should initially support low-risk, reversible tasks. Its productivity benefit should be measured against error rates, investigation quality and customer outcomes rather than adoption alone.

11. Implementation Blueprint and Best Practices

A practical implementation sequence is:

1. Define fraud typologies, decisions, costs and label windows.
2. Map data lineage and ensure every feature was available at decision time.
3. Establish temporal and entity-aware validation.
4. Build a transparent baseline using rules or logistic regression.
5. Compare supervised, anomaly and network approaches.
6. Calibrate scores and select thresholds using capacity, value and customer-impact analysis.
7. Validate fairness, security, robustness and operational latency.
8. Pilot with human oversight and clear rollback conditions.
9. Monitor drift, outcomes, complaints, overrides and data quality.
10. Retrain or redesign only through controlled change management.

The most reliable systems are layered rather than algorithmically monolithic. They combine prevention, authentication, detection, investigation, recovery and learning. They also distinguish fraud detection from anti-money-laundering surveillance, sanctions screening and cybersecurity analytics, while allowing controlled information sharing where lawful and useful.

Challenges and Limitations

First, confirmed fraud is a biased sample: it reflects what controls and customers reported, not all fraud. Second, delayed labels make real-time evaluation difficult. Third, models can amplify historical investigation bias. Fourth, high performance in one product or country may not transfer to another. Fifth, privacy, explainability and security requirements can conflict with unrestricted feature collection. Sixth, fraudsters adapt strategically, causing model decay. Finally, automation can create harm through false declines, account lockouts and inaccessible appeals.

These limitations favour modest claims, transparent monitoring and continuous validation. The question is not whether AI eliminates fraud; it is whether the overall control system produces better risk-adjusted outcomes while remaining lawful, secure and accountable.

Chapter Summary

Machine learning for fraud detection is a decision-system discipline rather than a contest for the highest accuracy. Supervised models learn from labelled fraud, unsupervised methods discover unusual behaviour, and hybrid systems combine rules, scores, networks and human review. Imbalanced outcomes require precision, recall, PR-AUC, calibration, value capture and capacity-based evaluation rather than accuracy alone. Thresholds should reflect costs, operational capacity and customer impact.

Reliable development requires leakage-free features, temporal and entity-aware validation, mature labels and controlled test data. Production systems must monitor data drift, concept drift, calibration, outcomes, fairness and adversarial manipulation. Explainability should support diagnosis, analyst decisions and appropriate customer communication, while human oversight must be designed to avoid automation bias. Responsible AI requires privacy, security, accountability, documentation, fairness analysis and contestability. Generative AI offers assistance in investigation and knowledge work, but introduces hallucination, confidentiality, manipulation and overreliance risks. Effective banking practice therefore uses layered controls, disciplined governance and continuous learning.

Key Terms

Artificial intelligence: The broad field of systems performing tasks associated with human intelligence.

Calibration: The correspondence between predicted probabilities and observed event frequencies.

Concept drift: Change in the relationship between inputs and the target outcome over time.

False positive: A legitimate transaction incorrectly treated as fraudulent or risky.

False negative: A fraudulent transaction incorrectly treated as legitimate.

Feature leakage: Use of information unavailable at the time a prediction is made.

Fraud typology: A defined pattern or method of fraudulent behaviour.

Human-in-the-loop: A design in which people supervise, review or control automated decisions.

Imbalanced data: Data in which one outcome class is much less frequent than another.

Model drift: Decline or change in model behaviour caused by changing data or relationships.

Precision: The proportion of alerted cases that are actually positive cases.

Recall: The proportion of actual positive cases detected by the system.

Responsible AI: Development and use of AI consistent with fairness, accountability, safety, privacy and lawful purpose.

Semi-supervised learning: Learning that combines limited labels with larger unlabelled data.

Supervised learning: Learning from examples with known outcome labels.

Target leakage: Another term for using future or outcome-derived information in prediction.

Threshold: A score boundary used to map a model output to an action.

Unsupervised learning: Learning patterns without requiring labelled outcomes.

Value recall: The proportion of total fraud value detected by a control.

Review Questions

Short Answer Questions

1. Why can accuracy be misleading in fraud detection?
2. Distinguish precision from recall.
3. What is target leakage, and why is it dangerous?
4. Give two examples of data or concept drift.
5. Why is human oversight important in high-impact banking decisions?

Long Answer Questions

1. Compare supervised, unsupervised and hybrid fraud-detection systems, including their advantages and limitations.
2. Explain how temporal validation, calibration and threshold selection should be conducted for an instant-payment model.
3. Discuss the governance requirements for explainable and responsible AI in banking fraud operations.
4. Analyse how adversarial behaviour can affect an ML fraud pipeline and propose layered defences.
5. Evaluate the opportunities and risks of generative AI in investigation and customer-protection workflows.

Analytical / Application Questions

1. A model identifies 80 fraud cases, misses 20, and raises 200 false alerts. Calculate precision and recall, then explain whether the model is operationally suitable without knowing queue capacity and fraud value.
2. Design a validation plan for a bank launching a new mobile-payment channel when labels arrive 30 days after transactions.
3. A model’s PR-AUC remains stable, but customer complaints and false declines rise sharply in one segment. Diagnose possible causes and recommend immediate and longer-term actions.

Multiple Choice Questions with Answers

Question 1

Which metric is generally most informative for evaluating a rare-positive fraud classifier when ranking quality and alert usefulness are important?

A. Accuracy alone

B. Precision-recall area under the curve

C. Number of model parameters

D. Training-set size alone

Correct Answer: B. Precision-recall area under the curve

Question 2

A transaction has estimated fraud probability p. Under the simplified cost model, intervention is justified when:

A. p × C_FP > (1 − p) × C_FN

B. p × C_FN > (1 − p) × C_FP

C. p = 0 for every transaction

D. C_FP and C_FN are ignored

Correct Answer: B. p × C_FN > (1 − p) × C_FP

Question 3

Which practice most directly prevents target leakage?

A. Using chargeback outcomes as real-time features

B. Calculating features only from information available at scoring time

C. Randomly duplicating the test set

D. Removing all validation data

Correct Answer: B. Calculating features only from information available at scoring time

Question 4

In a fraud investigation queue, precision measures:

A. The proportion of actual fraud cases detected

B. The proportion of alerts that are actually fraud

C. The proportion of legitimate transactions approved

D. The average transaction amount

Correct Answer: B. The proportion of alerts that are actually fraud

Question 5

Why can an unsupervised anomaly score produce false alerts?

A. Every unusual legitimate behaviour is necessarily fraud

B. Legitimate behaviour may be unusual without being fraudulent

C. Unsupervised methods require confirmed labels for every case

D. Anomaly methods cannot process transaction data

Correct Answer: B. Legitimate behaviour may be unusual without being fraudulent

Question 6

Which validation design is usually most appropriate for a production fraud model affected by changing behaviour?

A. A purely random split with future records mixed into training

B. A chronological split that preserves deployment order

C. Testing only on the training data

D. Selecting the threshold on the final test set repeatedly

Correct Answer: B. A chronological split that preserves deployment order

Question 7

A well-calibrated group of transactions assigned probability 0.20 should, approximately, have what observed fraud rate over a sufficiently large comparable sample?

A. 2%

B. 20%

C. 50%

D. 80%

Correct Answer: B. 20%

Question 8

Which is an example of concept drift?

A. A database column is renamed without changing behaviour

B. The relationship between device age and fraud changes after a new attack method appears

C. A model is stored in an encrypted file

D. An analyst changes a case description

Correct Answer: B. The relationship between device age and fraud changes after a new attack method appears

Question 9

Which statement about explainability is most accurate?

A. Feature importance always proves causation

B. A local explanation describes why a particular case received its score, subject to method limitations

C. Explanations eliminate the need for validation

D. Explanations should reveal all detection thresholds publicly

Correct Answer: B. A local explanation describes why a particular case received its score, subject to method limitations

Question 10

Which is an appropriate control for generative AI used in fraud investigations?

A. Allowing unreviewed model outputs to freeze accounts automatically

B. Using approved data, logging prompts and outputs, and requiring human review for consequential actions

C. Sending confidential case files to any public chatbot

D. Disabling all audit records

Correct Answer: B. Using approved data, logging prompts and outputs, and requiring human review for consequential actions