Data Foundations for Fraud Analytics

Introduction

Fraud analytics is often described as a modelling problem, but model performance is fundamentally constrained by the quality and meaning of the data supplied to the model. A sophisticated classifier cannot reliably distinguish legitimate activity from fraud if customer identities are duplicated, timestamps are inconsistent, labels are delayed, devices are shared, or features contain information that would not have been available when a decision was made. In banking, data preparation is therefore not a preliminary technical chore; it is a governed analytical capability.

Fraud data is generated across payment channels, core banking systems, mobile applications, customer relationship platforms, authentication services, device environments, merchant systems, credit bureaux, sanctions and watch-list providers, and investigative case-management tools. These sources differ in structure, granularity, ownership, retention period, reliability and legal status. The central task is to transform them into an integrated analytical representation while preserving provenance, privacy, temporal validity and fairness.

This chapter develops a framework for preparing transactional, customer, device, behavioural and external data for fraud analytics. It focuses on integration, entity resolution, feature engineering, quality control, lineage, privacy, consent, retention and analytical bias. It also examines how these concerns affect model development, operational decision-making and governance.

Learning Objectives

By the end of this chapter, readers should be able to:

1. Distinguish the principal data domains used in banking fraud analytics.
2. Design a reliable pipeline for integrating heterogeneous fraud data.
3. Explain deterministic and probabilistic entity-resolution methods and their risks.
4. Engineer temporally valid transactional, behavioural and network features.
5. Apply data-quality controls and quantify important quality dimensions.
6. Construct lineage, privacy, consent and retention controls for analytical data.
7. Identify sources of sampling, label, measurement and proxy bias.
8. Evaluate an applied fraud-data architecture from technical, managerial and ethical perspectives.

1. The Data Model of Fraud

1.1 Data domains

A fraud-analytics environment normally combines several domains:

A useful conceptual distinction is between events, entities and relationships. A transaction is an event occurring at a time. A customer, device, merchant and account are entities. A customer owning an account, a device accessing several accounts, or a merchant receiving payments are relationships. Fraud frequently appears in the relationships rather than in an isolated event. Consequently, a relational or graph-oriented representation may complement conventional tabular data.

1.2 Grain and the unit of analysis

The grain of a dataset states what one row represents. Examples include one transaction, one login session, one customer-day, one device-day or one investigation case. Many analytical errors arise when datasets with different grains are joined without aggregation. Joining a customer-level table directly to a transaction table is legitimate only when the intended result is transaction-level and the customer attributes are appropriately time-stamped. Joining two transaction-level tables on customer identifier alone can create a many-to-many multiplication and produce false frequencies and amounts.

Before integration, analysts should document:

• the row-level business event;
• the primary key and its uniqueness rule;
• event time and processing time;
• valid-from and valid-to periods for changing attributes;
• expected cardinality of joins; and
• whether the table contains facts, estimates, decisions or derived values.

1.3 Event time and decision time

Fraud models must respect the information available at the moment of a decision. Let t_e denote event time and t_d decision time. A feature is usable for real-time scoring only if its source information was available no later than t_d, not merely if it describes an event occurring before t_e. A delayed chargeback outcome, for example, may describe an earlier transaction but cannot be used to make the original authorization decision.

This distinction prevents temporal leakage, in which future information inadvertently enters training data. Feature computation should therefore use an as-of timestamp and, where possible, source-system availability timestamps.

2. Integration Architecture and Data Contracts

2.1 From source systems to analytical products

A robust architecture commonly contains four logical layers:

1. Source layer: operational databases, event streams, files and external feeds.
2. Raw or landing layer: immutable copies preserved with ingestion metadata.
3. Curated layer: standardized, validated and conformed entities and events.
4. Feature and serving layer: point-in-time features delivered to models, analysts and decision engines.

The architecture may be implemented using batch processing, streaming, or a hybrid approach. Streaming is valuable for immediate authorization and account-takeover detection, whereas batch processing remains appropriate for retrospective investigations, periodic feature computation and model monitoring.

A data pipeline should not merely move records. It should make transformations reproducible. Each transformation should specify its input, output, owner, logic, quality checks, timestamp semantics and failure response.

2.2 Data contracts

A data contract is an explicit agreement between a producer and consumer concerning schema, semantics, quality and operational behaviour. It should include field definitions, data types, permitted values, nullability, uniqueness, delivery frequency, latency, versioning and escalation procedures. For example, an amount field must state whether it is expressed in minor currency units, major currency units, or a decimal amount; a status field must distinguish declined, reversed, pending and completed transactions.

Schema compatibility is not sufficient. A field may retain the same technical type while its business meaning changes. A merchant category code that is remapped without notice can invalidate historical features. Semantic versioning, change approval and backward-compatibility tests reduce this risk.

2.3 Standardisation and temporal harmonisation

Integration requires common conventions for:

• time zones and daylight-saving transitions;
• currency and foreign-exchange treatment;
• country, language and address codes;
• phone and email formats;
• account and customer identifiers;
• transaction status and reversal logic; and
• character encoding and missing-value representation.

Amounts should not be compared across currencies without a defined conversion policy. If an amount in currency c is converted using rate r_c into a reporting currency, the converted amount is:

Here A_c is the amount in currency c, and r_c is the number of base-currency units per unit of currency c. The rate date, source, bid/ask convention and treatment of fees must be recorded. This formula is meaningful only when the rate direction and valuation time are defined.

3. Entity Resolution and Relationship Integrity

3.1 The identity problem

The same real-world customer may appear under multiple identifiers because of spelling variation, changed contact details, multiple products, migration between systems or deliberate identity manipulation. Conversely, different people may share an address, device, telephone number or network. Entity resolution must therefore balance false merges against missed matches.

A golden record is a governed representation of an entity that links source identifiers to a surviving analytical identity. It should preserve the source keys rather than erase them, because investigators may need to trace a result back to the originating system.

3.2 Deterministic matching

Deterministic matching applies exact or rule-based conditions, such as a verified national identifier, an authenticated customer key or a stable account relationship. It is transparent and often appropriate for high-confidence identifiers. Its weakness is brittleness: formatting changes, missing values and shared identifiers reduce recall.

Rules should distinguish strong identifiers from weak attributes. An exact match on a verified customer number may be conclusive; an exact match on surname and postcode usually is not.

3.3 Probabilistic matching

Probabilistic matching compares fields and estimates whether two records refer to the same entity. A classical record-linkage approach compares agreement patterns across fields. Let γ denote the observed comparison pattern for two records, M the event that they represent the same entity, and U the event that they do not. A likelihood-ratio score is:

Large values support a match; small values support a non-match. In practice, thresholds may define automatic matches, automatic non-matches and clerical review. The probabilities must be estimated from representative data, and the fields should not be treated as independent unless that assumption is reasonable. Correlated evidence, such as address and postcode, can otherwise exaggerate confidence.

Modern systems may use embeddings or supervised matching models, but explainability, threshold governance, bias testing and human review remain essential. A false merge can contaminate many customers and create unjustified suspicion; a missed match can conceal a fraud network.

3.4 Network and graph considerations

After resolution, relationships can be represented as a graph whose nodes include customers, accounts, devices, merchants, addresses and payment instruments. Edges may represent ownership, access, payment, common contact details or temporal co-occurrence. Useful controls include limits on shared identifiers, review of unusually dense components, and investigation of rapid changes in relationships.

Network signals are not proof of fraud. Shared devices in a household, common Wi-Fi in a workplace and payment processors serving many merchants can create legitimate high-connectivity patterns. Context and explainability are necessary before operational action.

4. Feature Engineering with Temporal Integrity

4.1 Principles

Feature engineering converts raw observations into variables that represent risk-relevant behaviour. A sound feature should have a clear definition, a known time window, a reproducible computation, a business rationale and a documented missing-value policy. Features should be calculated from information available at scoring time and should be stable enough for monitoring.

Common feature families include:

• Recency: time since the last login, device change or payment.
• Frequency: number of events in a defined interval.
• Velocity: count or value accumulated over a short interval.
• Deviation: difference from a customer’s historical norm.
• Sequence: ordering of authentication, beneficiary creation and payment.
• Network: number and type of connected entities.
• Outcome: prior declines, disputes or confirmed fraud, subject to label timing.

4.2 Aggregation and velocity

For an entity i, event values x_j and a look-back window of length W, a count feature at time t can be defined as:

Here 1 equals one when event j satisfies the condition and zero otherwise. The interval convention must be stated; the expression excludes the left endpoint and includes the right endpoint. For an authorization decision, the current event is often excluded from historical features to avoid circularity, so the upper bound may be less than t.

A value-velocity feature is:

where a_j is the transaction amount in a defined currency basis. Counts and sums should not be compared without considering exposure, customer segment and transaction opportunities.

4.3 Baselines and deviation

A customer-specific baseline may use a historical mean. For observations x₁, x₂, …, x_n, the arithmetic mean is:

A standardised deviation is:

where s is the sample standard deviation, defined for n > 1 as:

This measure assumes that the baseline is sufficiently representative and that the standard deviation is not zero. It can be distorted by highly skewed transaction values, behavioural change and fraud already present in the historical window. Robust alternatives, such as medians and interquartile ranges, may be preferable.

4.4 Missingness as information

Missing values may indicate system failure, optional data, customer choice or an unusual process. Replacing all missing values with zero can falsely imply that an event did not occur. A feature should often be accompanied by a missingness indicator, and the business meaning of missingness should be documented. Imputation must be performed within appropriate training partitions to prevent information leakage.

4.5 Labels and delayed outcomes

Fraud labels may derive from chargebacks, customer reports, investigator decisions, law-enforcement information or confirmed recovery. These labels differ in reliability, timing and coverage. A transaction not yet reported is not necessarily legitimate. Analysts should define an observation window and a label maturity period. For example, a training cohort may include transactions from an historical period and assign labels only after sufficient time for disputes and investigations to mature.

Class imbalance is common. The prevalence of fraud in a sample is:

where N_fraud is the number of labelled fraudulent cases and N_total is the total number of cases in the defined population. Sampling, delayed labels and operational queues can make observed prevalence differ from real prevalence. Evaluation should therefore report precision, recall, false-positive rate, financial impact and performance at the operating threshold rather than accuracy alone.

5. Data Quality and Control Framework

Data quality is multidimensional. A dataset may be complete but inaccurate, timely but inconsistent, or valid in format but wrong in meaning.

A simple completeness rate for a field is:

where n_nonmissing is the count of non-missing values and n_expected is the number of records for which the field is expected. This is a descriptive metric, not proof of correctness. Quality thresholds must be risk-based. A missing optional marketing attribute may be tolerable, whereas a missing transaction currency may make a financial feature unusable.

Controls should operate at ingestion, transformation and serving stages. Failed checks should produce alerts, quarantine affected data where appropriate, and record the decision to continue, delay or degrade a model. Silent correction is dangerous because it destroys evidence of the original defect.

6. Lineage, Privacy, Consent and Retention

6.1 Lineage and reproducibility

Data lineage records where a field originated, how it was transformed, when it changed and which analytical products use it. At minimum, lineage should connect source field, transformation logic, feature version, model version and decision outcome. This supports audit, incident investigation, model validation and reproducibility.

A feature catalogue should state the feature owner, definition, grain, time window, source systems, refresh frequency, known limitations, permitted uses and retirement conditions. Versioning is particularly important when a feature’s definition changes while its name remains the same.

6.2 Privacy and purpose limitation

Fraud prevention is a legitimate and important banking function, but it does not remove privacy obligations. Data collection and use should be necessary, proportionate and connected to a defined purpose. Sensitive personal data requires additional safeguards, and access should follow least privilege. Pseudonymisation reduces direct exposure but does not necessarily make data anonymous, especially when multiple datasets can be linked.

Privacy-preserving practices include tokenisation of account identifiers, encryption in transit and at rest, role-based access, controlled analyst workspaces, field-level masking, audit logs and secure deletion. Data protection impact assessments should address linkage, profiling, automated decisions, international transfers and the possibility of re-identification.

Consent is only one possible lawful basis and is not automatically appropriate for every fraud-control activity. Organisations must distinguish consent for a particular service from permission to reuse data for unrelated analytics. Notice, transparency and rights-management processes should be aligned with applicable law and institutional policy.

6.3 Retention and deletion

Retention should be based on legal requirements, fraud-investigation needs, model-development value and storage risk. Keeping data indefinitely increases exposure and may preserve obsolete or unfair information. Retention schedules should specify the period, triggering event, archive conditions, legal hold process and deletion method. Derived features and model outputs also require retention decisions; deleting raw data while retaining re-identifiable derived records may not achieve the intended privacy objective.

7. Analytical Bias and Responsible Use

Bias can enter through the population selected, the labels recorded, the variables measured and the actions taken after a model produces a score.

• Sampling bias: reported fraud reflects who notices and reports fraud, not necessarily all fraud.
• Selection bias: only transactions routed to an alert queue may receive investigative labels.
• Measurement bias: device or location data may be less accurate for some customers or channels.
• Label bias: investigator decisions can reflect workload, experience or unequal scrutiny.
• Historical bias: past rules and institutional practices may encode unequal treatment.
• Proxy bias: apparently neutral variables may encode protected or sensitive characteristics.
• Feedback-loop bias: customers flagged more often generate more recorded evidence, reinforcing the model’s apparent justification.

Fairness assessment should be appropriate to the use case and legal context. Analysts may compare error rates, calibration, alert rates and financial impact across relevant groups, while recognising that group definitions can be sensitive and incomplete. No single fairness metric resolves all trade-offs. A reduction in false positives for one group may alter recall or operational capacity elsewhere.

Fraud detection also differs from credit underwriting: the objective is to identify suspicious activity, not to deny opportunity based on a predicted propensity. Nevertheless, adverse effects, explainability and appeal or review mechanisms remain important. Human investigators should receive relevant evidence rather than an unexplained score, and overrides should be monitored for systematic patterns.

8. Applied Case: Integrating Data for Account-Takeover Detection

The following is an illustrative case, not a report of a particular bank.

Background

A retail bank observes a rise in account-takeover losses affecting mobile and online banking. Attackers obtain credentials through phishing, register new devices, add beneficiaries and initiate transfers. Existing rules generate many alerts because legitimate customers also change devices while travelling or replacing phones.

Problem

The bank’s transaction table uses account identifiers, the mobile platform uses device tokens, and the identity platform uses customer identifiers. Timestamps are recorded in local time in one system and UTC in another. Investigation outcomes arrive several weeks after the transfer. Shared household devices produce apparent links among unrelated customers.

Analysis

A data-quality review finds duplicate mobile events caused by retry messages, inconsistent handling of reversed transfers and a high null rate for device location. A direct join of customer and transaction tables also multiplies records because multiple account relationships are active over time. The bank therefore defines event grain, idempotency keys, event time, processing time and effective-dated customer-account relationships.

Entity resolution links verified customer keys to accounts and tokenised device identifiers. Device sharing is retained as a relationship rather than treated as proof that all linked customers are one person. Features are computed as of each transfer decision: hours since device registration, number of beneficiaries added in the previous 24 hours, transfer velocity, deviation from the customer’s usual channel and the number of accounts accessed by the device.

The label pipeline waits for a defined outcome-maturity period and distinguishes confirmed fraud, customer-disputed transactions, investigator suspicion and unresolved cases. The model-development sample is split by time, ensuring that future labels and future feature values are unavailable during training.

Approach and solution

The bank introduces:

1. immutable raw event storage with ingestion metadata;
2. a canonical event schema and time-zone standard;
3. data contracts for mobile, identity and payments teams;
4. point-in-time feature generation;
5. quality thresholds for duplication, missingness and event latency;
6. lineage from feature to source event and model version;
7. investigator explanations based on recent device and beneficiary changes; and
8. fairness and false-positive monitoring across customer segments and access channels.

A staged decision process uses a low-friction authentication challenge for moderate risk and human review or transaction delay for high risk. The bank avoids automatically blocking every transaction associated with a shared device.

Outcome and learning points

The principal improvement is not merely a higher model score. The bank gains trustworthy temporal data, more precise investigative evidence and a controlled way to respond when source quality deteriorates. The case demonstrates that fraud analytics depends on the alignment of data grain, identity, time, labels, governance and operational action. It also shows why a technically accurate link can still be an inappropriate basis for a customer decision.

9. Implementation Guidance and Contemporary Developments

A mature implementation should establish a data-governance council involving fraud operations, engineering, privacy, legal, model risk, customer protection and business owners. It should maintain a critical-data-element register, data-quality service-level objectives, feature documentation and incident-response procedures.

Event-driven architectures and online feature stores can reduce scoring latency, but they introduce consistency challenges between online and offline features. Feature parity testing is necessary so that a model is trained and served using equivalent definitions. Graph analytics can expose coordinated activity, while privacy-enhancing technologies such as secure enclaves, federated learning or privacy-preserving record linkage may support collaboration without unrestricted data pooling. These technologies do not eliminate governance obligations: leakage, re-identification, model drift and unequal data coverage remain possible.

Generative AI may assist investigators by summarising evidence or translating complex event sequences into natural language. It should not invent facts, conceal uncertainty or replace authoritative records. Retrieval controls, citation of source events, access restrictions and human review are essential.

Challenges and Limitations

Data integration is difficult because systems evolve at different speeds and because fraud itself adapts to controls. Stable identifiers may be unavailable, external data may have uncertain provenance, and labels may be incomplete or delayed. Real-time processing can create race conditions, duplicate messages and inconsistent state. Retrospective corrections can make a feature impossible to reproduce unless historical versions are preserved.

No data foundation can guarantee fraud detection. Legitimate behaviour changes, attackers imitate normal customers, and controls create economic and customer-experience trade-offs. High-quality data may also enable excessive surveillance if purpose limitation and access controls are weak. Consequently, data engineering, model validation, privacy governance and operational accountability must be designed as one system.

Chapter Summary

Fraud analytics begins with a defensible representation of events, entities and relationships. Transactional, customer, device, behavioural, case and external data must be integrated through explicit grains, data contracts, common semantics and temporal controls. Entity resolution should combine strong deterministic evidence with carefully governed probabilistic methods, while preserving uncertainty and source traceability.

Feature engineering is useful only when definitions, windows, units, missingness and availability are explicit. Velocity, recency, deviation and network features must exclude future information and use labels only after suitable maturity. Data-quality controls should measure completeness, validity, accuracy, consistency, uniqueness, timeliness and integrity, with risk-based thresholds and documented failure responses.

Lineage enables audit and reproducibility. Privacy, consent, access control and retention constrain legitimate analytical use, even when fraud prevention is an important purpose. Sampling, label, measurement, historical and proxy bias can distort both models and decisions. A responsible fraud-data foundation therefore combines technical reliability with fairness assessment, explainability, human oversight and continuous governance.

Key Terms

Analytical grain: The business meaning of one row in a dataset.

As-of time: The time at which information is considered available for a decision.

Data contract: An agreement defining a data product’s schema, semantics, quality and operational expectations.

Data lineage: The documented origin and transformation history of data.

Data quality: The degree to which data is fit for its intended use.

Entity resolution: The process of determining which records refer to the same real-world entity.

Feature engineering: Transformation of raw data into variables used for analysis or modelling.

Golden record: A governed analytical representation of an entity linked to source records.

Graph representation: A model of nodes and relationships among entities and events.

Label maturity: The time required before an outcome can be observed with reasonable reliability.

Likelihood ratio: A comparison of the probability of observed evidence under a match and a non-match hypothesis.

Point-in-time correctness: The condition that a feature uses only information available at the relevant decision time.

Pseudonymisation: Replacing direct identifiers with controlled substitutes while retaining a possibility of linkage.

Temporal leakage: Use of future or unavailable information in training or scoring.

Velocity feature: A count or value accumulated over a specified time window.

Sampling bias: Systematic distortion caused by a sample that does not represent the target population.

Proxy variable: A variable that indirectly encodes a sensitive or protected characteristic.

Retention schedule: A documented rule specifying how long data is kept and when it is deleted or archived.

Review Questions

Short Answer Questions

1. Why is dataset grain important when integrating customer and transaction data?
2. Distinguish event time from processing time.
3. What is the purpose of a data contract?
4. Give two examples of temporal leakage in fraud analytics.
5. Name four dimensions of data quality.

Long Answer Questions

1. Explain how deterministic and probabilistic entity resolution can be combined in a banking fraud environment. Discuss the risks of false matches and missed matches.
2. Describe a point-in-time-correct feature-engineering process for payment fraud detection.
3. Discuss how lineage, privacy, consent and retention requirements affect the design of a fraud-data platform.
4. Explain the main sources of analytical bias in fraud labels and propose controls to reduce them.
5. Evaluate the role of graph-based relationships in fraud analytics, including legitimate explanations for high connectivity.

Analytical / Application Questions

1. A bank joins two transaction tables using customer identifier only and observes a sharp increase in transaction counts. Diagnose the likely data problem and design validation tests.
2. A device is associated with 18 accounts, five of which have received fraud alerts. Explain why this is not conclusive proof of fraud and specify additional temporal, behavioural and relationship features you would investigate.
3. A real-time model uses a feature derived from customer disputes received within 30 days after a payment. Assess whether the feature is valid at authorization time and recommend an alternative design.

Multiple Choice Questions with Answers

Question 1

Which statement best describes point-in-time correctness?

A. A feature is calculated using the most recent data available at any time.

B. A feature uses only information available when the relevant decision was made.

C. A feature contains no missing values.

D. A feature is refreshed at least once per day.

Correct Answer: B. A feature uses only information available when the relevant decision was made.

Question 2

What is the principal danger of joining two transaction-level tables on customer identifier alone?

A. Currency conversion error

B. Many-to-many row multiplication

C. Loss of all customer attributes

D. Automatic pseudonymisation

Correct Answer: B. Many-to-many row multiplication

Question 3

In the likelihood-ratio expression for record linkage, a large value generally indicates that the comparison pattern is:

A. More consistent with a true match than a non-match

B. More likely to contain a missing value

C. Proof that two records belong to different people

D. Independent of the comparison fields

Correct Answer: A. More consistent with a true match than a non-match

Question 4

Which practice best reduces temporal leakage?

A. Using future investigation outcomes as training features

B. Computing features with an as-of timestamp

C. Replacing all missing values with zero

D. Randomly shuffling all events before feature creation

Correct Answer: B. Computing features with an as-of timestamp

Question 5

A completeness rate for a required field is calculated as:

A. Invalid records divided by valid records

B. Non-missing expected values divided by expected records

C. Duplicate records divided by total records

D. Late records divided by missing records

Correct Answer: B. Non-missing expected values divided by expected records

Question 6

Why should a shared device not automatically be treated as proof that linked accounts belong to one fraudster?

A. Devices cannot be identified technically.

B. Shared devices may occur in legitimate households or workplaces.

C. Device data is always legally unusable.

D. Graph relationships cannot be analysed.

Correct Answer: B. Shared devices may occur in legitimate households or workplaces.

Question 7

Which item is most appropriately recorded in a data lineage system?

A. An employee’s personal opinion about a customer

B. The source field and transformation used to create a feature

C. A model’s unverified prediction as historical truth

D. Only the final model score

Correct Answer: B. The source field and transformation used to create a feature

Question 8

What is label maturity in fraud analytics?

A. The age of a machine-learning model

B. The time required before an outcome can be observed reliably

C. The period for encrypting a database

D. The time needed to train an investigator

Correct Answer: B. The time required before an outcome can be observed reliably

Question 9

Which is an example of proxy bias?

A. A currency field using the wrong decimal separator

B. A neutral-looking variable indirectly encoding a sensitive characteristic

C. A duplicate event caused by message retry

D. A feature being calculated in UTC

Correct Answer: B. A neutral-looking variable indirectly encoding a sensitive characteristic

Question 10

If a fraud sample contains 240 confirmed fraud cases among 12,000 transactions, what is the observed fraud prevalence?

A. 0.2%

B. 2%

C. 5%

D. 20%

Correct Answer: B. 2%