Insurance Data, Measurement and Architecture

Introduction

Predictive analytics in insurance is often described as a modelling discipline, but modelling is only one stage in a longer information chain. An insurer must first identify what is being insured, when exposure exists, how transactions are recorded, which events generate claims, and whether the available data were created for the analytical purpose now intended. A technically sophisticated algorithm trained on poorly defined or historically biased data can produce precise-looking but unreliable decisions.

Insurance data are distinctive because they combine contracts, financial transactions, uncertain future events, human behaviour, physical assets, geography and regulation. A policy may be amended several times; a claim can develop over months or years; premiums may be earned over time rather than at the date of payment; and a customer may interact through an intermediary, a digital channel, a repair network or a call centre. These features make time, provenance, measurement and lineage central to predictive analytics.

This chapter develops a framework for understanding insurance data and the architecture through which data become usable analytical assets. It covers policy, exposure, premium, claims, customer, intermediary, telematics, geospatial and external data. It then examines data quality, missingness, leakage, bias and temporal structure before relating warehouses, data lakes, APIs and cloud platforms to insurance analytics.

Learning objectives

By the end of this chapter, the reader should be able to:

1. Distinguish the principal classes of insurance data and explain how they support underwriting, pricing, claims and customer decisions.
2. Define exposure, written premium, earned premium, incurred loss and loss ratio, while recognising their measurement assumptions.
3. Assess data provenance, quality, lineage, missingness, leakage, bias and temporal structure.
4. Design a conceptual data architecture linking operational systems, analytical storage, governance and model deployment.
5. Evaluate the professional, ethical, regulatory and managerial implications of using behavioural, geospatial and external data.
6. Identify appropriate controls for reproducible and decision-useful predictive analytics.

1. The insurance information model

Insurance data can be understood through four related questions:

• Object: What is being observed—policy, vehicle, person, property, claim or payment?
• Event: What happened—quotation, inception, endorsement, accident, settlement or cancellation?
• Time: When did it happen, when was it recorded, and for which coverage period does it apply?
• Purpose: Was the data collected for administration, accounting, regulation, service, fraud control or modelling?

A useful analytical record therefore contains more than a value. It should also carry an identifier, timestamp, source, definition, unit, version and access history. For example, “premium = 500” is incomplete unless the analyst knows the currency, whether the amount is gross or net of commission and tax, whether it is written or earned, the coverage period, and the accounting basis.

1.1 Core insurance entities

The following entities commonly form the basis of an insurer’s conceptual data model.

Entity resolution is essential. A customer may appear under different names, addresses or identifiers in policy administration, claims and billing systems. Poor linkage can create duplicate customers, incorrectly joined claims or leakage between training and validation samples. A persistent, governed identifier is preferable to joining records on names or approximate addresses alone.

2. Major data domains in insurance

2.1 Policy and contract data

Policy data describe the contractual relationship between insurer and insured. They include product type, coverage sections, limits, exclusions, deductibles, inception and expiry dates, endorsements, cancellations, renewal status, payment plan and distribution channel.

Policy data are central to underwriting and pricing, but they require versioning. An endorsement changes the contract after inception; overwriting the original record destroys the historical state. A better design stores effective dates and transaction dates. The effective date indicates when a change applies to the risk, whereas the transaction date indicates when the system recorded it. The difference matters when reconstructing what was known at a historical decision point.

2.2 Exposure data

Exposure measures the opportunity for an insured event to occur. It is not necessarily the number of policies. Examples include policy-years for motor insurance, property-years for household insurance, payroll or employee-years for workers’ compensation, and insured value-months for commercial property.

If 1,000 vehicles are insured for six months each, exposure is 500 vehicle-years. In general, for homogeneous units,

where E is total exposure in exposure-years and tᵢ is the insured time for unit i, expressed in years. The formula assumes that the chosen unit is meaningful and that exposure is measured over the relevant coverage period.

Exposure is the denominator of many insurance rates. A count of 50 claims means little without knowing whether the portfolio had 500 or 50,000 vehicle-years. Exposure can also be an amount, such as kilometres driven or payroll, when that quantity better represents opportunity for loss.

2.3 Premium data and measurement

Premium may be recorded in several forms:

• Written premium: premium attached to policies issued or renewed during a period.
• Earned premium: the portion of written premium relating to protection already provided.
• Gross premium: before reinsurance ceded and sometimes before other deductions, depending on the reporting convention.
• Net premium: after specified deductions, often including ceded reinsurance.
• Technical or risk premium: an analytical estimate of expected cost of risk, distinct from the market premium charged.

For a simple constant-risk policy with premium P covering a period from inception to expiry, the earned amount over a time interval is approximately proportional to elapsed covered time. If a 12-month policy costs 1,200 monetary units and four months of cover have elapsed under a pro-rata assumption, earned premium is 400 monetary units. This is a simplified illustration; actual accounting may use daily earning, special earning patterns or applicable financial reporting requirements.

Premium records should retain currency, tax treatment, commission, fees, instalment status, payment date and refund information. Confusing invoice date with coverage date can introduce temporal errors into profitability or lapse models.

2.4 Claims data

Claims data usually contain notification, loss, assessment, reserve and payment events. Important dates include:

• date of loss;
• date reported;
• date registered;
• date assessed;
• payment dates;
• closure and reopening dates.

These dates answer different questions. A claim reported in March for a loss occurring in January belongs to January for occurrence analysis but may belong to March for claims-operational workload analysis.

Claim cost can be represented as paid loss, case reserve, incurred loss or ultimate loss. A common operational measure is incurred loss:

where I is incurred loss, P is cumulative paid loss and R is the outstanding case reserve, under the stated accounting convention. This identity excludes or includes components such as claim expenses according to the organisation’s definition. It should not be treated as ultimate loss: ultimate loss additionally reflects future development and may require actuarial estimation.

A basic loss ratio is:

Both numerator and denominator must relate to the same portfolio, period, currency and accounting basis. A loss ratio above 100% indicates that incurred loss exceeds earned premium before considering other expenses; it does not by itself measure total profitability.

2.5 Customer and intermediary data

Customer data may include contact details, tenure, interactions, complaints, payment behaviour, consent status and product holdings. Intermediary data include agent or broker identifiers, appointments, channel, commission, geographic territory, quote conversion and persistency.

These data enable retention analysis, service prioritisation and distribution management. However, customer and intermediary attributes can also encode protected or proxy characteristics. Postal area, language, occupation or channel may correlate with legally sensitive characteristics. Their predictive value must therefore be considered alongside fairness, explainability, necessity and regulatory requirements.

2.6 Telematics and Internet-of-Things data

Telematics data can include timestamp, latitude and longitude, speed, acceleration, braking, distance, trip duration and device quality. In motor insurance, such data may support usage-based insurance, driving feedback, theft recovery and claims reconstruction. In property insurance, sensors may report temperature, water leakage or occupancy-related signals.

Telematics creates measurement issues. A missing GPS interval may mean poor reception rather than safe driving. A harsh-braking event depends on device calibration, road conditions and sampling frequency. Data minimisation is important: collecting continuous location data when aggregated distance is sufficient increases privacy and governance risk.

2.7 Geospatial data

Geospatial information includes coordinates, administrative areas, road networks, elevation, flood zones, crime statistics, distance to emergency services and environmental hazards. It may be used for accumulation management, catastrophe pricing, claims triage and underwriting.

Spatial variables are not automatically objective. A postcode can act as a proxy for socioeconomic or ethnic composition, and hazard maps may vary by resolution, date and modelling method. Analysts should record the source, geographic reference system, resolution, release date and version of every external spatial layer.

2.8 External data

External data may include weather, economic indicators, public registers, building characteristics, market prices, traffic conditions, satellite observations and catastrophe models. Before use, the insurer should establish legal rights, update frequency, methodology, quality, representativeness and permitted purposes.

The fact that data are publicly accessible does not mean that their use is unrestricted or fair. External data can contain historical institutional bias, incomplete coverage and unstable relationships. Validation should test whether the variable remains reliable across regions, customer groups and time.

3. Measurement frameworks and insurance metrics

Insurance analytics depends on carefully aligned numerators, denominators and time windows. Common metrics include claim frequency, average severity, loss ratio, expense ratio, combined ratio, retention and conversion.

Claim frequency is often defined as:

Average severity is:

Under a simple decomposition, expected loss cost per exposure unit is:

This relationship is useful when frequency and severity are defined consistently and their product is interpreted as an expected average cost. It is not a claim that every individual claim equals the portfolio average, nor does it account automatically for dependence, development, inflation or policy limits.

For example, suppose a portfolio has 2,000 vehicle-years, 80 claims and incurred losses of 240,000 monetary units. Frequency is 80 ÷ 2,000 = 0.04 claims per vehicle-year. Severity is 240,000 ÷ 80 = 3,000 monetary units per claim. The implied loss cost is 0.04 × 3,000 = 120 monetary units per vehicle-year. The calculation is internally consistent because the exposure unit and loss period are aligned.

The combined ratio is commonly expressed as:

The precise expense definition varies. Acquisition costs, administrative expenses, premium taxes and other items may be treated differently across organisations and reporting bases. Analysts must consult the metric dictionary rather than assuming universal definitions.

4. Data quality, provenance and lineage

4.1 Dimensions of data quality

Data quality is multidimensional. A dataset may be accurate but untimely, complete but incorrectly defined, or consistent internally but wrong relative to reality.

A quality rule should specify the field, condition, threshold, owner, monitoring frequency and remediation process. For example, “at least 98% of new claims must contain a valid policy identifier within 24 hours” is more actionable than “claims data must be good.”

4.2 Provenance and lineage

Provenance records where data originated and under what circumstances they were collected. Lineage records how data move and change from source to report, feature or decision. Lineage should cover extraction, transformation, joins, aggregation, imputation, feature creation, model version and deployment.

A lineage record allows an analyst to answer: Which source produced this value? What code transformed it? Which version was active on the decision date? Who approved the transformation? Can the result be reproduced?

Data dictionaries and business glossaries support semantic consistency. Terms such as “active policy,” “new customer,” “open claim” and “earned premium” require explicit definitions, including inclusion rules and time basis.

4.3 Missing data

Missingness is not a single phenomenon. A value may be missing because it was not applicable, not collected, unavailable at the time, deliberately withheld or lost through a system failure. Treating all missing values as zero is especially dangerous: zero claims, zero mileage and unknown deductible are not equivalent.

Three classical labels are useful conceptually:

• MCAR: missing completely at random;
• MAR: missingness related to observed variables after conditioning on them;
• MNAR: missingness related to unobserved or missing values themselves.

These labels are assumptions about a data-generating process, not properties that can always be proven from the dataset. In insurance, missingness is often informative. Customers with complex risks may provide incomplete information; telematics may be missing in areas with weak connectivity; claims fields may be absent for recently reported claims.

Practical responses include separate “unknown” categories for categorical fields, carefully justified imputation, missingness indicators, source-system correction and sensitivity analysis. Imputation parameters must be estimated using training data only when evaluating predictive performance, otherwise information can flow from validation data into training.

5. Leakage, bias and temporal structure

5.1 Data leakage

Leakage occurs when a model uses information unavailable at the intended prediction time, or information caused by the outcome. Examples include using final claim settlement status to predict claim severity at notification, using a cancellation flag updated after the lapse event, or calculating a customer’s annual claims count using claims that occurred after the quote date.

A useful test is the as-of question: “Could this value have been known, in this exact form, at the moment the decision was made?” If not, it must be excluded, delayed, reconstructed historically or treated as a later-stage variable.

Leakage can also arise through preprocessing. Randomly splitting repeated records from the same policy across training and test sets may allow near-duplicate information to cross the boundary. Target encoding performed before cross-validation can similarly expose outcome information.

5.2 Bias and representativeness

Bias may enter through sampling, measurement, labels, historical decisions, missingness, proxy variables or deployment processes. A claims dataset does not represent all losses; it represents losses reported, accepted, investigated and recorded under particular organisational practices. A fraud label may reflect investigation intensity rather than fraud itself.

Important questions include:

• Which populations are absent or underrepresented?
• Does measurement quality differ by region, device, language or channel?
• Are historical labels affected by previous decisions?
• Does model error differ materially across relevant groups?
• Is the variable necessary and proportionate to the decision?

Fairness assessment should be linked to the decision context and applicable law. No single statistical fairness measure is universally correct. Trade-offs may exist between calibration, equal error rates, privacy and operational utility. Governance should document the chosen approach and its rationale.

5.3 Temporal structure

Insurance data are inherently longitudinal. Policies renew, claims develop, economic conditions change and underwriting rules evolve. Random train-test splitting can overstate performance when future-like records appear in training.

For forecasting or prospective prediction, a chronological split is generally more realistic: earlier observations are used for training and later observations for validation or testing. Rolling-origin evaluation can repeat this process across several historical cut-off dates. The design must also respect claim development and reporting delays.

Important temporal concepts include:

• Event time: when the insured event occurred;
• Observation time: when the event was recorded;
• Decision time: when a model decision was made;
• As-of date: the latest information permitted for that decision;
• Censoring: when the full future outcome is not yet observable.

A policy renewed in 2025 may have a 2024 claim history, but a model trained in 2024 must not use a 2025-developed claim outcome. Time-aware data marts should preserve snapshots rather than reconstructing history from today’s overwritten fields.

6. Architecture for insurance analytics

A modern architecture commonly separates operational capture, analytical storage, transformation, governance and consumption.

Figure: A layered architecture for transforming insurance source data into governed analytical products.

6.1 Operational systems and ingestion

Operational systems are designed to process transactions reliably: policy administration, claims management, billing, CRM, broker platforms and telematics services. They are not necessarily optimised for large-scale analytics. Ingestion may occur through scheduled batch files, change-data capture, message streams or APIs.

An API is a controlled interface through which systems request or exchange data. APIs support real-time quotation, identity checks and model scoring, but require authentication, rate limits, versioning, monitoring and failure handling. Batch processes may be simpler and more economical for daily portfolio analysis.

6.2 Warehouse, lake and lakehouse

A data warehouse stores structured, curated data organised for consistent reporting and analysis. It typically enforces schemas and business definitions. It is suitable for trusted premium, exposure and claims marts.

A data lake stores large volumes of structured, semi-structured and unstructured data in relatively raw form. It supports exploration, data science and retention of source detail, but without governance it can become a “data swamp.”

A lakehouse seeks to combine lake-scale storage and flexible data types with warehouse-like table management, transaction control and governance. The choice is architectural rather than ideological. Workload, latency, skills, regulatory requirements, cost and control objectives should determine the design.

6.3 Cloud platforms and analytical products

Cloud platforms provide scalable compute, managed storage, identity services, orchestration and machine-learning capabilities. Their benefits include elasticity, faster experimentation and managed resilience. Risks include uncontrolled consumption costs, misconfigured access, concentration risk, cross-border transfer issues and dependence on a provider.

The analytical output should be treated as a governed product. A feature store, for example, may provide approved variables with definitions, owners, freshness information and point-in-time correctness. A model-serving API should record model version, input schema, response, timestamp and reason codes where appropriate. Monitoring should detect data-quality failures, concept drift, latency degradation and changes in outcome distributions.

7. Governance, ethics and regulation

Insurance data governance is both a control function and a source of analytical value. Core practices include:

• data ownership and stewardship;
• classification of confidential and personal data;
• purpose limitation and retention schedules;
• consent and lawful processing where applicable;
• encryption in transit and at rest;
• role-based access and segregation of duties;
• audit logs and reproducible transformations;
• incident response and breach management;
• model documentation and independent validation.

Privacy-by-design principles encourage collecting only what is necessary, separating identifiers from analytical attributes, restricting access and deleting or anonymising data when retention is no longer justified. Sensitive data should not be included merely because predictive performance improves.

Regulatory obligations vary by jurisdiction, product and use case. Nevertheless, common expectations include fair treatment, explainable or reviewable decisions, accurate records, secure processing, governance of outsourcing and evidence that models are fit for purpose. Analytics teams should involve legal, compliance, actuarial, risk, information security and business stakeholders early rather than treating governance as a final approval step.

8. Applied case: point-in-time motor renewal analytics

This is an illustrative case, not a report of a specific insurer.

Background

A motor insurer wishes to predict whether a private motor policy will renew within 30 days after expiry. It holds policy transactions, payment records, claims, customer contacts, broker information and optional telematics data.

Problem

An initial model shows excellent validation performance but performs poorly after deployment. Investigation finds that the training table includes a “renewal completed” status updated after expiry. It also uses the customer’s annual claim count calculated from the current claims database, which includes claims reported after the historical renewal decision.

Analysis

The team defines the prediction point as 30 days before expiry. Every feature must therefore be reconstructed as known at that date. Policy endorsements are selected using effective and transaction dates. Claims are included only if they were recorded by the as-of date. Payment behaviour is measured using historical transactions, not the present account balance. Telematics summaries are restricted to trips available before the cut-off, and missing device data are distinguished from zero activity.

A chronological evaluation is used: policies expiring in earlier months form training data, a later month is validation, and the final months are held out for testing. Performance is examined overall and by channel, tenure, geography and telematics availability. The team also checks whether broker channel is acting as a proxy for customer groups and whether contact frequency reflects prior retention interventions.

Approach and solution

The insurer creates a point-in-time feature pipeline with:

1. immutable source events;
2. effective and observation timestamps;
3. approved feature definitions;
4. automated leakage tests;
5. data-quality checks for duplicates, missingness and invalid dates;
6. model monitoring after deployment;
7. human review for borderline or vulnerable-customer cases.

Rather than automatically targeting every predicted non-renewer, the business uses the model to prioritise service outreach. Customers are offered clear renewal information and an appropriate communication channel. The insurer excludes variables that are unnecessary, legally restricted or difficult to explain.

Outcome

The revised evaluation produces lower but more realistic performance. Operational teams accept the result because it reflects the information available at decision time. Monitoring identifies deterioration when a new payment platform changes the meaning of a transaction code. The feature owner updates the data dictionary and retrains only after the change has been assessed.

Learning points

The case demonstrates that point-in-time correctness is not merely a technical detail. It affects the validity of performance estimates, fairness of customer treatment, auditability and managerial confidence. A modest model supported by trustworthy data is more valuable than a highly accurate model whose inputs could not have been known when decisions were made.

9. Best-practice checklist

Before approving an insurance dataset or feature for predictive use, ask:

1. What business question and decision point does it support?
2. What is the unit of observation and exposure denominator?
3. Are definitions, units, currency and accounting basis documented?
4. Can the value be reconstructed as known at the decision time?
5. Is the source authoritative, licensed and appropriately governed?
6. What are the missingness mechanism and remediation strategy?
7. Could the variable encode protected characteristics or historical decisions?
8. Are training, validation and test populations temporally and operationally realistic?
9. Are lineage, versioning, access and retention controlled?
10. How will quality, drift, fairness and business outcomes be monitored after deployment?

Chapter Summary

Insurance predictive analytics rests on a chain linking real-world risks and events to governed data and decisions. Policy, exposure, premium, claims, customer, intermediary, telematics, geospatial and external data each describe different aspects of the insurance system and must not be treated as interchangeable. Exposure supplies the denominator for rates; premium requires careful distinction between written and earned amounts; claims require separation of occurrence, reporting, payment, reserve and ultimate development.

Reliable measurement depends on aligned definitions, units, time periods and accounting bases. Data quality includes accuracy, completeness, validity, consistency, timeliness, uniqueness and integrity. Provenance and lineage make data interpretable and reproducible. Missingness may itself carry information, while leakage can produce falsely impressive models. Bias may arise from sampling, measurement, labels, historical decisions and proxy variables. Temporal structure requires as-of logic and chronological evaluation.

Warehouses, lakes, lakehouses, APIs and cloud platforms provide different capabilities within a broader architecture. Their value depends on governance, metadata, security, privacy, quality controls and model monitoring. The central professional principle is simple: an analytical result is only as credible as the definitions, information timing, lineage and governance supporting it.

Key Terms

Exposure: The measured opportunity for an insured event to occur, such as vehicle-years or payroll.

Written premium: Premium associated with policies issued or renewed during a period.

Earned premium: The portion of premium relating to insurance protection already provided.

Incurred loss: A defined measure commonly combining paid loss and outstanding case reserve.

Loss ratio: Incurred loss divided by earned premium under a stated basis.

Claim frequency: Number of claims divided by the relevant exposure.

Severity: Aggregate claim cost divided by claim count.

Provenance: Information about the origin, context and collection of data.

Data lineage: The record of data movement and transformations from source to output.

Data quality: The fitness of data for a specified purpose across dimensions such as accuracy and completeness.

Missingness: The absence of an expected value, with potentially informative causes.

Data leakage: Use of information unavailable at the intended prediction time or caused by the outcome.

As-of date: The latest date up to which information may be used for a decision.

Temporal structure: The ordering and timing relationships among insurance observations and events.

Data warehouse: Curated, structured analytical storage designed for consistent reporting.

Data lake: Flexible storage for large volumes of raw or varied data.

API: A controlled software interface for exchanging data or services.

Feature store: A governed system for managing reusable analytical variables and their definitions.

Telematics: Sensor or communication data describing movement, usage or operating behaviour.

Entity resolution: The process of determining which records refer to the same real-world entity.

Review Questions

Short Answer Questions

1. Why is exposure necessary when interpreting claim counts?
2. Distinguish written premium from earned premium.
3. What is the difference between provenance and lineage?
4. Give two examples of data leakage in insurance analytics.
5. Why can missing telematics data be informative?

Long Answer Questions

1. Explain the principal insurance data domains and their uses in underwriting, pricing and claims.
2. Discuss the dimensions of data quality and propose a quality-control framework for claims data.
3. Compare data warehouses, data lakes and lakehouses for an insurer’s analytics programme.
4. Explain why temporal structure and point-in-time correctness are essential in predictive modelling.
5. Evaluate the ethical and governance issues associated with geospatial, telematics and external data.

Analytical / Application Questions

1. A portfolio has 12,000 vehicle-years, 360 claims and incurred losses of 1,080,000 monetary units. Calculate claim frequency, average severity and implied loss cost per vehicle-year. State the assumptions required for interpretation.
2. Design a point-in-time data set for predicting motor-policy renewal 30 days before expiry. Specify the permitted dates, at least five features, two leakage risks and two validation controls.
3. An insurer observes that claim reporting is much less complete in rural areas because of connectivity limitations. Analyse how this could affect model bias, missingness treatment, performance measurement and governance.

Multiple Choice Questions with Answers

Question 1

Which measure is the denominator in a basic claim-frequency calculation?

A. Earned premium

B. Exposure

C. Claim severity

D. Case reserve

Correct Answer: B. Exposure

Question 2

A 12-month policy costs 1,200 monetary units. Under a simple pro-rata assumption, how much premium is earned after four months?

A. 100

B. 300

C. 400

D. 800

Correct Answer: C. 400

Question 3

Which date represents when a policy change applies to the risk?

A. Transaction date

B. Effective date

C. Payment date

D. Extraction date

Correct Answer: B. Effective date

Question 4

Which situation is an example of data leakage?

A. Using vehicle age at quotation

B. Using historical payment count before quotation

C. Using final settlement status to predict claim cost at notification

D. Using policy type recorded at inception

Correct Answer: C. Using final settlement status to predict claim cost at notification

Question 5

What is the main purpose of data lineage?

A. To increase the number of variables

B. To record how data move and are transformed

C. To replace data quality checks

D. To eliminate the need for access controls

Correct Answer: B. To record how data move and are transformed

Question 6

A portfolio has 100 claims and 2,500 exposure-years. What is its claim frequency?

A. 0.004 claims per exposure-year

B. 0.025 claims per exposure-year

C. 0.04 claims per exposure-year

D. 25 claims per exposure-year

Correct Answer: C. 0.04 claims per exposure-year

Question 7

Which architecture component is generally designed for curated, structured analytical reporting?

A. Data warehouse

B. Telematics device

C. API gateway only

D. Raw sensor buffer

Correct Answer: A. Data warehouse

Question 8

Why is chronological validation often preferable for prospective insurance prediction?

A. It guarantees perfect accuracy

B. It prevents all missing values

C. It better represents learning from past data to predict future outcomes

D. It removes the need for a test set

Correct Answer: C. It better represents learning from past data to predict future outcomes

Question 9

Which statement about missing data is most appropriate?

A. All missing values should be replaced with zero

B. Missingness can sometimes contain information about the data-generating process

C. Missingness is always completely random

D. Missing values should always be deleted

Correct Answer: B. Missingness can sometimes contain information about the data-generating process

Question 10

What does a lakehouse seek to combine?

A. Claims handling and underwriting functions

B. Lake-scale flexibility with warehouse-like management and governance

C. Only real-time APIs and spreadsheets

D. Premium accounting with regulatory reporting exclusively

Correct Answer: B. Lake-scale flexibility with warehouse-like management and governance