Insurance Data, Analytics and Managerial Decision-Making

Introduction

General insurance is increasingly managed as an information business. Insurers collect policy, exposure, premium, claims, payment, customer, broker, repair, location, weather, economic and operational data. The competitive value of this information does not arise merely from possessing large databases. It arises when data are governed, transformed into reliable measures, analysed with appropriate models, and connected to decisions concerning price, acceptance, claims handling, fraud, retention, capital and strategy.

Insurance analytics must therefore be understood as a management capability rather than as a collection of statistical techniques. A technically accurate predictive model can still produce poor business outcomes if its data are biased, its target is incorrectly defined, its predictions are not calibrated, or its recommendations cannot be implemented. Conversely, a simple dashboard may create substantial value when it exposes deterioration in claims frequency early enough for management to act.

This chapter examines an integrated analytical framework for non-life insurance. It covers data governance, portfolio segmentation, descriptive and predictive analytics, geospatial analysis, optimisation, model validation, dashboards and managerial decision-making. Particular attention is given to the relationship between analytical evidence and professional judgement, regulatory accountability, fairness and operational control.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Explain the data lifecycle and governance requirements of a general insurer.
2. Segment insurance portfolios for pricing, underwriting, claims and retention decisions.
3. Distinguish descriptive, diagnostic, predictive and prescriptive analytics.
4. Interpret important insurance measures and predictive-model outputs.
5. Apply geospatial analysis to catastrophe, accumulation and distribution problems.
6. Explain optimisation, model validation and monitoring requirements.
7. Design decision-useful executive dashboards.
8. Evaluate ethical, regulatory and managerial risks in insurance analytics.

1. The Insurance Data Value Chain

An insurance data value chain links business events to decisions:

Capture → Validate → Integrate → Analyse → Decide → Act → Monitor

Data are captured when a quotation is requested, a policy is issued, a payment is made, a loss is reported or a claim is settled. Validation checks whether fields are complete, plausible and consistent. Integration connects information across policy, claims, finance, customer and external systems. Analysis produces measures, patterns or forecasts. Decisions determine actions such as accepting a risk, changing a deductible, assigning a claim to special investigation, or contacting a policyholder. Monitoring then tests whether the decision generated the intended result.

The value chain is iterative. A claims model may reveal that a field originally treated as optional is essential for prediction. A new underwriting rule may alter the composition of future data, creating a feedback effect. Consequently, data and models require continuing governance rather than one-time development.

1.1 Main classes of insurance data

A fundamental distinction is between exposure, event and outcome. Exposure describes the time or units at risk; an event is a claim or other occurrence; an outcome is the eventual cost, settlement, cancellation or customer response. Confusing these concepts produces biased rates and misleading performance measures.

1.2 Data governance

Data governance assigns accountability for data definitions, quality, access, retention and use. A mature governance framework normally includes:

• Data ownership: a named business owner is accountable for meaning and fitness for purpose.
• Data stewardship: operational staff maintain definitions, quality rules and issue resolution.
• Metadata: each field has a definition, unit, source, permissible values and update frequency.
• Lineage: users can trace a dashboard measure back to source transactions.
• Quality controls: completeness, validity, consistency, timeliness, uniqueness and accuracy are assessed.
• Access control: data are available according to role and legitimate business need.
• Security and privacy: personal and sensitive information are protected throughout the lifecycle.
• Retention and deletion: records are retained for legal, actuarial and operational reasons, then securely disposed of.
• Change management: alterations to systems, definitions or models are documented and approved.

For a data-quality dimension, a simple rate can be calculated as:

The measure is meaningful only when the denominator and definition of “required” are stable. High completeness does not prove accuracy: a field can be filled in systematically with an incorrect value.

2. Portfolio Segmentation and Insurance Performance Measures

Segmentation divides a heterogeneous portfolio into groups with sufficiently similar risk, behaviour or management needs. Segments may be based on product, geography, customer tenure, distribution channel, vehicle class, construction type, claims history or combinations of variables. Segmentation should be legally permissible, actuarially defensible, operationally usable and sufficiently stable to support decisions.

2.1 Frequency, severity and pure premium

Let:

• N = number of claims in a period;
• E = exposure units, such as vehicle-years or property-years;
• C_i = cost of claim i;
• S = aggregate claims cost.

Claim frequency is:

Average claim severity, conditional on a claim occurring, is:

The expected claims cost per exposure unit, often called pure premium or loss cost, is:

The decomposition assumes that the same period, claims definition and exposure basis are used in all terms. It is especially useful because management can ask whether deterioration arises from more claims, larger claims, or both.

Suppose a motor segment has 10,000 vehicle-years, 800 claims and total incurred claims of $1,600,000. Frequency is 0.08 claims per vehicle-year, severity is $2,000, and pure premium is $160 per vehicle-year. If claims inflation raises severity by 10% while frequency remains constant, the projected pure premium is $176, before considering expenses, reinsurance, profit and changes in exposure.

2.2 Loss ratio and combined ratio

The loss ratio compares claims cost with earned premium:

The expense ratio is:

The combined ratio is:

These ratios are conventionally expressed as percentages. A combined ratio below 100% indicates an underwriting profit before investment income and certain other items, subject to the accounting definitions used. Incurred claims may include case reserves, incurred-but-not-reported estimates and adjustments; therefore, the measurement basis must be specified.

2.3 Segmentation risks

Excessive segmentation can produce unstable estimates, operational complexity and unfairly granular pricing. Sparse segments may show extreme results because of random variation. A useful practice is to combine statistical evidence with credibility, business knowledge and minimum-volume rules. Segments should be reviewed for proxy discrimination: a variable may appear neutral while closely representing a protected characteristic or socioeconomic disadvantage.

3. From Reporting to Prediction

Analytics can be organised into four levels:

The levels are connected but not interchangeable. A predictive model does not establish causation. A variable associated with claims may be useful for prediction but unsuitable as a management lever. Prescriptive analysis requires explicit objectives, constraints and consequences.

3.1 Predictive modelling concepts

A supervised model estimates an outcome from explanatory variables. For a binary outcome such as claim occurrence, a logistic model may be written as:

Here, p(x) is the estimated probability of the event, x_j are predictors, β_j are coefficients, β₀ is the intercept, and exp is the exponential function. The model assumes that the log-odds of the event are linear in the predictors unless transformations or interactions are added. The estimated probability is not automatically a price; it must be combined with severity, expenses, capital, reinsurance and commercial considerations.

For a claim-cost model, expected cost can be conceptualised as:

The two-part structure is useful when many observations have zero cost and positive claims have a skewed distribution. It requires careful treatment of exposure, truncation, large losses, inflation and claims development.

3.2 Training, testing and leakage

Data are commonly divided into training and test samples. The training sample estimates model parameters; the test sample evaluates performance on observations not used for fitting. For time-dependent insurance data, a chronological split is often more realistic than a random split because future information must not influence past decisions.

Data leakage occurs when a predictor contains information that would not have been available at the decision date. Examples include a final settlement amount in a model intended for first-notice-of-loss triage, or a cancellation code entered after a retention decision. Leakage can produce impressive but unusable results.

3.3 Performance measures

For classification, a confusion matrix records true positives, false positives, true negatives and false negatives. Common measures include:

Precision is important when investigation is expensive; recall is important when missing a serious fraud case or safety risk is costly. Threshold selection should reflect business costs, not merely maximise a generic score.

For continuous predictions, mean absolute error is:

where y_i is the observed value, ŷ_i is the prediction and n is the number of observations. MAE has the same units as the outcome and is less sensitive to extreme errors than squared-error measures.

Calibration is distinct from discrimination. A model is calibrated if, among cases predicted at approximately 20%, the event occurs about 20% of the time. A model may rank risks well while systematically overestimating or underestimating probabilities.

4. Dashboards and Management Information

A dashboard should answer a defined management question, not display every available metric. Effective dashboards identify the population, period, data refresh date, denominator, target, variance and responsible owner.

A useful executive dashboard might include:

• written and earned premium;
• claims frequency, severity and loss ratio;
• combined ratio and expense movement;
• quote-to-bind conversion;
• renewal and cancellation rates;
• open claims, ageing and reserve adequacy indicators;
• suspected-fraud referrals and confirmed outcomes;
• catastrophe exposure by location;
• complaints and service-level performance;
• model drift and data-quality exceptions.

A rate of change can be stated as:

The comparison period must be appropriate. Seasonality, portfolio mix, earned-versus-written premium timing and claims development can make a simple month-on-month comparison misleading. Dashboards should permit drill-down from enterprise results to product, region, channel and operational cause while preserving consistent definitions.

5. Geospatial Analytics

Location is central to property, motor, agriculture, marine and catastrophe insurance. Geospatial analysis can identify hazard concentration, accumulation, access problems, repair networks and potential fraud patterns.

Important methods include:

• mapping exposure and claims by coordinates or geographic areas;
• overlaying locations with flood, wildfire, storm or seismic hazard layers;
• calculating distance to roads, hospitals, repairers or fire stations;
• identifying accumulation within catastrophe zones or supply-chain corridors;
• detecting unusual clusters of claims, addresses or repair providers.

Geospatial results depend on positional accuracy, geocoding quality, boundary definitions and temporal alignment. A claim location may represent the incident site, policy address or repair location; these are not interchangeable. Aggregation at very small geographic units can create privacy risks and unstable estimates. Maps should therefore be accompanied by counts, exposure denominators and uncertainty information.

6. Optimisation and Decision Design

Prediction estimates what may happen; optimisation selects an action under objectives and constraints. An insurer may seek to maximise expected contribution while respecting capacity, regulatory, service and risk constraints.

A generic decision problem can be expressed as:

Here, a_i indicates whether opportunity i is selected, v_i is its expected value, r_i is its consumption of a constrained resource, and R is the available resource. This is a simplified binary optimisation model. Real decisions may require continuous quantities, multiple constraints, uncertainty, fairness restrictions and scenario analysis.

Applications include selecting risks within underwriting capacity, allocating claims staff, scheduling inspections, choosing fraud-investigation thresholds and optimising reinsurance structures. The objective function must include relevant costs. For example, maximising premium alone can encourage undesirable growth; a better objective may consider expected claims, expenses, capital usage, customer value and compliance risk.

7. Model Validation and Lifecycle Control

Model validation is an independent challenge to whether a model is conceptually sound, correctly implemented, fit for purpose and appropriately governed. It is not limited to calculating an accuracy score.

A robust validation framework examines:

1. Conceptual soundness: Is the target defined correctly? Are causal assumptions, exposure measures and exclusions defensible?
2. Data quality: Are source data complete, representative, stable and available at the point of decision?
3. Implementation accuracy: Does production code reproduce development results? Are transformations and units correct?
4. Performance: How well does the model discriminate, calibrate and predict across important segments?
5. Stability: Does performance persist across time, geography, products and economic conditions?
6. Fairness and legality: Do outcomes create unjustified adverse effects or use prohibited information?
7. Use and controls: Are overrides, thresholds, human review and escalation documented?
8. Monitoring: Are drift, calibration, data quality and outcome changes tracked after deployment?

Let the population event rate be p and the model predicted rate be q. A simple calibration difference is q − p, although practical calibration should be assessed across risk bands and time periods. Population stability measures and characteristic-level monitoring can identify changing input distributions, but alerts require business interpretation rather than automatic model replacement.

Model risk includes the possibility that a model is wrong, misused or applied outside its design conditions. Independent validation, version control, approval committees, documented limitations and controlled retirement are essential. Human judgement should not mean undocumented discretion; overrides should be recorded and analysed for consistency and outcomes.

8. Fraud, Claims and Retention Applications

Fraud analytics combines rules, anomaly detection, network analysis, predictive scores and investigator judgement. A suspicious indicator is not proof of fraud. Excessive automated suspicion can delay legitimate claims, harm vulnerable customers and create reputational or regulatory exposure. Models should support proportionate investigation, with clear evidence standards and appeal mechanisms.

In claims, analytics can assist first-notice-of-loss triage, reserve review, severity forecasting, repairer assignment and settlement leakage detection. Automation is most appropriate for routine, well-understood decisions with low harm from error. Complex, disputed, vulnerable-customer or high-severity claims require skilled human oversight.

Retention models estimate the probability of renewal or cancellation. A commercially useful intervention model should go beyond predicting who will lapse. It should estimate whether a particular action—such as service recovery, a coverage review or an offer—changes the outcome enough to justify its cost. This is an uplift or treatment-effect perspective, not merely a propensity score.

9. Ethical, Regulatory and Professional Issues

Insurance analytics affects access to protection and the price paid for it. Ethical governance therefore requires more than technical accuracy.

Key principles include:

• Purpose limitation: collect and use data for specified legitimate purposes.
• Proportionality: use no more information or intervention than necessary.
• Transparency: explain material factors and decisions in understandable language.
• Accountability: assign responsibility to people and committees, not to “the algorithm.”
• Fairness: test for unjustified disparate outcomes and proxy effects.
• Privacy and security: minimise exposure of personal information and protect it against misuse.
• Human review: provide meaningful oversight and routes for correction or appeal.
• Professional competence: ensure that analysts and managers understand limitations.

Regulatory requirements differ by jurisdiction, but insurers commonly face obligations relating to privacy, discrimination, consumer protection, record keeping, outsourcing, cybersecurity, actuarial standards and governance of automated decisions. Compliance should be designed into the analytical lifecycle through impact assessments, documentation and auditability.

10. Contemporary Developments

Cloud platforms, application programming interfaces, telematics, Internet-of-Things sensors, satellite imagery, graph databases and generative artificial intelligence are expanding analytical possibilities. However, technology does not remove foundational requirements. More frequent data can increase surveillance concerns; alternative data can reproduce social inequality; and generative systems may produce unsupported explanations or confidential-data leakage.

Machine-learning systems can capture nonlinear relationships and interactions, but complex models may be difficult to explain and vulnerable to drift. Explainability should be matched to the decision: a technical feature-importance chart may not be an adequate customer explanation. Synthetic data can support development and testing, but it does not automatically preserve statistical validity or privacy.

A prudent adoption sequence is: define the business problem, assess legal and ethical permissibility, establish data quality, develop a baseline model, test incremental value, validate independently, pilot under controls, monitor outcomes and retain a safe fallback process.

Applied Case: Meridian General Insurance

This is an illustrative case, constructed for educational purposes.

Background

Meridian General Insurance writes private motor business through brokers and digital channels. Its annual loss ratio has risen from 67% to 78% over two years, while renewal retention has fallen from 84% to 79%. Management initially attributes the deterioration to careless driving, but the available evidence is fragmented across policy, claims, repair and customer systems.

Problem

The board requests a plan to improve profitability without indiscriminate premium increases or deterioration in claims service. Management must identify the causes of deterioration, improve risk selection, control claims leakage and retain profitable customers.

Analysis

The analytics team first establishes common definitions and reconciles earned premium, exposure and incurred claims. It discovers that a change in claims coding caused some glass claims to be classified inconsistently and that written premium had been compared with incurred claims in an early dashboard. After correction, the loss ratio is 75%, not 78%.

Segmentation shows three relevant patterns. Urban electric vehicles have low frequency but high repair severity because parts availability and calibration costs are high. A particular channel has average frequency but unusually high bodily-injury severity. A group of new customers has acceptable loss performance but high cancellation after the first renewal quotation.

A claims model is developed using only information available at first notification. The model is evaluated chronologically, calibrated by decile and reviewed for differential outcomes. It is used to prioritise complex claims for experienced adjusters, not to deny claims. A separate repair-network analysis identifies concentration around two suppliers and reveals long cycle times in one metropolitan area.

A retention analysis tests interventions rather than simply ranking cancellation risk. Customers with high cancellation probability but low expected loss cost receive a coverage review and service contact. High-risk, low-value blanket discounts are avoided.

Approach and solution

Meridian introduces a governed data dictionary, a monthly reconciliation control and dashboard ownership at product and claims level. Underwriting reviews electric-vehicle repair assumptions and updates severity factors after analysing parts and calibration costs. Claims leadership pilots model-assisted triage with human review and measures both savings and customer complaints. Procurement expands the repair network and negotiates service-level commitments.

The company also establishes a model committee that approves purposes, thresholds, validation evidence, monitoring metrics and retirement criteria. Each automated recommendation has an audit trail, and customers retain access to normal claims procedures.

Outcome

After two renewal cycles, the illustrative pilot reports improved repair cycle time, more consistent complex-claim allocation and lower avoidable retention expenditure. The overall loss ratio improves modestly, but management concludes that data correction and operational changes contributed as much as predictive modelling. The board therefore treats analytics as a cross-functional control system rather than a standalone technology project.

Learning points

1. Correct definitions and reconciliation precede sophisticated modelling.
2. Risk selection, claims management and retention require different targets.
3. Prediction should be separated from adverse action.
4. Intervention effectiveness matters more than propensity ranking alone.
5. Model governance must include monitoring, human accountability and customer impact.

Challenges and Limitations

Insurance data are affected by reporting delays, reserve development, changing policy wording, inflation, legal decisions, catastrophe events, operational coding and portfolio mix. Historical data may reflect previous underwriting rules and therefore not represent future business. Rare events produce limited observations but potentially large losses. Correlation can be unstable under climate change, economic stress or technological change.

Analytics can also create false precision. A probability estimate with several decimal places does not imply equivalent certainty. Dashboards may encourage target chasing, while optimisation may shift risk to unmeasured dimensions. Managers should use scenario analysis, sensitivity testing, expert review and stress testing alongside point forecasts.

Best-Practice Framework

A professionally managed insurance analytics programme should:

1. Start with a clearly defined decision and accountable owner.
2. Specify the observation date, outcome window and exposure basis.
3. Maintain documented data definitions, lineage and quality controls.
4. Establish a transparent baseline before adopting complex methods.
5. Separate development, validation and production responsibilities.
6. Test performance over time and across material segments.
7. Measure calibration, business value, fairness and customer impact.
8. Keep human review for high-impact or uncertain decisions.
9. Monitor drift, overrides, complaints, outcomes and security incidents.
10. Retire or revise models when their assumptions no longer hold.

Chapter Summary

Insurance analytics converts governed data into evidence for underwriting, pricing, claims, fraud, retention and executive decisions. The process begins with reliable definitions, exposure measurement, lineage, quality and access controls. Portfolio segmentation helps reveal variation in frequency, severity, pure premium, loss ratio and customer behaviour, but excessive or proxy-based segmentation creates instability and fairness risks.

Descriptive, diagnostic, predictive and prescriptive analytics answer different questions. Predictive models require appropriate targets, leakage controls, temporal validation, calibration and monitoring. Geospatial analysis supports catastrophe accumulation and operational planning but depends on accurate location definitions and privacy safeguards. Optimisation formalises choices under constraints, while model validation tests conceptual soundness, implementation, performance, stability, fairness and ongoing use.

Dashboards are effective when they connect clearly defined measures to owners and actions. Advanced technologies can improve capability, but they do not replace professional judgement, governance, transparency or accountability. The central managerial lesson is that analytical value depends on the complete decision lifecycle: trustworthy data, suitable models, disciplined implementation, measured outcomes and responsible human oversight.

Key Terms

Data governance: The system of accountability, standards and controls governing data.

Data lineage: The traceable path from an original data source to a reported or modelled result.

Exposure: The units or time during which insurance risk is subject to occurrence.

Claim frequency: Claims divided by exposure.

Claim severity: Claims cost divided by the number of claims.

Pure premium: Expected claims cost per exposure unit.

Loss ratio: Incurred claims divided by earned premium.

Combined ratio: Loss ratio plus expense ratio.

Segmentation: Division of a portfolio into relatively homogeneous groups.

Predictive model: A model estimating an unknown or future outcome.

Data leakage: Use of information unavailable at the intended decision time.

Calibration: Agreement between predicted probabilities and observed event rates.

Precision: The proportion of positive predictions that are true positives.

Recall: The proportion of actual positives detected by a model.

Geospatial analytics: Analysis using location and spatial relationships.

Optimisation: Selection of actions that best satisfy an objective and constraints.

Model validation: Independent assessment of a model’s soundness, performance and use.

Model drift: Change in data, relationships or performance after deployment.

Uplift modelling: Estimation of how an intervention changes an individual outcome.

Data leakage: Inclusion of future or otherwise unavailable information in model development.

Review Questions

Short Answer Questions

1. Distinguish exposure, claim event and claim outcome data.
2. Define pure premium and explain its frequency–severity decomposition.
3. What is data leakage, and why does it invalidate predictive evaluation?
4. Distinguish calibration from discrimination in a predictive model.
5. State three responsibilities of data governance.

Long Answer Questions

1. Explain how an insurer can design a complete data lifecycle from capture to monitoring.
2. Discuss the benefits and risks of portfolio segmentation in general insurance.
3. Explain the role of dashboards in managerial decision-making and identify common interpretation errors.
4. Compare descriptive, predictive and prescriptive analytics using insurance examples.
5. Discuss the ethical and regulatory requirements for using automated claims or underwriting models.

Analytical / Application Questions

1. An insurer records 12,000 vehicle-years, 900 claims and $2,250,000 incurred claims. Calculate frequency, average severity and pure premium. Explain which management actions might follow if frequency is stable but severity has increased.
2. Design a validation plan for a fraud model used at first notification of loss. Specify the target, permissible predictors, validation split, performance measures, human controls and monitoring indicators.
3. A property portfolio shows a profitable average loss ratio but severe concentration of exposure in a coastal flood zone. Explain how geospatial analysis, scenario testing and optimisation could support underwriting and reinsurance decisions.

Multiple Choice Questions with Answers

Question 1

Which measure equals incurred claims divided by earned premium?

A. Expense ratio

B. Loss ratio

C. Retention ratio

D. Conversion ratio

Correct Answer: B. Loss ratio

Question 2

An insurer has 5,000 exposure units and 250 claims. What is the claim frequency?

A. 0.02 claims per exposure unit

B. 0.05 claims per exposure unit

C. 20 claims per exposure unit

D. 250 claims per exposure unit

Correct Answer: B. 0.05 claims per exposure unit

Question 3

Which situation is an example of data leakage?

A. Using vehicle age known at quotation

B. Using weather conditions recorded before the loss

C. Using final settlement cost in a first-notice-of-loss model

D. Using policy tenure at renewal

Correct Answer: C. Using final settlement cost in a first-notice-of-loss model

Question 4

What does calibration assess?

A. Whether predictions rank cases correctly only

B. Whether predicted probabilities agree with observed event rates

C. Whether a model has the lowest possible number of variables

D. Whether data are stored in a cloud platform

Correct Answer: B. Whether predicted probabilities agree with observed event rates

Question 5

Which analytics level asks what action should be taken under constraints?

A. Descriptive analytics

B. Diagnostic analytics

C. Predictive analytics

D. Prescriptive analytics

Correct Answer: D. Prescriptive analytics

Question 6

If claims cost is $1,200,000 and exposure is 8,000 units, what is the pure premium?

A. $15 per unit

B. $150 per unit

C. $960 per unit

D. $1,500 per unit

Correct Answer: B. $150 per unit

Question 7

Which is a principal purpose of geospatial analysis in property insurance?

A. Eliminating the need for exposure data

B. Identifying hazard and accumulation concentration

C. Guaranteeing that every claim is fraudulent

D. Replacing all underwriting judgement

Correct Answer: B. Identifying hazard and accumulation concentration

Question 8

Which statement about model validation is most accurate?

A. It is limited to calculating accuracy on training data

B. It is performed only after a model is retired

C. It examines conceptual soundness, implementation, performance and use

D. It guarantees that future predictions are correct

Correct Answer: C. It examines conceptual soundness, implementation, performance and use

Question 9

In a fraud model, precision is the proportion of:

A. Actual fraud cases detected

B. Positive predictions that are true positives

C. All claims that are correctly classified

D. False positives among all negatives

Correct Answer: B. Positive predictions that are true positives

Question 10

Why should a retention model consider intervention effects rather than only cancellation probability?

A. Because every likely-to-cancel customer should receive a discount

B. Because prediction alone does not show whether an action changes the outcome

C. Because cancellation probability cannot be estimated statistically

D. Because retention decisions never involve costs

Correct Answer: B. Because prediction alone does not show whether an action changes the outcome