Feature Engineering and Data Preparation

Introduction

Predictive models do not learn directly from business concepts such as “poor risk quality,” “fraud propensity,” or “customer persistency.” They learn from measurable representations of those concepts. Feature engineering is the disciplined process of converting raw observations into variables that contain useful, timely, and defensible predictive information. Data preparation ensures that these variables are complete, correctly formatted, comparable, and available under the same conditions that will exist when predictions are made.

In insurance, this work is especially important because data are heterogeneous and operationally generated. A policy record may contain dates, coverage limits, premium, distribution channel, and location. A claims record may contain transaction histories, adjuster notes, payment development, and litigation indicators. Telematics and digital behaviour produce high-frequency time series, while text and geospatial sources add further complexity. Poor preparation can create leakage, biased decisions, unstable models, and regulatory exposure even when the selected algorithm is technically sophisticated.

The central principle is temporal and operational realism: a feature must be computed only from information that would have been available at the prediction date, using a reproducible procedure.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Distinguish raw variables, derived features, labels, and leakage.
2. Construct predictors from policy, claims, behavioural, text, time-series, and geospatial data.
3. Select suitable encoding, scaling, transformation, imputation, and outlier-treatment methods.
4. Address class imbalance without contaminating validation or test data.
5. Design reproducible train–validation–test splits for insurance data.
6. Evaluate feature quality, stability, fairness, interpretability, and operational usefulness.

1. The Feature Engineering Problem

A feature is a measurable input to a predictive model. Let the observation for policyholder i be represented by a feature vector x_i = (x_i1, x_i2, …, x_ip), and let yᵢ denote the target, such as claim occurrence, fraud, lapse, or loss amount. A feature-engineering pipeline applies transformations to raw data rᵢ:

Here, T is the transformation pipeline and θ contains parameters learned from training data, such as means, medians, category dictionaries, or vocabulary lists. The parameters must be estimated using training data only. Applying a transformation learned from the complete dataset can transmit information from validation or test observations into model development.

A useful feature should satisfy five tests:

• Availability: it exists at the time of prediction.
• Validity: its definition and units are correct.
• Predictive relevance: it is plausibly related to the target.
• Stability: its meaning and distribution remain sufficiently stable.
• Governance: its use is lawful, explainable, and fair.

Feature engineering differs from variable selection. Engineering creates or reformulates variables; selection chooses which variables to retain. Both must be guided by the prediction task, the decision process, and the information set available at deployment.

Targets, observation windows, and prediction horizons

A well-designed insurance dataset specifies three times:

1. Reference date: when the prediction is made.
2. Observation window: the historical period from which features are calculated.
3. Outcome window: the future period in which the target is measured.

For example, for lapse prediction, features may use policy and payment history from the previous 180 days, while the target is whether the policy lapses during the next 60 days. Claims reported after the reference date must not be used in the predictors, even if they are present in the data warehouse today.

2. Data Sources and Feature Families

Policy and customer data

Common policy features include age of vehicle or building, coverage type, insured value, deductible, premium, tenure, payment frequency, prior cancellations, renewal count, channel, and product changes. Ratios often have clearer meaning than raw values. Examples include premium per unit of insured value and claim count per policy-year.

A rate requires an appropriate exposure denominator. If N claims occur over E policy-years, the observed claim frequency is:

The units are claims per policy-year. Using policy count instead of exposure can be misleading when policies have different durations.

Claims data

Claims features may include claim frequency, severity, time since last claim, open-claim count, reporting delay, payment-to-reserve ratio, repair duration, attorney involvement, and repeated claimant–provider relationships. Aggregations must be date-controlled. A claim count used for underwriting should include only claims known at underwriting time, not later development.

Severity is often right-skewed. A log transformation can reduce extreme influence:

where y is a non-negative monetary amount and z is the transformed amount. The inverse transformation is y = e^z - 1. This transformation is appropriate when zero values exist and a multiplicative rather than additive scale is plausible; it does not remove the need to investigate data errors.

Behavioural and telematics data

Digital interactions can yield login frequency, quotation abandonment, response delay, payment retries, mileage, harsh-braking frequency, night-driving proportion, and device or session patterns. High-frequency observations should be summarised using features such as mean, median, standard deviation, quantiles, trend, volatility, and event counts.

For a time series x₁, x₂, …, x_T, the mean is:

where T is the number of valid observations. Dispersion can be represented by the sample standard deviation:

The sample formula assumes observations are being treated as a sample and that T > 1. In operational settings, missingness and unequal observation intervals must be handled explicitly.

Trend may be estimated by fitting x_t = a + bt + ε_t, where b represents average change per time unit. A trend is meaningful only when time units, sampling frequency, and minimum observation requirements are defined.

Text data

Claims notes, call-centre transcripts, and adjuster narratives can be represented by word or character n-grams, term-frequency inverse-document-frequency (TF–IDF), embeddings, or supervised language-model features. TF–IDF gives greater weight to terms frequent in a document but uncommon in the corpus. Text processing requires controls for spelling, personally identifiable information, negation, timestamps, and copied boilerplate.

Text can leak the outcome. For example, a note containing “fraud confirmed” may be written after the investigation decision. Such text is invalid for an early fraud-screening model unless the use case explicitly concerns post-investigation prediction.

Geospatial data

Location may be represented by postcode, coordinates, territory, distance to a repair network, elevation, flood zone, crime rate, weather exposure, or proximity to road infrastructure. Coordinates can be transformed into distance. For two points with latitude and longitude, the Haversine distance is approximately:

Here, φ denotes latitude, λ longitude, angles are in radians, and R is Earth’s radius, approximately 6,371 kilometres. This formula assumes a spherical Earth and is suitable for many practical distances, but not for high-precision surveying.

Geospatial variables may act as proxies for protected characteristics or socioeconomic status. Their predictive value must therefore be considered alongside fairness and regulatory requirements.

3. Encoding Categorical Variables

Categorical variables may be nominal, ordinal, binary, or hierarchical. One-hot encoding creates an indicator for each category except a reference category when an intercept is used. If a channel has categories broker, direct, and partner, two indicators can represent it without perfect linear dependence in a linear model.

Ordinal encoding is appropriate only when order is meaningful and the spacing assumption is defensible. For example, low, medium, and high risk may be encoded as 1, 2, and 3, but this implies a structured relationship that may not hold.

High-cardinality variables such as broker ID, repairer ID, or postcode require care. Alternatives include:

• frequency or count encoding;
• carefully smoothed target encoding;
• hashing;
• learned embeddings; and
• grouping rare categories.

Target encoding must be calculated out-of-fold. For a category c, a smoothed estimate can be expressed as:

where n_c is the number of training observations in category c, μ_c is its category-specific target mean, μ is the global training mean, and m controls shrinkage. This formula is applicable to a training partition; validation and test rows must receive encodings computed without using their outcomes. For classification, smoothing should also consider the target definition and prevalence.

Unknown categories at deployment require a defined policy, such as an “other” category, global mean, or frequency-based fallback. The policy should be tested before production.

4. Scaling and Transformation

Scaling is important for distance-based methods, regularised regression, neural networks, and clustering. Standardisation uses:

where μ and σ are the training mean and standard deviation. The resulting variable is dimensionless, with training mean approximately zero and standard deviation approximately one. Standardisation is not generally required for tree-based models, although consistent preprocessing may still support pipeline simplicity.

Min–max scaling maps a training range to [0, 1]:

It is sensitive to future values outside the training range and to outliers. Robust scaling uses the median and interquartile range (IQR):

where IQR = Q₃ − Q₁. Logarithmic, square-root, and power transformations may improve symmetry or linearity. Transformations should be selected based on the model, distribution, interpretability, and business meaning—not merely because a variable is non-normal.

5. Missing Values and Data Quality

Missingness can arise from non-collection, system failure, inapplicability, customer choice, or delayed reporting. These mechanisms have different meanings. A missing driving score may indicate that telematics was not installed, not that the customer has an average score.

Common treatments include:

• median imputation for skewed continuous variables;
• mode or explicit “unknown” categories;
• constant values with a missingness indicator;
• model-based imputation; and
• time-aware forward filling where operationally justified.

For a variable x, an indicator can be defined as:

The imputed value and indicator should be fitted within the training pipeline. Imputation using future or test observations creates leakage. Missingness indicators may improve prediction but can also encode process changes, access differences, or discriminatory practices. Their stability should be monitored.

Data-quality checks should cover uniqueness, valid ranges, dates, referential integrity, duplicate records, currency, units, impossible sequences, and reconciliation with source systems. For example, a claim closure date preceding its notification date is a data-quality exception, not an outlier to be silently replaced.

6. Outliers and Extreme Values

An outlier is an observation unusual relative to a defined reference distribution; it is not automatically an error. A large property claim may be the most important genuine observation in a portfolio. Treatment options include correction, exclusion under documented rules, winsorisation, robust transformation, segmentation, or use of models less sensitive to extremes.

The IQR rule flags values below Q₁ − 1.5IQR or above Q₃ + 1.5IQR. This is a screening convention, not a universal deletion rule. Thresholds must be learned from training data where appropriate and reviewed by subject-matter experts. Monetary amounts should be treated consistently with currency, inflation, exposure, and claim development.

7. Class Imbalance and Rare Events

Fraud, catastrophic loss, and certain severe injuries are often rare. Accuracy can therefore be misleading. If only 1% of claims are fraudulent, a classifier predicting “not fraud” for every case has 99% accuracy but no practical value.

Useful measures include precision, recall, specificity, F-score, area under the precision–recall curve, calibration, and expected cost. For a binary classifier:

Here, TP, FP, and FN are true positives, false positives, and false negatives. The F1 score is:

These metrics are meaningful only with a clearly defined positive class and decision threshold.

Approaches include class-weighted learning, threshold adjustment, under-sampling, over-sampling, and synthetic methods such as SMOTE. Resampling must occur inside each training fold; applying it before splitting allows duplicated or synthetic information to cross partitions. The operational cost of investigation, customer friction, missed fraud, and regulatory remediation should guide the threshold.

8. Reproducible Train–Validation–Test Design

A random split is often inappropriate for insurance because records are time-dependent, clustered by customer, and exposed to changing underwriting or claims practices. A robust design usually includes:

• Training set: fit preprocessing and model parameters.
• Validation set: select features, tune hyperparameters, and choose thresholds.
• Test set: final unbiased performance estimate, used once or sparingly.

For temporal prediction, use an earlier period for training and later periods for validation and testing. For repeated customers, keep all records for a customer in one partition where possible. For policy portfolios, grouped splitting prevents the same claimant, vehicle, or household from appearing in both training and test sets.

A reproducible pipeline records data versions, extraction dates, feature definitions, random seeds, software versions, transformations, exclusions, and model artifacts. Reproducibility does not mean that every result is permanent; it means that the process and inputs can be reconstructed.

Validation design must mirror deployment. If a model will score new policies monthly, rolling-origin evaluation is more credible than a randomly shuffled split. Performance should be examined across time, product, geography, channel, and relevant customer groups.

9. A Practical Feature Pipeline

A defensible pipeline may follow this sequence:

1. Define the decision, reference date, target, observation window, and outcome window.
2. Freeze source data according to an as-of timestamp.
3. Audit quality, units, duplicates, identifiers, and missingness.
4. Join policy, claims, behavioural, text, and geospatial tables using controlled keys.
5. Create domain features and document each definition.
6. Split data using temporal and group constraints.
7. Fit imputers, encoders, scalers, vocabularies, and resampling methods on training data only.
8. Train candidate models and tune them using validation data.
9. Assess discrimination, calibration, stability, fairness, and business cost.
10. Lock the pipeline, evaluate once on the test set, and establish monitoring.

Feature documentation should include name, definition, source, unit, granularity, availability time, transformation, missing-value policy, owner, and known limitations.

Applied Case: Illustrative Motor Fraud Screening

Background

An insurer wants to prioritise motor claims for investigation within 24 hours of notification. The target is whether a claim is confirmed as fraudulent after investigation. This is an illustrative case, not a report of an actual insurer.

Problem

The data contain 420,000 historical claims, 3,100 confirmed fraud cases, policy details, claimant history, repairer information, notification timestamps, call-centre text, and location. Initial analysis reveals that fraud labels are delayed and that some notes contain post-investigation conclusions.

Analysis

The team defines the reference date as the first notification timestamp. The observation window is the 24 months preceding notification, and the outcome window is the investigation result. Features such as “prior claims in 24 months,” “reporting delay,” “number of policy amendments,” and “repairer’s historical referral rate” are allowed only when available at notification. Text created after referral to the investigation unit is excluded.

A random split is rejected because the same claimant and repairer appear repeatedly and fraud patterns change over time. Instead, the team uses the first 18 months for training, the next three months for validation, and the final three months for testing, while controlling leakage through claimant and claim identifiers.

Approach and solution

Numeric variables receive training-set median imputation and missingness indicators where operationally meaningful. Nominal variables use one-hot encoding for low-cardinality fields and out-of-fold smoothed encoding for repairer ID. Monetary variables are transformed using ln(1 + x). Text is represented by a vocabulary fitted only on training documents; post-notification notes are removed. Class weights are used rather than duplicating rare cases. The decision threshold is selected using the cost of investigation, false alarms, and missed fraud.

Performance is reported using precision–recall measures, calibration, and subgroup analysis. A high ranking score alone is not accepted: investigators need reason codes, and customers must not be automatically denied solely because a model assigns a high score.

Outcome

The resulting system produces a ranked queue rather than an automatic adverse decision. Monitoring tracks precision by month, changes in fraud prevalence, missingness, repairer representation, investigator override rates, and group-level error patterns. A model review is triggered when data definitions or investigation procedures change.

Learning points

The case illustrates that feature engineering is inseparable from label design, time control, resampling, explainability, and governance. Removing obvious leakage may reduce apparent validation performance while substantially improving credibility in production.

Governance, Ethics, and Regulation

Insurance features can influence pricing, underwriting, claims handling, and fraud investigation; these uses have material consequences. Governance should address lawful basis, purpose limitation, data minimisation, retention, access control, cybersecurity, and individual rights under applicable data-protection and insurance regulation.

Proxy discrimination is a central risk. A feature need not explicitly contain protected status to reproduce its effects. Geography, language, device type, occupation, and purchasing channel may correlate with protected or vulnerable characteristics. Fairness assessment should examine relevant error rates, calibration, access, and business impact, recognising that fairness criteria can conflict and must be interpreted in context.

Explainability requires more than listing feature importance. The organisation should be able to explain the data used, the feature definition, the decision purpose, uncertainty, appeal route, and human oversight. Feature stores and automated pipelines improve consistency but can amplify errors at scale. Every production feature needs ownership, lineage, quality thresholds, and a retirement process.

Contemporary Developments and Best Practice

Modern insurance analytics increasingly combines structured data with language models, embeddings, graph features, satellite imagery, connected-device data, and real-time event streams. These methods may improve representation but introduce challenges involving computational cost, drift, privacy, intellectual-property rights, and reproducibility. Large language models should not be treated as authoritative extractors without validation, especially when text contains sensitive or ambiguous information.

Recommended practices are to:

• begin with a simple, interpretable baseline;
• define an as-of date for every feature;
• separate feature computation from label computation;
• use pipelines that prevent train–test contamination;
• test temporal, geographic, and group stability;
• preserve units and business meaning;
• monitor missingness and category drift;
• document exclusions and transformations;
• evaluate calibration and decision costs, not only ranking;
• conduct fairness and privacy review before deployment; and
• design retraining and rollback procedures in advance.

Chapter Summary

Feature engineering converts insurance data into predictors that are available, valid, relevant, stable, and governable at decision time. Policy and claims records support exposures, frequencies, severities, recency, and ratios; behavioural and time-series data support aggregates, volatility, and trends; text requires temporal controls and privacy protection; geospatial features require both technical care and proxy-risk assessment. Encoding, scaling, transformation, imputation, and outlier treatment must be fitted within training data. Rare-event problems require suitable metrics, cost-sensitive decisions, and fold-contained resampling. Temporal and grouped train–validation–test designs are generally more credible than naive random splitting. Finally, deployment requires lineage, monitoring, calibration, explainability, fairness assessment, and human governance.

Key Terms

Feature: A measurable input supplied to a predictive model.

Feature engineering: Creating or transforming variables to represent predictive information.

Label: The target outcome that a model is trained to predict.

Data leakage: Use of information unavailable at prediction time or improperly shared across evaluation partitions.

Observation window: Historical period from which predictors are calculated.

Outcome window: Future period in which the target is measured.

Exposure: Time or units at risk, such as policy-years.

One-hot encoding: Representing categories with binary indicator variables.

Target encoding: Replacing a category with an outcome-related aggregate estimated from training data.

Imputation: Replacing missing values according to a defined rule.

Missingness indicator: A variable identifying whether an original value was missing.

Standardisation: Rescaling using a training mean and standard deviation.

Winsorisation: Capping extreme values at specified limits.

Class imbalance: Unequal prevalence of target classes.

Precision: Proportion of predicted positives that are actual positives.

Recall: Proportion of actual positives that are identified.

Calibration: Agreement between predicted probabilities and observed frequencies.

Data drift: Change in input distributions or relationships over time.

Feature lineage: Traceable record of a feature’s source and transformations.

As-of timestamp: The time at which information is considered available for prediction.

Review Questions

Short Answer Questions

1. Why must feature transformations be fitted on training data only?
2. Distinguish an observation window from an outcome window.
3. When is ordinal encoding appropriate?
4. Why can accuracy be misleading for fraud detection?
5. State two reasons for using a temporal rather than random data split.

Long Answer Questions

1. Explain how policy, claims, behavioural, text, and geospatial data can be converted into insurance predictors. Discuss availability and leakage risks.
2. Compare one-hot encoding, frequency encoding, and smoothed target encoding for high-cardinality insurance variables.
3. Discuss imputation and outlier treatment, including statistical, operational, and governance considerations.
4. Explain how a reproducible train–validation–test pipeline should be designed for a lapse or claims model.
5. Evaluate the ethical and regulatory risks of using location, behavioural, and text-derived features in insurance.

Analytical / Application Questions

1. An insurer observes 80 claims over 400 policy-years. Calculate the observed claim frequency and state its units. Explain why policy count alone may be insufficient.
2. A fraud model identifies 90 fraudulent claims and 30 are confirmed fraud; there are 10 additional fraud cases missed. Calculate precision, recall, and F1 score.
3. Design a feature and validation strategy for a monthly motor-renewal model where customers may hold multiple policies and policy records are updated after renewal. Identify at least three leakage risks and controls.

Multiple Choice Questions with Answers

Question 1

Which principle is most important when constructing a predictor for an insurance decision?

A. Use every variable available in the data warehouse

B. Use only information available at the prediction time

C. Prefer variables with the largest numerical values

D. Randomly replace missing values before splitting the data

Correct Answer: B. Use only information available at the prediction time

Question 2

Which transformation is commonly appropriate for a non-negative, right-skewed monetary variable containing zeros?

A. ln(1 + y)

B. 1 ÷ y

C. y − maximum(y)

D. y ÷ zero

Correct Answer: A. ln(1 + y)

Question 3

For a binary classifier, precision is defined as:

A. TP ÷ (TP + FN)

B. TN ÷ (TN + FP)

C. TP ÷ (TP + FP)

D. (TP + TN) ÷ all observations

Correct Answer: C. TP ÷ (TP + FP)

Question 4

Which method is most suitable when category order is meaningful and spacing is defensible?

A. Ordinal encoding

B. Random hashing only

C. Dropping the variable

D. Replacing every category with zero

Correct Answer: A. Ordinal encoding

Question 5

Why should SMOTE or other resampling methods be applied inside training folds?

A. To increase the test-set size

B. To prevent information crossing into validation or test data

C. To guarantee perfect calibration

D. To remove all outliers

Correct Answer: B. To prevent information crossing into validation or test data

Question 6

If a portfolio has 80 claims over 400 policy-years, its observed claim frequency is:

A. 0.02 claims per policy-year

B. 0.20 claims per policy-year

C. 3.20 claims per policy-year

D. 320 claims per policy-year

Correct Answer: B. 0.20 claims per policy-year

Question 7

Which feature is most clearly an example of potential target leakage in early fraud screening?

A. Prior claims known before notification

B. Policy tenure at notification

C. A note stating that fraud was confirmed after investigation

D. Vehicle age at notification

Correct Answer: C. A note stating that fraud was confirmed after investigation

Question 8

What does standardisation generally do?

A. Maps all values to exactly 0 or 1

B. Subtracts the training mean and divides by the training standard deviation

C. Deletes categorical variables

D. Guarantees a normal distribution

Correct Answer: B. Subtracts the training mean and divides by the training standard deviation

Question 9

Which evaluation design is generally most credible for a model deployed on future monthly policies?

A. A completely shuffled split with future and past mixed

B. A temporal split reflecting deployment order

C. A test set selected after reviewing its outcomes

D. Training on the test set and validating on training data

Correct Answer: B. A temporal split reflecting deployment order

Question 10

Why can postcode or location variables require additional governance review?

A. They can never be predictive

B. They may act as proxies for protected or socioeconomic characteristics

C. They always contain only numerical errors

D. They eliminate the need for human oversight

Correct Answer: B. They may act as proxies for protected or socioeconomic characteristics