Data Preparation and Feature Engineering

Introduction

Supervised and unsupervised learning algorithms do not learn directly from business problems; they learn from representations of data. A predictive model may be mathematically sophisticated yet operationally weak if its inputs are inaccurate, inconsistently defined, incomplete, contaminated by future information, or measured on incompatible scales. Data preparation and feature engineering therefore constitute a central part of analytical modelling rather than a preliminary clerical activity.

Data preparation transforms raw observations into an analysis-ready data matrix while preserving the information needed for valid inference and prediction. Feature engineering constructs variables that express useful patterns in the domain. Both activities must be performed within a disciplined workflow that protects against target leakage, sampling bias, and train–test contamination.

Let a dataset contain n observations and p candidate predictors. It may be represented as a matrix X = [xᵢⱼ], where xᵢⱼ is the value of feature j for observation i, together with a target vector y for supervised learning. In unsupervised learning, y is absent or deliberately excluded. The objective is not simply to maximize the number of columns. It is to create features that are valid, reproducible, available at the time of decision, and related to the analytical objective.

Learning Objectives

By the end of this chapter, students should be able to:

1. Audit data quality, structure, provenance, and measurement validity.
2. Design sampling and data-splitting procedures that reduce bias and contamination.
3. Diagnose and treat missing values and outliers appropriately.
4. Encode categorical and temporal information for learning algorithms.
5. Apply scaling and transformations under appropriate assumptions.
6. Select and construct features using statistical, predictive, and domain criteria.
7. Identify target leakage and train–test contamination.
8. Evaluate the governance, fairness, reproducibility, and operational implications of feature engineering.

1. Data Auditing: Establishing What the Data Mean

A data audit is a systematic examination of a dataset before modelling. It should answer four questions:

• What does each row represent?
• What does each variable measure, and in what units?
• How were observations collected and linked?
• Are the data available and reliable at the time of the intended decision?

A data dictionary should record variable name, business definition, type, unit, permissible range, source system, collection frequency, missing-value code, and whether the variable is available at prediction time. The apparent storage type is not necessarily the analytical type: a customer identifier stored as an integer is nominal, not quantitative; a postal code is usually categorical; and a date is temporal rather than an ordinary number.

1.1 Structural and logical checks

Typical audit checks include:

For a variable x, the sample mean and variance are commonly written as:

Here x̄ is the sample mean, s² is the unbiased sample variance under independent observations, xᵢ is observation i, and n is the number of non-missing observations. These summaries are meaningful only when the variable is quantitative and its units are understood. A mean of a customer identifier has no substantive interpretation.

Audit results should be documented as data-quality findings, not silently corrected. A negative transaction amount might be a refund, an error, or a valid accounting convention. Domain investigation must precede replacement.

2. Sampling, Splitting, and the Information Boundary

2.1 Sampling bias

Sampling bias occurs when the observed sample systematically differs from the population to which conclusions will be applied. Sources include convenience sampling, nonresponse, survivorship, undercoverage, and changes in data-collection systems. A model trained mostly on urban customers may perform poorly for rural customers even when its cross-validation score is high.

Stratified sampling can preserve important subgroup proportions. If a population has strata h and Nₕ units in stratum h, a proportionate sample of total size n allocates approximately:

where N is the population size. Proportionate allocation is not always optimal: if a small but important subgroup must be estimated precisely, deliberate oversampling may be appropriate, followed by suitable weighting for population-level estimates.

2.2 Train, validation, and test sets

A common supervised-learning design separates data into training, validation, and test sets. The training set estimates model parameters; the validation set supports model selection and hyperparameter tuning; the test set provides a final estimate of generalization. The test set must remain untouched until the analytical decisions are complete.

Random splitting is inappropriate when observations are temporally ordered, clustered, or repeated for the same entity. For forecasting, train on earlier periods and test on later periods. For customer-level prediction, all records for one customer should generally remain in one partition. Otherwise, the model may recognize customer-specific patterns rather than learn transferable relationships.

A robust workflow is:

1. Define the prediction target and prediction timestamp.
2. Split observations according to deployment conditions.
3. Fit imputation, encoding, scaling, selection, and transformation rules using training data only.
4. Apply the fitted rules unchanged to validation and test data.
5. Evaluate once on the untouched test set.

This principle applies to unsupervised learning as well. If principal components, clusters, or anomaly thresholds are calculated using the entire dataset before evaluation, information from the evaluation sample influences the representation.

2.3 Leakage and contamination

Target leakage occurs when a predictor contains information that would not be available when the prediction is made, or is a direct or indirect consequence of the target. For example, “days until loan default” is highly predictive of default but is unavailable at loan approval. Train–test contamination is broader: it occurs when information from validation or test observations influences preprocessing, feature selection, model tuning, or representation learning.

Examples include:

• imputing missing values using the global dataset median;
• selecting variables using correlations calculated on all observations;
• scaling before splitting;
• allowing duplicate customers across partitions;
• using a post-outcome refund code to predict purchase cancellation.

Leakage often produces impressive offline performance and disappointing operational performance. The relevant question is not “Does this variable predict the target?” but “Would this value be known, in this form, at the time of action?”

3. Missing-Value Treatment

Missingness can arise from nonresponse, system failure, inapplicability, censoring, or deliberate withholding. It is useful to distinguish:

• MCAR: missing completely at random;
• MAR: missingness related to observed variables;
• MNAR: missingness related to unobserved or missing values themselves.

These are assumptions about the missingness mechanism, not labels that can be proven from the observed data alone. Missingness itself may carry information. For example, absence of an income field may reflect an application channel or a customer’s reluctance to disclose information.

3.1 Main strategies

Deletion is acceptable only when missingness is limited, plausibly random, and the loss of observations does not distort representation. Single-value imputation includes mean, median, mode, or a domain-specific constant. Median imputation is more robust than mean imputation for skewed numerical variables, but it reduces variance and can weaken relationships.

More advanced approaches include regression imputation, k-nearest-neighbour imputation, and multiple imputation. Multiple imputation creates several plausible completed datasets, analyses each, and combines estimates to reflect uncertainty. For prediction, the most appropriate method depends on deployment constraints, computational cost, and whether missingness at prediction time is expected.

A missingness indicator can be added:

The imputation value and the indicator must be learned from the training set. A model can then distinguish an imputed value from an observed value when that distinction is substantively useful.

Imputation is not a substitute for investigating why data are missing. A value such as “not applicable” should not automatically be treated as an ordinary missing value, and the code −999 should not be interpreted as a legitimate numerical measurement.

4. Outliers and Anomalous Observations

An outlier is an observation unusually distant from the pattern of the data. It may be an error, a rare but valid case, a measurement from a different population, or an important event. Deleting outliers solely because they reduce model fit is poor practice.

For a roughly symmetric quantitative variable, the standardized score is:

where x̄ and s are the training-sample mean and standard deviation. A large absolute z-score may indicate an unusual observation, but there is no universal cutoff that proves an error.

The interquartile range is:

where Q₁ and Q₃ are the first and third quartiles. The conventional boxplot fences are Q₁ − 1.5IQR and Q₃ + 1.5IQR. These fences are useful screening rules, not automatic deletion rules.

Treatment options include correcting source errors, retaining valid extremes, winsorizing, applying a transformation, using robust statistics, or modelling a separate process. In fraud detection, the extreme observations may be the business objective. In ordinary least squares regression, extreme predictor values can exert high leverage, while extreme response values can strongly affect residual fit. Diagnostics should therefore examine both data validity and model influence.

5. Encoding Categorical and Temporal Variables

Categorical variables require representations compatible with the algorithm. For a nominal variable with k categories, one-hot encoding creates binary indicators. If exactly one category is present for each record, using all k indicators with an intercept creates perfect linear dependence; regression implementations may therefore omit one reference category. Tree-based methods do not require the same scaling assumptions, but careless integer encoding can falsely imply order.

Ordinal encoding is appropriate only when categories have a defensible order, such as bronze, silver, and gold. Even then, equal numerical spacing should not automatically be interpreted as equal substantive differences.

High-cardinality variables create sparse, computationally costly matrices. Alternatives include frequency encoding, hashing, carefully regularized target encoding, or domain-based grouping. Target encoding is especially vulnerable to leakage and should be computed within training folds, with smoothing and out-of-fold construction.

Dates should rarely be represented only as a raw integer. Useful features may include day of week, month, elapsed time, season, holiday status, and time since the previous event. Cyclical variables can preserve circular adjacency:

Here t is the position in a cycle and P is the cycle length, such as 7 for day-of-week or 12 for month. The two features prevent December and January from appearing maximally distant in a linear representation.

6. Scaling and Transformation

Scaling is important for distance-based, gradient-based, and regularized models. It is often less important for ordinary decision trees, although preprocessing may still support consistent pipelines.

Standardization is:

where μₜᵣₐᵢₙ and σₜᵣₐᵢₙ are the training-set mean and standard deviation. The transformed variable is dimensionless. Min–max scaling is:

where a and b are training-set minimum and maximum values. It maps training values to [0, 1], but future values may fall outside this interval. Robust scaling uses the training median and IQR and is less sensitive to extreme values.

Logarithmic transformation can reduce right skew:

for x > 0. For nonnegative data containing zero, ln(1+x) is often used, but its interpretation differs from a conventional elasticity interpretation. A power transformation such as Box–Cox is defined for strictly positive values; the transformation parameter must be estimated from training data. Transformations should be selected for a modelling purpose, not applied mechanically.

7. Feature Selection

Feature selection reduces noise, computation, overfitting, and operational burden. It can also improve interpretability, but removing variables solely for convenience may eliminate important signals.

7.1 Filter methods

Filter methods use properties of the data independently of a final model. Examples include variance thresholds, correlation screening, mutual information, and statistical tests. They are computationally efficient but may miss interactions and may be inappropriate when assumptions are violated. Correlation measures linear association; a low Pearson correlation does not imply absence of a nonlinear relationship.

7.2 Wrapper and embedded methods

Wrapper methods evaluate subsets using model performance, such as recursive feature elimination. They can be computationally expensive and must be nested within resampling to avoid optimistic estimates. Embedded methods perform selection during training. Lasso regression, for example, minimizes squared error plus an absolute-coefficient penalty:

Here β₀ is the intercept, βⱼ are coefficients, λ ≥ 0 controls regularization, and n and p denote observations and predictors. The formula assumes a regression setting with squared-error loss; predictors are commonly standardized so that the penalty treats coefficients comparably. Lasso can set some coefficients exactly to zero, but correlated predictors may be selected inconsistently.

Feature selection should account for stability, domain validity, fairness, availability, and maintenance cost. A small performance improvement may not justify a feature requiring an expensive or unreliable data pipeline.

8. Feature Construction and Representation Learning

Feature construction creates variables from existing information. Useful patterns include:

• ratios, such as revenue per active customer;
• counts, such as purchases in the previous 30 days;
• recency and elapsed-time measures;
• rolling means and volatility measures;
• interactions, such as price × promotion;
• aggregated customer, product, or geographic histories;
• text-derived counts, embeddings, or topic representations.

Every constructed feature needs a precise time window. “Average spend in the last 30 days” must specify whether the endpoint is the prediction timestamp and whether the current transaction is included. Including future transactions is leakage.

Interaction terms allow the effect of one variable to depend on another. In a regression model, an interaction may be represented as x₁x₂. Its coefficient describes the change in the effect of x₁ for a one-unit change in x₂, subject to the model’s scale and specification. Centering continuous variables can improve interpretability and reduce nonessential collinearity, but it does not eliminate substantive dependence.

Unsupervised representation methods such as principal component analysis (PCA) construct orthogonal directions of variation. If X is centered, PCA seeks directions w of unit length that maximize projected variance, with the first direction solving:

PCA is sensitive to scaling and outliers. Its components maximize variance, not predictive relevance or interpretability. The fitted rotation and any scaling must be learned from training data when PCA is used in a predictive pipeline.

9. A Reproducible Preparation Pipeline

A professional pipeline combines auditing, transformations, modelling, and evaluation into a traceable sequence. A typical design is:

Figure: A leakage-resistant preparation and evaluation pipeline.

A pipeline should store fitted parameters, software versions, feature definitions, random seeds where relevant, and data snapshots or hashes. Reproducibility means that another analyst can reconstruct not only the final model but also the exact information available at training time.

10. Illustrative Case: Predicting Subscription Churn

Background

A telecommunications firm wants to predict whether a customer will cancel within the next 30 days. The model will guide retention offers.

Problem

The raw data include billing records, customer-service contacts, usage histories, contract information, and cancellation records. Initial modelling reports unusually high accuracy.

Analysis

An audit finds that the dataset contains a “disconnect request date,” a refund flag generated after cancellation, and multiple monthly records for the same customer. Missing income values are more frequent for prepaid customers. Usage is strongly right-skewed, and several negative charges are legitimate refunds rather than errors.

The original random row-level split places records from the same customer in both training and test sets. Furthermore, the refund flag is created after cancellation and therefore leaks the target.

Approach and solution

The team defines the prediction timestamp as the end of each billing day and the target as cancellation during the subsequent 30 days. All records from a customer are assigned to one partition, and a time-based holdout is used for the final test. Post-cancellation fields are removed. Features are reconstructed using only events before the prediction timestamp, including recent usage, number of service contacts in the previous 60 days, tenure, payment failures, and a rolling change in usage.

Missing numeric values are imputed using training-set medians with missingness indicators. Legitimate refunds are retained and represented separately from data-quality errors. Usage is log-transformed, and high-cardinality service categories are grouped using training data. All operations are implemented in a single reproducible pipeline.

Outcome

Performance falls after leakage removal, but the estimate becomes credible. The model performs less well for customers with sparse histories, so the firm adds a minimum-history rule and monitors subgroup performance. Retention decisions are reviewed for fairness: a missing income field is not treated as evidence of low ability to pay, and explanations are provided to customer-service staff.

Learning points

The case illustrates that lower but valid test performance is preferable to inflated performance caused by leakage. It also shows that feature engineering is a temporal and organizational task: the quality of a feature depends on when and why it was created, not merely on its statistical association with churn.

11. Challenges, Ethics, and Governance

Data preparation can reproduce historical discrimination. Proxy variables such as location, language, device type, or purchasing channel may encode protected characteristics. Removing an explicit protected attribute does not guarantee fairness if correlated proxies remain. Analysts should evaluate performance and error rates across relevant groups, document intended use, and involve domain and legal experts where decisions affect access to employment, credit, insurance, housing, or essential services.

Privacy risks arise when feature construction combines datasets or creates detailed behavioural histories. Data minimization, purpose limitation, access controls, retention limits, and appropriate de-identification are governance requirements, not optional technical refinements. Aggregation can reduce exposure but does not guarantee anonymity.

Automated feature generation and large language models can accelerate experimentation, but they may create unverifiable variables, encode sensitive information, or use inappropriate external knowledge. Generated features require human review, provenance records, leakage testing, and stability assessment.

12. Best-Practice Checklist

• Define the unit of analysis, target, prediction time, and deployment population.
• Create and maintain a data dictionary and provenance record.
• Audit duplicates, ranges, units, timestamps, joins, and missingness.
• Split by time, entity, or group when deployment requires it.
• Fit every learned preprocessing step on training data only.
• Treat missingness and outliers as substantive questions, not merely technical defects.
• Use encodings that respect nominal, ordinal, and cyclical structure.
• Scale models that depend on distance, gradients, or coefficient penalties.
• Construct features with explicit historical windows and no future information.
• Select features inside resampling procedures.
• Test stability, subgroup performance, interpretability, cost, and availability.
• Version data, code, feature definitions, and fitted preprocessing parameters.
• Monitor production drift and revisit assumptions after deployment.

Chapter Summary

Data preparation establishes the validity of supervised and unsupervised learning. A rigorous process begins with an audit of meaning, provenance, quality, time, and sampling. Sampling design and data partitioning must reflect deployment conditions, especially for temporal, clustered, and repeated observations. Missing values require consideration of their mechanism and operational meaning; outliers must be investigated rather than automatically deleted.

Categorical, temporal, numerical, and high-cardinality variables require appropriate representations. Scaling and transformation should match algorithmic assumptions and must be fitted only on training data. Feature selection can use filter, wrapper, or embedded methods, but selection must occur within resampling to avoid optimistic estimates. Feature construction can reveal domain patterns through aggregation, recency, ratios, interactions, and temporal windows, while PCA and related methods provide unsupervised representations with important interpretive limitations.

The central discipline is preservation of the information boundary. No target-derived, future, test-set, or post-outcome information may influence training or preprocessing. Ethical governance further requires attention to fairness, privacy, documentation, reproducibility, and operational monitoring.

Key Terms

Data audit: Systematic examination of data structure, meaning, quality, and provenance.

Data dictionary: Record describing variables, units, definitions, sources, and permissible values.

Sampling bias: Systematic difference between the observed sample and the target population.

Train–test contamination: Influence of evaluation data on preprocessing, selection, tuning, or modelling.

Target leakage: Use of information unavailable at prediction time or caused by the target.

Missing completely at random (MCAR): Missingness unrelated to observed or unobserved values.

Missing at random (MAR): Missingness related to observed information after conditioning.

Missing not at random (MNAR): Missingness related to unobserved or missing information.

Imputation: Replacement or modelling of missing values.

Outlier: An observation unusually distant from the prevailing data pattern.

Winsorization: Limiting extreme values to selected percentile or boundary values.

One-hot encoding: Representation of categories using binary indicator variables.

Target encoding: Replacing a category with a target-related summary, requiring leakage controls.

Standardization: Rescaling using a training mean and standard deviation.

Feature transformation: Mathematical alteration of a variable, such as a logarithm.

Feature selection: Choosing a subset of available predictors.

Feature construction: Creating new predictors from existing information.

Interaction: A relationship in which the effect of one variable depends on another.

Principal component analysis: Unsupervised projection onto directions of maximum variance.

Feature drift: Change in the distribution or meaning of a feature after deployment.

Review Questions

Short Answer Questions

1. Why is a data dictionary important before modelling?
2. Distinguish target leakage from ordinary predictive association.
3. When might median imputation be preferable to mean imputation?
4. Why should date variables not always be represented as raw integers?
5. Which types of algorithms are especially sensitive to feature scale?

Long Answer Questions

1. Explain how a leakage-resistant supervised-learning pipeline should be designed from data splitting through final testing.
2. Compare filter, wrapper, and embedded feature-selection methods, including their main strengths and risks.
3. Discuss the statistical and operational considerations involved in treating outliers.
4. Explain how missingness mechanisms influence imputation decisions and model interpretation.
5. Evaluate the ethical and governance risks of constructing features from customer behavioural data.

Analytical / Application Questions

1. A bank uses a variable called “collections contact completed” to predict default at loan approval. Diagnose the problem and propose a valid alternative feature design.
2. A retailer has ten years of customer transactions, with several records per customer and seasonal demand. Design an appropriate training, validation, and test strategy and justify it.
3. A churn model performs well under random row-level splitting but poorly on next-quarter customers. Identify possible causes and propose an audit and remediation plan.

Multiple Choice Questions with Answers

Question 1

Which action most directly prevents preprocessing leakage?

A. Fit imputation and scaling parameters using the training set only

B. Remove all variables with missing values

C. Use a larger test set than training set

D. Encode every variable as an integer

Correct Answer: A. Fit imputation and scaling parameters using the training set only

Question 2

A postal code stored as 1042 should ordinarily be treated as which type of variable?

A. Continuous numerical

B. Nominal categorical

C. Ratio-scale measurement

D. Response variable

Correct Answer: B. Nominal categorical

Question 3

For a positive, right-skewed variable, which transformation is commonly appropriate?

A. Random permutation

B. Logarithmic transformation

C. Replacing every value with its rank only

D. Duplicating the largest observations

Correct Answer: B. Logarithmic transformation

Question 4

What is the main danger of using a post-cancellation refund flag to predict churn?

A. Underfitting

B. Multicollinearity only

C. Target leakage

D. Excessive standardization

Correct Answer: C. Target leakage

Question 5

In the conventional boxplot rule, observations above which boundary are flagged for screening?

A. Q₁ + 1.5IQR

B. Q₃ + 1.5IQR

C. Mean + 0.5IQR

D. Median + 3Q₁

Correct Answer: B. Q₃ + 1.5IQR

Question 6

Which split is generally most appropriate for forecasting future demand?

A. Random split of all rows

B. Split by alphabetical product name

C. Train on earlier periods and test on later periods

D. Place future observations in training to increase sample size

Correct Answer: C. Train on earlier periods and test on later periods

Question 7

Why can integer encoding be inappropriate for nominal categories?

A. It always creates missing values

B. It may falsely imply an ordering and distance

C. It prevents any model from training

D. It guarantees target leakage

Correct Answer: B. It may falsely imply an ordering and distance

Question 8

What does Lasso regularization potentially do to coefficients?

A. Force all coefficients to be positive

B. Set some coefficients exactly to zero

C. Make all predictors categorical

D. Guarantee causal interpretation

Correct Answer: B. Set some coefficients exactly to zero

Question 9

For a cyclical month variable, why can sine and cosine features be useful?

A. They preserve adjacency between December and January

B. They remove every seasonal pattern

C. They convert the target into a category

D. They guarantee normally distributed data

Correct Answer: A. They preserve adjacency between December and January

Question 10

In the churn case, why should all records for one customer generally remain in one partition?

A. To increase the number of columns

B. To prevent the model from learning customer-specific information across partitions

C. To ensure every partition has identical row counts

D. To eliminate the need for validation

Correct Answer: B. To prevent the model from learning customer-specific information across partitions