Data Preparation and Feature Engineering

Introduction

Machine-learning models learn from the representation of data supplied to them. In business settings, that representation is rarely ready for direct use. Customer records may contain missing values, inconsistent categories, duplicate transactions, extreme observations, dates stored as text, and variables measured on incompatible scales. A model trained on such data may be inaccurate, unstable, difficult to explain, or impossible to reproduce.

Data preparation is the disciplined process of converting raw data into a reliable analytical dataset. Feature engineering goes further: it creates, transforms, selects, or combines variables so that the data expresses patterns relevant to the business problem. These activities are not merely technical housekeeping. They influence model validity, fairness, computational cost, interpretability, and the credibility of managerial decisions.

A central principle governs the entire chapter: information available when a prediction is made must be distinguished from information observed later. Violating this principle creates target leakage and produces deceptively strong validation results.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the role of data preparation and feature engineering in the machine-learning workflow.
2. Design reproducible preprocessing pipelines for numerical, categorical, textual, and temporal data.
3. Select appropriate methods for missing values, outliers, scaling, transformation, encoding, and feature selection.
4. Identify target leakage and train–test contamination.
5. Address dimensionality, class imbalance, and operational constraints.
6. Evaluate preprocessing choices using validation data and business-relevant criteria.

1. The Analytical Dataset and the Prediction-Time Boundary

A supervised-learning dataset commonly consists of a feature matrix X and a target vector y. Each row represents an observational unit, such as a customer, invoice, loan application, or store-day. Each column in X is a candidate predictor, while y is the outcome to be predicted.

The observational unit must be defined before preprocessing begins. For example, a churn model may use one row per customer at the end of each month. Mixing customer-level and transaction-level rows can create duplicated information and incorrect labels. The prediction time must also be stated: which facts would actually be known at the moment of decision?

A sound workflow separates data into:

• Training data: used to estimate model parameters and preprocessing parameters.
• Validation data: used to compare models and tune hyperparameters.
• Test data: held back until final evaluation.
• Production data: new observations processed using the already-fitted pipeline.

For a random split, observations are assigned to partitions, but random splitting is inappropriate when time, customers, households, or other groups create dependence. A forecasting problem should usually use a chronological split. If the same customer appears in both training and test data, a group-based split may be necessary to measure performance on genuinely new customers.

Leakage and contamination

Target leakage occurs when a feature contains information caused by, or recorded after, the target event. A loan-default model that uses “collections action taken” may appear accurate because collections action occurs after financial distress is recognized. A hospital readmission model using discharge-summary information may similarly use facts unavailable at admission.

Train–test contamination occurs when information from validation or test data influences preprocessing or model selection. For example, calculating a mean using the full dataset before splitting allows test observations to affect the training transformation. Even if the target is not used, this violates strict evaluation.

The remedy is to fit every data-dependent transformation only on the training portion within each validation split, then apply the fitted transformation to validation, test, and production observations.

2. A Reproducible Preprocessing Pipeline

A preprocessing pipeline is an ordered sequence of transformations followed by a predictive model. Reproducibility requires that the sequence, parameters, software versions, data definitions, and random seeds be recorded. A typical pipeline is:

Raw data → quality checks → split data → fit preprocessing on training data
          → transform training/validation/test data → fit model → evaluate → deploy

A pipeline should avoid manually transforming separate datasets because this encourages inconsistent column ordering, accidental leakage, and undocumented decisions. In a production environment, the same serialized pipeline should process both training and future observations.

A useful design distinguishes:

• Deterministic transformations: parsing a date or converting currency using a specified exchange rate.
• Fitted transformations: imputing a median, estimating category levels, scaling, selecting features, or learning a text vocabulary.
• Model fitting: estimating predictive parameters.

Only the training data should determine fitted transformations. A compact pseudocode pattern is:

split raw data into training and test sets
fit preprocessing on training features only
transform training and test features with the fitted preprocessing
fit model on transformed training data
evaluate once on transformed test data

For cross-validation, the preprocessing must be refitted inside every training fold. This is normally achieved by placing preprocessing and the estimator in one pipeline object.

3. Data Quality and Missing Values

Data-quality preparation begins with structural checks:

• Are identifiers unique where they should be unique?
• Are data types correct?
• Are units consistent?
• Are impossible values present, such as negative age or a discount above 100%?
• Are duplicated records genuine repeated events or accidental copies?
• Are missing values encoded as blanks, zero, “unknown,” or special codes?

Missingness is not automatically an error. A missing income value may reflect nonresponse; a missing laboratory test may indicate that a clinician did not order it. The pattern itself can contain business information.

Three concepts are useful:

• MCAR: missing completely at random; missingness is unrelated to observed and unobserved values.
• MAR: missing at random conditional on observed variables.
• MNAR: missingness depends on the missing value itself or unobserved factors.

These assumptions cannot usually be proven from the observed dataset. They guide, rather than mechanically determine, the treatment strategy.

Imputation methods

For numerical variables, common choices include mean, median, model-based, and groupwise imputation. The median is generally more robust than the mean when distributions are skewed. For a training variable x with observed values x₁, …, xₙ, the mean is:

where n is the number of observed training values. The imputation statistic must be calculated from training data only. A missingness indicator can be added:

This allows a model to distinguish an imputed value from an originally observed value, although indicators should be used thoughtfully because they may encode operational processes rather than meaningful phenomena.

For categorical variables, common approaches are a dedicated “Missing” category or the most frequent training category. Dropping rows may be appropriate only when missing records are few and plausibly missing at random; otherwise, deletion can introduce bias and reduce sample size.

Imputation should be documented, monitored, and revisited when production missingness differs from training missingness. A model may fail not because the algorithm changed, but because the data-collection process changed.

4. Numerical Transformation and Scaling

Scaling

Many algorithms are sensitive to the scale of variables. Distance-based methods, k-nearest neighbors, k-means, support-vector machines, and regularized linear models can be dominated by variables with large numerical ranges.

Standardization transforms a value x into a z-score:

where μ is the training mean and σ is the training standard deviation. It is appropriate when a roughly symmetric scale is useful and σ is nonzero. The result is dimensionless, with training mean approximately 0 and standard deviation approximately 1.

Min–max scaling maps a value to a selected interval, commonly [0, 1]:

where xₘᵢₙ and xₘₐₓ are training minimum and maximum values. It is sensitive to outliers and future values can fall outside [0, 1].

Robust scaling uses the median and interquartile range (IQR):

The IQR is the third quartile minus the first quartile. Robust scaling is useful when extreme values are legitimate but can distort mean and standard-deviation estimates.

Tree-based models generally do not require scaling because their split rules depend on order thresholds rather than Euclidean distance. Nevertheless, consistent scaling may simplify a shared pipeline or support a later model comparison.

Skewness and transformations

Revenue, claim amounts, and customer tenure may be strongly right-skewed. A logarithmic transformation compresses large values:

The “1” permits transformation of zero values, assuming x ≥ 0. For values that can be negative, a different transformation or a carefully chosen shift is required. Transformations should preserve business meaning and should not be applied automatically merely to make distributions look normal.

Outliers should be investigated rather than removed reflexively. An extreme transaction may be a data-entry error, fraud signal, or genuine high-value event. Options include correction, capping (winsorization), robust transformation, separate indicators, or retention. The chosen method must be based on domain knowledge and documented rules.

5. Encoding Categorical and Temporal Variables

Categorical variables contain labels rather than numerical magnitudes. Assigning arbitrary numbers such as 1 = bronze, 2 = silver, and 3 = gold implies an ordering and distance that may not exist.

One-hot encoding

For a categorical variable with categories A, B, and C, one-hot encoding creates indicator variables. An observation in category B might be represented as (0, 1, 0). For a linear model with an intercept, one category is often omitted as a reference category to avoid perfect multicollinearity. This is called dummy-variable encoding.

Unknown or newly appearing categories must be handled explicitly. A production pipeline can group rare categories into “Other” and map unseen categories to an unknown level. Fitting the category vocabulary on the full dataset before splitting is a form of contamination.

Ordinal, frequency, and target encoding

Ordinal encoding is suitable only when categories possess a meaningful order, such as low, medium, and high. Frequency encoding replaces a category with its training frequency, but may obscure distinctions between categories. Target encoding replaces a category with an estimate of its average target; it can be powerful for high-cardinality variables but is highly vulnerable to leakage. It must be computed within training folds, often with smoothing and out-of-fold estimates.

For a binary target y ∈ {0, 1}, a category-level target mean is:

where n_c is the number of training observations in category c. This statistic must never be calculated using validation or test targets.

Dates and time

Dates should rarely be treated as opaque strings. Useful features may include year, month, day of week, elapsed days, season, holiday indicator, or time since a relevant event. Cyclical variables such as hour of day can be represented using sine and cosine so that midnight and 23:00 are close:

where t is the time position and P is the cycle length, such as 24 hours or 12 months. Temporal features must respect the information available at prediction time. A “days until contract termination” feature is leakage if the termination date is known only after the prediction is made.

6. Feature Engineering for Business Problems

Feature engineering translates domain knowledge into measurable predictors. Examples include:

• Recency: days since the customer’s last purchase.
• Frequency: number of purchases in a defined period.
• Monetary value: total or average spending.
• Ratios: revenue per employee or claims per policy.
• Aggregations: average order value by customer or store.
• Interactions: advertising spend combined with season or region.
• Rolling measures: seven-day average demand.
• Text features: term frequency, inverse document frequency, or sentiment indicators.

Every engineered feature should specify its observation window, unit, aggregation level, and cutoff date. Aggregations must be calculated using only records available before the prediction timestamp. For example, a customer’s “lifetime spend” for a January prediction must exclude February purchases.

Business value should guide feature creation. A highly predictive feature that is expensive, delayed, legally restricted, or unavailable in production may not be useful. Feature documentation should include its source, owner, transformation logic, refresh frequency, and permitted use.

7. Feature Selection and Dimensionality Reduction

High dimensionality can increase computational cost, overfitting risk, storage requirements, and difficulty of explanation. Feature selection is preferable to indiscriminate deletion because it connects statistical evidence with business purpose.

Filter methods

Filter methods rank features independently of the final model. Examples include variance thresholds, correlation screening, mutual information, and statistical tests. They are computationally efficient but may miss interactions and can select redundant variables.

Correlation measures linear association, not causation, and a low correlation does not prove that a variable is useless. A feature may matter through a nonlinear relationship or interaction.

Wrapper and embedded methods

Wrapper methods evaluate subsets using model performance, such as recursive feature elimination, but can be computationally expensive. Embedded methods perform selection during model fitting. Lasso regression, for example, minimizes a loss plus an L1 penalty:

where L(β) is the model loss, βⱼ is the coefficient for feature j, p is the number of features, and λ ≥ 0 controls the penalty. Larger λ generally encourages more coefficients to become exactly zero. The value of λ must be selected using training data and validation, not the test set.

Principal component analysis

Principal component analysis (PCA) creates new orthogonal directions that capture variance in the predictors. It can reduce dimensionality but sacrifices direct interpretability. PCA should generally be fitted on training data and often preceded by scaling when variables have different units. It is not a feature-selection method in the strict sense because it creates combinations rather than retaining original variables.

8. Class Imbalance and Sampling

In fraud detection, equipment failure, and disease screening, the positive class may be rare. Accuracy can then be misleading. If 99% of transactions are legitimate, a model that always predicts “legitimate” has 99% accuracy but detects no fraud.

Useful metrics include precision, recall, F1 score, area under the precision–recall curve, and cost-weighted measures. For binary classification:

Here TP, FP, and FN denote true positives, false positives, and false negatives. These formulas require nonzero denominators. Precision focuses on the reliability of positive alerts; recall focuses on the proportion of actual positives detected.

Possible responses include class-weighted learning, threshold adjustment, oversampling the minority class, undersampling the majority class, and synthetic methods such as SMOTE. Resampling must occur inside the training fold only. Applying oversampling before splitting can place near-duplicate or synthetic information in both training and test data.

The preferred threshold depends on business costs. If a missed fraud case costs much more than an investigation, the organization may accept lower precision to achieve higher recall. The decision should be based on a cost matrix, operational capacity, and fairness analysis rather than on a default threshold of 0.5.

9. Worked Example: A Churn-Model Pipeline

Consider a telecommunications company predicting whether a customer will cancel in the next 30 days. The unit is one customer-month, and the prediction cutoff is the last day of the current month.

Raw variables include monthly charges, contract type, customer tenure, number of support contacts, payment method, and service events. The target is cancellation during the following 30 days.

A defensible pipeline is:

1. Remove duplicate customer-month records and verify one row per customer-month.
2. Split chronologically: earlier months for training, a later month for validation, and the most recent month for final testing.
3. Impute numerical missing values using training medians and add missingness indicators where operationally meaningful.
4. Encode contract type and payment method using one-hot encoding, with an unknown-category policy.
5. Standardize numerical variables for a regularized logistic-regression model.
6. Create recency and frequency features using only support and service records available by the cutoff date.
7. Exclude cancellation date, retention-offer acceptance, and post-cutoff service events because they reveal future information.
8. Use class weights or training-fold resampling if cancellations are rare.
9. Select the operating threshold using validation costs, then evaluate once on the untouched test month.

Suppose the validation confusion matrix at a selected threshold contains TP = 180, FP = 120, and FN = 60. Then:

The model identifies 75% of actual churners, while 60% of flagged customers actually churn. Management must compare the value of successfully retaining customers with the cost of contacting false positives. The threshold should not be chosen from F1 alone if the business has explicit intervention costs.

10. Illustrative Case: Credit-Risk Preparation Failure

Background. A lender develops a model to predict whether a new applicant will default within twelve months.

Problem. The first model reports an area under the ROC curve of 0.97, far above historical performance. Investigation shows that the dataset includes “days past due” measured three months after application and a collections-status code. Both variables are strongly related to default, but neither is available at application time.

Analysis. The model has target leakage. In addition, the team standardized variables using the complete dataset before cross-validation and selected features after examining the test results. These practices contaminate evaluation.

Approach. The team defines the application timestamp as the prediction boundary, removes post-application variables, and rebuilds all features from information available at or before that timestamp. A chronological holdout is used to represent future applicants. Imputation, encoding, scaling, and feature selection are placed inside a cross-validation pipeline. Performance is assessed using discrimination, calibration, approval-rate constraints, and subgroup error rates.

Outcome. The reported performance falls, but it becomes credible and operationally reproducible. The lender can now monitor whether applicant characteristics, missingness patterns, and approval outcomes change over time.

Learning points. Extraordinary performance warrants a leakage audit. A realistic evaluation is more valuable than an inflated score. The prediction-time boundary, not convenience of data access, determines whether a feature is legitimate.

11. Governance, Fairness, and Operational Monitoring

Preprocessing decisions can create discriminatory outcomes. Imputation may treat groups differently if missingness reflects unequal access to services. Location, language, device type, or postal code may act as proxies for protected characteristics. Feature selection based solely on predictive strength can retain variables that conflict with organizational policy or law.

Governance should address data minimization, consent, lawful purpose, retention, access control, explainability, and documentation. Sensitive attributes may need to be excluded from prediction but retained separately for fairness auditing, subject to applicable law and policy. Removing a protected variable does not guarantee fairness because correlated proxies can remain.

After deployment, teams should monitor:

• missing-value rates and category drift;
• feature distributions and data types;
• prediction distributions and calibration;
• performance when labels become available;
• subgroup precision, recall, and error rates;
• pipeline failures and unexpected categories.

A preprocessing pipeline is therefore a governed software artifact, not a one-time notebook. Version control, automated tests, data contracts, lineage records, and rollback procedures improve reliability.

12. Best Practices and Common Errors

Strong practice includes:

• define the observational unit and prediction cutoff;
• split before fitting data-dependent transformations;
• use pipelines and cross-validation correctly;
• preserve raw data and document every transformation;
• investigate outliers and missingness rather than applying automatic deletion;
• use time- or group-based splits when random splitting is unrealistic;
• select thresholds using business costs and operational capacity;
• test production behavior with unseen categories and missing fields;
• monitor drift and retrain only under explicit governance.

Common errors include fitting an imputer on all data, scaling before splitting, target encoding without out-of-fold logic, oversampling before the split, using future information in rolling features, treating identifiers as meaningful predictors, and reporting only accuracy for imbalanced outcomes.

Contemporary Developments

Modern feature engineering increasingly uses automated feature-generation systems, embeddings for text and categorical entities, feature stores, and data-quality monitoring. These technologies can improve productivity but do not remove the need for temporal reasoning, leakage controls, interpretability, and governance. Deep-learning representations may reduce manual engineering in images, audio, and language, while tabular business data still benefits substantially from carefully defined aggregates, missingness treatment, and domain-informed variables.

The most effective practice combines automation with human review. Automated systems can generate candidate features, but analysts must verify availability, meaning, stability, legal acceptability, and value in production.

Chapter Summary

Data preparation and feature engineering determine how a business problem is represented to a machine-learning model. Reliable work begins by defining the observational unit, target, prediction-time boundary, and appropriate train–validation–test strategy. Missing values require an explicit treatment based on data meaning and missingness patterns. Numerical variables may need imputation, robust handling of outliers, transformation, or scaling; categorical variables require encoding that respects nominal or ordinal structure. Dates, text, and transactions can yield valuable engineered features when their time windows and aggregation levels are controlled.

Feature selection and dimensionality reduction can reduce overfitting, computation, and explanation burdens, but all selection must occur within the training process. Class imbalance requires suitable metrics, sampling or weighting strategies, and business-informed thresholds. The central validity requirement is prevention of target leakage and train–test contamination. Reproducible pipelines, documentation, monitoring, fairness review, and governance convert preprocessing from an informal data-cleaning exercise into a dependable analytical capability.

Key Terms

Analytical dataset: Structured data containing predictors, targets, and defined observational units for analysis.

Target leakage: Use of information unavailable at prediction time or caused by the outcome.

Train–test contamination: Allowing validation or test information to influence training or preprocessing.

Imputation: Replacement of missing values using a defined rule or estimated value.

Missingness indicator: A variable recording whether another value was missing.

Standardization: Transformation using a training mean and standard deviation.

Min–max scaling: Mapping values using training minimum and maximum values.

One-hot encoding: Representation of categories as binary indicator variables.

Target encoding: Replacing categories with target-based statistics.

Feature engineering: Creation or transformation of predictors using domain and data knowledge.

Feature selection: Choosing a subset of predictors for modeling.

Dimensionality reduction: Representing data with fewer dimensions, often through new components.

PCA: Principal component analysis, a variance-based dimensionality-reduction method.

Class imbalance: A substantial difference in the frequencies of outcome classes.

Precision: The proportion of predicted positives that are true positives.

Recall: The proportion of actual positives that are detected.

Data drift: Change in input-data distributions over time.

Prediction-time boundary: The timestamp defining which information may legitimately be used.

Review Questions

Short Answer Questions

1. Why must fitted preprocessing transformations be learned from training data only?
2. Distinguish target leakage from train–test contamination.
3. When is one-hot encoding preferable to ordinal encoding?
4. State the difference between precision and recall.
5. Why might a chronological split be preferable to a random split?

Long Answer Questions

1. Explain how you would design a reproducible preprocessing pipeline for a customer-churn problem.
2. Compare mean imputation, median imputation, missingness indicators, and deletion of incomplete records.
3. Discuss the advantages and limitations of filter, wrapper, and embedded feature-selection methods.
4. Explain how class imbalance affects model evaluation and describe several appropriate responses.
5. Discuss the ethical and governance risks associated with feature engineering in credit or employment decisions.

Analytical / Application Questions

1. A fraud model has 50 true positives, 30 false positives, and 20 false negatives. Calculate precision, recall, and F1 score, and explain which operational concern each metric represents.
2. A retailer creates a customer feature called “total annual spending” using all purchases in the calendar year, then predicts churn in June. Identify the validity problem and propose a corrected feature definition.
3. A model’s validation score is excellent, but production data contains many new categories and a much higher missing-value rate. Diagnose two possible causes and recommend a monitoring and remediation plan.

Multiple Choice Questions with Answers

Question 1

Which action best prevents preprocessing leakage during cross-validation?

A. Fit the imputer and scaler on the complete dataset before cross-validation

B. Fit preprocessing separately inside each training fold

C. Fit preprocessing only on the test set

D. Skip preprocessing for validation data

Correct Answer: B. Fit preprocessing separately inside each training fold

Question 2

Which feature is most clearly an example of target leakage in an application-time credit model?

A. Applicant age

B. Requested loan amount

C. Collections status recorded after application

D. Employment length at application

Correct Answer: C. Collections status recorded after application

Question 3

What does standardization usually produce when applied using training-set parameters?

A. Values restricted between 0 and 1

B. Values with approximately zero mean and unit standard deviation

C. Only integer-valued observations

D. A ranking of categorical levels

Correct Answer: B. Values with approximately zero mean and unit standard deviation

Question 4

Which encoding is generally appropriate for a nominal variable such as payment method?

A. One-hot encoding

B. Ordinal encoding with arbitrary numeric ranks

C. Replacing every category with the target value

D. Deleting the variable automatically

Correct Answer: A. One-hot encoding

Question 5

A classifier produces TP = 80 and FP = 20. What is its precision?

A. 0.20

B. 0. cinc

C. 0.80

D. 0.90

Correct Answer: D. 0.90

Question 6

Which procedure is most appropriate for a forecasting problem?

A. Randomly mixing all dates before splitting

B. Using a chronological training and test split

C. Using the test target to select features

D. Oversampling the full dataset before splitting

Correct Answer: B. Using a chronological training and test split

Question 7

Why can accuracy be misleading for a rare-event fraud problem?

A. Accuracy cannot be calculated for binary outcomes

B. A majority-class model can achieve high accuracy while missing nearly all fraud

C. Accuracy always equals recall

D. Fraud data never contains missing values

Correct Answer: B. A majority-class model can achieve high accuracy while missing nearly all fraud

Question 8

What is the main purpose of a missingness indicator?

A. To guarantee that missing values are random

B. To record whether a value was originally missing

C. To convert categories into principal components

D. To remove all outliers

Correct Answer: B. To record whether a value was originally missing

Question 9

In the transformation z = (x − μ)/σ, what does μ represent?

A. The training mean

B. The training maximum

C. The target prevalence

D. The number of features

Correct Answer: A. The training mean

Question 10

Which practice is correct when using target encoding?

A. Calculate category target means using test targets

B. Calculate encodings using all labels before splitting

C. Use training-fold or out-of-fold target statistics

D. Use the same target mean for every category without checking data

Correct Answer: C. Use training-fold or out-of-fold target statistics