Data Preparation and Feature Engineering

Introduction

Machine-learning performance depends not only on the choice of algorithm but also on the quality, structure, and provenance of the data supplied to it. Data preparation converts raw observations into an analytical dataset whose variables are usable, comparable, informative, and available at the time a prediction would actually be made. Feature engineering then transforms or constructs explanatory variables so that the statistical structure of the problem is represented more effectively.

In business analytics, this work is often the difference between a model that performs well in a controlled experiment and one that remains reliable in production. Poor preparation can create biased samples, distorted distributions, invalid comparisons, data leakage, unstable estimates, and unfair outcomes. This chapter presents a disciplined workflow covering sampling, missing values, encoding, scaling, transformations, feature selection, dimensionality, leakage prevention, class imbalance, and reproducible pipelines.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Construct an analytical dataset from heterogeneous raw sources.
2. Select an appropriate sampling and validation strategy.
3. Diagnose and treat missing values, categorical variables, outliers, and skewed distributions.
4. Choose scaling, transformation, selection, and dimensionality-reduction methods according to model requirements.
5. Prevent target leakage and evaluate class-imbalance interventions correctly.
6. Design reproducible, auditable preprocessing pipelines.
7. Critically assess the ethical and operational consequences of feature engineering decisions.

1. The Analytical Dataset and the Data-Generating Process

An analytical dataset consists of rows representing units of analysis and columns representing variables. A row may represent a customer, transaction, claim, loan application, patient visit, or time period. The unit must be defined before data are joined. Joining customer-level variables to transaction-level records, for example, can duplicate customer attributes and inadvertently change the estimand.

Let X denote the feature matrix with n observations and p predictors, and let y denote the target. In supervised learning, the training sample is commonly represented as D = {(x_i, y_i)}_i=1ⁿ. The objective is not merely to fit the observed sample but to estimate performance on future observations generated by a relevant deployment process.

A useful preparation audit asks:

• What is the unit and time of observation?
• Which variables are known before the prediction decision?
• Are records duplicated or incorrectly joined?
• What population does the sample represent?
• Which fields are identifiers, measures, labels, or metadata?
• Are definitions stable across systems and time?

Data quality dimensions include completeness, validity, consistency, uniqueness, timeliness, and accuracy. These dimensions are related but not interchangeable: a complete dataset can still contain systematically inaccurate values.

2. Sampling, Splitting, and Validation

Sampling determines which observations enter the analytical dataset. A simple random sample gives each population unit an equal selection probability. If subgroup representation matters, stratified sampling divides the population into strata and samples within each. Cluster sampling selects groups such as branches or hospitals, while systematic sampling selects every kth unit after a random start.

For a population of size N, a simple-random-sampling estimate of a proportion has approximate standard error

where p̂ is the sample proportion and n is the sample size. This approximation assumes independent observations and is most appropriate when the sampling design is simple random sampling and the sample is not a substantial fraction of the population. Complex designs may require weights and design-specific variance estimation.

A random train-test split is inappropriate when observations are temporally ordered, grouped, or repeated for the same entity. Time-series problems should use forward-looking validation: train on earlier periods and validate on later periods. Grouped data should keep all observations from an entity in one partition to prevent memorization of entity-specific information. Stratification is useful for classification when preserving class proportions, but it does not solve temporal or group dependence.

The test set should be reserved for final evaluation. Decisions about imputation, transformations, feature selection, and hyperparameters must be made using training data and, where needed, validation data. Repeated cross-validation can estimate variability, but it does not eliminate bias caused by leakage.

3. Missing-Value Treatment

Missingness may be classified as:

• MCAR (missing completely at random): missingness is unrelated to observed and unobserved values.
• MAR (missing at random): missingness depends on observed variables after conditioning.
• MNAR (missing not at random): missingness depends on the unobserved value or an unmeasured process.

These labels concern a data-generating mechanism, not merely the percentage of empty cells. Missingness patterns should be examined by variable, subgroup, time, and outcome status. A missingness indicator can be informative, but it may also encode sensitive operational practices.

Common treatments include deletion, simple imputation, model-based imputation, and explicit “unknown” categories. Mean imputation preserves the sample mean under simple conditions but reduces variance and weakens correlations. Median imputation is more robust to skew and outliers. For a training-set variable with observed values x₁, ..., x_m, the median is preferable when a robust location estimate is desired. Imputation statistics must be fitted on training data only.

Multiple imputation creates several plausible completed datasets and combines estimates, reflecting uncertainty about missing values. It is often more defensible for statistical inference than single imputation, although its assumptions must be justified. In prediction, sophisticated imputation is not automatically superior: it should be evaluated by nested or pipeline-based validation.

4. Categorical Variables and Encoding

Categorical variables may be nominal, ordinal, binary, or high-cardinality. Nominal categories have no natural order; ordinal categories do. One-hot encoding creates a binary indicator for each category. With an intercept, one level is usually omitted to avoid perfect linear dependence in ordinary least squares; this is called reference or treatment coding.

Ordinal encoding is appropriate only when the numeric order represents a meaningful and sufficiently stable order. Assigning values 1, 2, and 3 to “basic,” “standard,” and “premium” imposes equal spacing that may not exist. Target encoding replaces a category with a target-related statistic, but naive target encoding leaks outcome information. It must be computed within training folds, often with smoothing and minimum-count rules.

High-cardinality variables may be handled using hashing, frequency encoding, learned embeddings, grouped categories, or domain-specific aggregation. Each method trades interpretability, memory, variance, and leakage risk. An identifier such as account number is generally not a legitimate explanatory feature unless the modeling purpose explicitly concerns persistent entity effects and deployment conditions support it.

5. Scaling and Transformation

Scaling changes the numerical representation of predictors without necessarily changing their information. Standardization uses training-set mean μ_j and standard deviation σ_j:

Here x_ij is the value of feature j for observation i. The result is dimensionless, with training mean approximately zero and standard deviation approximately one. Standardization is important for distance-based, gradient-based, and regularized models, but usually unnecessary for tree-based methods.

Min–max scaling maps a feature to a selected interval, commonly [0, 1]:

This is sensitive to extreme values. Robust scaling instead uses the median and interquartile range:

where IQR_j = Q_3j − Q_1j. Scaling parameters must be learned from the training partition and then applied unchanged to validation, test, and production data.

Transformations can reduce skewness or represent domain mechanisms. The logarithmic transformation log(1 + x) is useful for non-negative variables with many small values and a long right tail. The Box–Cox transformation for strictly positive x is

The parameter λ is estimated from the training data. The transformation is not defined for non-positive values without an appropriate shift, and arbitrary shifting can impair interpretation. Yeo–Johnson transformations can accommodate zero and negative values.

Outliers must be investigated rather than automatically deleted. They may represent errors, rare but valid events, fraud, or a changing regime. Winsorization, robust transformations, quantile methods, or algorithms resistant to outliers may be appropriate. Any exclusion rule should be documented and applied consistently.

6. Feature Construction and Selection

Feature engineering uses domain knowledge to create variables such as recency, frequency, monetary value, utilization ratio, rolling averages, interaction terms, and lagged measures. A feature must be available at prediction time and must have a stable operational definition. Ratios require attention to zero denominators, units, and whether the denominator is itself post-outcome.

Feature selection reduces noise, computation, and overfitting. Filter methods rank variables independently of a final model, using measures such as variance, correlation, mutual information, or statistical tests. Wrapper methods evaluate subsets through model performance, while embedded methods perform selection during fitting, as with L1 regularization.

For linear regression with coefficient vector β, the lasso objective is commonly written:

Here λ ≥ 0 controls the penalty. Larger values generally increase shrinkage and may set coefficients exactly to zero. The penalty assumes predictors are comparably scaled when coefficient selection is to be interpreted fairly. Correlated predictors can lead lasso to select one variable unpredictably; elastic net combines L1 and L2 penalties and can be more stable for correlated groups.

Selection must occur inside cross-validation when performance estimates are reported. Selecting variables once on the full dataset before cross-validation allows validation information to influence the model and produces optimistic estimates.

7. Dimensionality Reduction

When p is large relative to n, models may suffer from computational burden, unstable estimates, multiple-testing problems, and the curse of dimensionality. More dimensions do not always improve prediction because data become sparse relative to the volume of the feature space.

Principal component analysis (PCA) constructs orthogonal linear combinations of centered variables. If X_c is the centered feature matrix, PCA uses the covariance matrix

and obtains eigenvectors of S. The first principal component is the direction of maximum projected variance, and subsequent components maximize remaining variance subject to orthogonality. PCA is sensitive to scale, so standardization is often needed when variables have different units. High explained variance does not guarantee high predictive value, and components may be difficult to interpret.

Feature selection preserves original variables and interpretation; dimensionality reduction replaces them with a smaller representation. The choice depends on the balance between performance, governance, interpretability, and operational cost.

8. Leakage Prevention and Class Imbalance

Target leakage occurs when a predictor contains information unavailable at the prediction time or information caused by the outcome. Examples include post-default collections status in a loan-default model, discharge diagnosis in an admission-risk model, and a full-period revenue total used to predict an earlier-period purchase.

Leakage can also arise indirectly when preprocessing is fitted before splitting. For example, calculating global means, scaling parameters, selected features, oversampling, or target encodings before cross-validation permits information from validation data to influence training. The remedy is a pipeline that fits every learned transformation within each training fold.

Class imbalance occurs when one class is much less frequent than another. Accuracy can be misleading: predicting the majority class for every case may yield high accuracy while detecting no minority cases. Relevant metrics include precision, recall, specificity, F1 score, area under the precision–recall curve, and cost-weighted expected loss.

For binary classification, precision and recall are

where TP, FP, and FN denote true positives, false positives, and false negatives. The F1 score is the harmonic mean:

These metrics depend on a decision threshold and should be connected to business costs. Resampling methods include random undersampling, oversampling, and synthetic methods such as SMOTE. Synthetic observations must be generated inside the training fold; otherwise, information can cross into validation data. Class weights alter the learning objective and may be preferable when preserving the original data distribution is important.

9. Reproducible Pipelines

A production-grade pipeline explicitly orders operations: data extraction, validation, splitting, imputation, encoding, scaling or transformation, feature construction, selection, model fitting, threshold selection, and monitoring. A column transformer can apply different procedures to numeric and categorical fields while preserving a single fitted artifact.

Reproducibility requires versioned source data or immutable snapshots, fixed random seeds where appropriate, documented software environments, deterministic feature definitions, and stored preprocessing parameters. Reproducibility does not mean that every run must be identical: parallel computation and evolving data may create legitimate variation. It means that the sources, decisions, code, and environment are sufficiently recorded to explain and recreate results.

A pipeline should be tested for schema changes, unseen categories, missing required fields, extreme values, unit changes, and time availability. Monitoring should track feature distributions, missingness, prediction distributions, latency, and outcome-based performance. Population stability indicators or divergence measures can signal drift, but drift is not automatically evidence of degraded performance.

Applied Case: Illustrative Customer Churn Prediction

Background

A telecommunications firm wants to identify customers at high risk of cancelling within the next 30 days so that retention offers can be targeted.

Problem

The raw data contain monthly billing records, service tickets, contract information, usage measures, marketing contacts, and cancellation records. The initial dataset reports 92% accuracy, but the retention team finds few useful alerts.

Analysis

Investigation reveals that cancellation status was joined from a table updated after the cancellation event. A “final account balance” field also includes post-cancellation adjustments. Both variables leak the target. The data are longitudinal, and multiple monthly records belong to each customer. A random row split therefore places the same customer in training and validation sets.

The minority class represents 8% of customer-month observations. Accuracy is inappropriate as the primary metric. The business estimates that missing a genuinely cancellable customer costs three times as much as contacting a non-cancellable customer.

Approach and Solution

The team defines the prediction timestamp and constructs only features available at that time: days since last support contact, rolling 90-day usage, payment regularity, contract tenure, and recent service interruptions. Customers, rather than rows, are grouped during splitting; a later time period is reserved for a final test.

Numeric missing values are imputed using training medians, while missingness indicators are added for variables where absence may be informative. Nominal service-plan fields are one-hot encoded, rare categories are grouped, and numeric variables are standardized for a regularized logistic model. Feature selection is performed within cross-validation. Class weights are evaluated against training-fold oversampling. The decision threshold is chosen using the estimated cost of false negatives and false positives, not the default threshold of 0.5.

Outcome

The revised evaluation shows lower headline accuracy but substantially better recall at an acceptable contact rate. The retention team receives fewer duplicate alerts because customer-level grouping prevents memorization. A monitoring plan is established to detect changes in plan types, usage behavior, and intervention response.

Learning Points

The case demonstrates that valid time alignment, entity-aware splitting, leakage prevention, appropriate metrics, and operational threshold selection are more important than optimizing an isolated accuracy score. It also shows that feature engineering is a governance activity: every feature requires a definition, owner, timestamp, and justification.

Challenges, Ethics, and Governance

Preprocessing choices can reproduce historical discrimination. Imputation may treat groups differently when missingness reflects unequal access or administrative practice. Proxy variables, such as postal location, may encode protected characteristics. Class weighting can improve minority detection but may change the distribution of errors across groups.

Fairness should be assessed using relevant group-level measures, such as error rates, calibration, or selection rates, while recognizing that different criteria can conflict. Sensitive attributes may be needed for auditing even when excluded from prediction. Governance should document data provenance, lawful basis, retention, access controls, feature definitions, model purpose, validation results, and known limitations.

Privacy risks arise when detailed behavioral histories are combined across systems. Data minimization, pseudonymization, access restriction, and secure processing are important, but pseudonymization does not make data anonymous. Regulatory requirements vary by jurisdiction and application; high-impact uses may require impact assessments, human oversight, contestability, and explanations.

Contemporary Developments and Best Practices

Modern feature platforms support reusable definitions, point-in-time correct retrieval, lineage, and shared monitoring. Automated feature engineering can discover transformations and interactions, but automation does not remove the need for leakage checks, domain validation, or fairness review. Learned representations and embeddings are powerful for text, images, and high-cardinality entities, yet they can be opaque and difficult to govern.

A robust preparation workflow follows these principles:

1. Define the unit, target, prediction time, and deployment population.
2. Split data according to time, entity, and sampling design.
3. Fit all learned preprocessing only on training data.
4. Prefer domain-valid features over indiscriminate feature volume.
5. Evaluate transformations and imbalance methods within complete pipelines.
6. Preserve interpretability where decisions affect people.
7. Record provenance, versions, assumptions, and rejection rules.
8. Monitor both data quality and post-deployment model performance.

Chapter Summary

Data preparation establishes whether a machine-learning problem is statistically and operationally well defined. Sampling and splitting must reflect time, entities, and the deployment population. Missing values require consideration of their mechanism and uncertainty. Categorical variables need encoding consistent with their measurement scale, while scaling and transformation should match the algorithm and distributional properties.

Feature engineering can reveal meaningful business structure, but constructed variables must be available at prediction time. Feature selection and PCA address high dimensionality in different ways: selection preserves original variables, whereas PCA creates new combinations. Leakage prevention requires point-in-time reasoning and fold-specific fitting. Imbalance should be addressed with suitable metrics, costs, weights, or training-fold resampling. Finally, reproducible pipelines, documentation, monitoring, and fairness review turn preprocessing from an ad hoc activity into a reliable analytical capability.

Key Terms

Analytical dataset: A structured dataset prepared for statistical or machine-learning analysis.

Sampling frame: The operational list or process from which observations are selected.

Stratified sampling: Sampling separately within defined subgroups.

MCAR: Missing completely at random.

MAR: Missing at random conditional on observed information.

MNAR: Missing not at random.

Imputation: Replacing or modeling missing values.

One-hot encoding: Representing categories with binary indicator variables.

Target encoding: Representing categories using outcome-related statistics.

Standardization: Subtracting a mean and dividing by a standard deviation.

Robust scaling: Scaling using the median and interquartile range.

Feature engineering: Constructing predictors from raw or existing variables.

Feature selection: Choosing a subset of available predictors.

Lasso: Regression with an L1 penalty.

Dimensionality reduction: Representing data with fewer dimensions.

PCA: Principal component analysis.

Data leakage: Use of information unavailable at prediction time or improperly shared across validation boundaries.

Class imbalance: Unequal representation of target classes.

Point-in-time correctness: Ensuring features reflect only information available at the prediction timestamp.

Pipeline: An ordered, reproducible sequence of data transformations and model operations.

Review Questions

Short Answer Questions

1. Why must preprocessing parameters be fitted only on the training data?
2. Distinguish MCAR, MAR, and MNAR missingness.
3. When is ordinal encoding inappropriate?
4. Why can accuracy be misleading under class imbalance?
5. State one difference between feature selection and PCA.

Long Answer Questions

1. Explain how time-based and group-based validation differ from a random row split, and give examples of when each is necessary.
2. Compare mean imputation, multiple imputation, missingness indicators, and explicit unknown categories.
3. Explain standardization, min–max scaling, and robust scaling, including their assumptions and limitations.
4. Discuss target leakage, including direct leakage and leakage introduced through preprocessing.
5. Design a reproducible preparation pipeline for a high-cardinality customer dataset.

Analytical / Application Questions

1. A fraud dataset has 1% fraud cases. Propose an evaluation framework, including metrics, threshold selection, and a suitable validation design.
2. A hospital model uses admission data to predict readmission within 30 days. Identify three possible leakage variables and propose point-in-time-correct alternatives.
3. A sales dataset contains customer, transaction, and monthly records. Explain how you would define the unit of analysis, construct rolling features, split the data, and prevent customer-level leakage.

Multiple Choice Questions with Answers

Question 1

Which preprocessing operation must be fitted separately within each training fold to avoid validation leakage?

A. Computing the global mean for imputation

B. Displaying a confusion matrix

C. Counting the number of rows in the test set

D. Naming the target variable

Correct Answer: A. Computing the global mean for imputation

Question 2

Which encoding is generally most appropriate for a nominal variable with no meaningful order?

A. Ordinal encoding with arbitrary ranks

B. One-hot encoding

C. Replacing categories with target values computed globally

D. Converting categories to dates

Correct Answer: B. One-hot encoding

Question 3

In the standardization formula, what does σ_j represent?

A. The target mean

B. The feature median

C. The training standard deviation of feature j

D. The number of observations

Correct Answer: C. The training standard deviation of feature j

Question 4

Which situation is a clear example of target leakage?

A. Using customer tenure known at application time

B. Using a post-cancellation account adjustment to predict cancellation

C. Using training-fold medians for missing-value treatment

D. Using a pre-event rolling usage average

Correct Answer: B. Using a post-cancellation account adjustment to predict cancellation

Question 5

Why is accuracy often unsuitable as the only metric for a severely imbalanced classification problem?

A. It cannot be calculated for binary outcomes

B. It always penalizes false positives more than false negatives

C. A majority-class classifier can obtain high accuracy while missing the minority class

D. It is independent of class frequencies

Correct Answer: C. A majority-class classifier can obtain high accuracy while missing the minority class

Question 6

Which statement about PCA is correct?

A. It preserves every original feature unchanged

B. It creates orthogonal linear combinations of variables

C. It requires the target to be categorical

D. It is unaffected by feature scale

Correct Answer: B. It creates orthogonal linear combinations of variables

Question 7

If precision is 0.80 and recall is 0.50, what is the F1 score?

A. 0.40

B. 0.50

C. 0.615

D. 1.30

Correct Answer: C. 0.615

Question 8

What is the main reason to keep all records from the same customer in one data partition?

A. To increase the number of variables

B. To prevent entity-specific information from crossing partitions

C. To guarantee a balanced target

D. To eliminate missing values

Correct Answer: B. To prevent entity-specific information from crossing partitions

Question 9

Which method is designed to reduce the influence of extreme values when scaling?

A. Robust scaling using median and IQR

B. One-hot encoding

C. Reference coding

D. Random shuffling of labels

Correct Answer: A. Robust scaling using median and IQR

Question 10

Where should SMOTE-generated observations normally be created during cross-validation?

A. Before the complete dataset is split

B. Only in the final test set

C. Inside each training fold

D. After deployment monitoring

Correct Answer: C. Inside each training fold