Data Preparation and Feature Engineering

Introduction

Machine-learning performance depends not only on the choice of algorithm but also on the quality and representation of the data supplied to it. Real-world data are commonly incomplete, inconsistent, noisy, duplicated, imbalanced, and recorded in forms that are unsuitable for statistical learning. Data preparation converts raw observations into a reliable analytical dataset, while feature engineering constructs informative variables from available evidence.

These activities are not merely preliminary programming tasks. They involve statistical judgment, domain knowledge, ethical decisions, and careful experimental design. An apparently accurate model may be invalid if information from the future or from the test set has leaked into training. Conversely, excessive cleaning or feature selection may remove meaningful signals. The objective is therefore not to make data look perfect, but to create a representation that is valid, useful, reproducible, and appropriate for the intended decision.

Learning Objectives

By the end of this chapter, students should be able to:

• perform exploratory data analysis (EDA) using numerical and visual techniques;
• identify common data-quality problems and select suitable cleaning strategies;
• treat missing values while recognizing the assumptions behind each method;
• encode categorical and other non-numeric variables;
• scale and transform numerical variables appropriately;
• construct, select, and evaluate features;
• design sampling strategies and address class imbalance;
• prevent data leakage through correct train–validation–test procedures; and
• communicate the limitations, risks, and governance implications of a prepared dataset.

1. Data Preparation as a Machine-Learning Process

A dataset consists of observations (rows), variables or features (columns), and often a target variable to be predicted. A feature is an input variable; a label or target is the output used for supervised learning. Data preparation commonly includes profiling, cleaning, splitting, imputing, encoding, transforming, scaling, sampling, and feature construction.

A sound workflow separates two questions:

1. What does the data actually contain?

2. What transformations can be learned from the training data and safely applied to unseen data?

The first question is exploratory. The second is predictive and operational. Confusing them can produce leakage or misleading conclusions.

A typical supervised-learning workflow is:

1. define the prediction task and the unit of analysis;

2. identify the target, prediction time, and permissible information;

3. audit data quality and conduct initial EDA;

4. split data into training, validation, and test sets;

5. fit preprocessing operations using training data only;

6. apply the fitted operations to validation and test data;

7. construct and select features;

8. train, evaluate, and document the complete pipeline.

The test set should be used sparingly for final performance estimation, not repeatedly for design decisions.

2. Exploratory Data Analysis

EDA is the systematic examination of distributions, relationships, structure, and anomalies before modeling. It is not a substitute for statistical modeling, and visual patterns should be treated as hypotheses rather than proof of causation.

2.1 Univariate examination

For numerical variables, useful summaries include count, minimum, maximum, mean, median, quartiles, variance, and standard deviation. The arithmetic mean is

mean = (1/n) Σ x_i,

where x_i is observation i and n is the number of observed values. The sample variance is

s² = [1/(n − 1)] Σ(x_i − x̄)²,

where x̄ is the sample mean. The n − 1 denominator provides an unbiased estimator of population variance under standard random-sampling assumptions.

Histograms, density plots, box plots, and empirical cumulative distributions reveal skewness, multimodality, outliers, and suspicious boundaries. For categorical variables, frequency tables and bar charts show rare levels, dominant categories, and unexpected spelling variants.

2.2 Bivariate and multivariate examination

Scatter plots can reveal nonlinear relationships, clusters, heteroscedasticity, and influential observations. Grouped summaries compare feature distributions across target classes. Correlation measures linear association but does not establish causality and may miss nonlinear relationships. Pearson correlation between X and Y is

r = cov(X,Y)/(s_X s_Y),

provided both variables have nonzero standard deviation. It is sensitive to outliers and should not be interpreted as a complete measure of dependence.

A correlation matrix, pair plot, contingency table, and missingness heat map can expose redundancy and structural patterns. Domain review is essential: a variable recorded after the outcome, such as a hospital discharge code used to predict admission mortality, may be highly predictive but unusable at prediction time.

2.3 Data profiling and quality checks

A data profile should record data types, units, valid ranges, unique-value counts, missingness, duplicate records, and likely identifiers. Examples of quality rules include:

• age must be within a plausible range;
• transaction date must not precede account creation date;
• a product code must belong to the approved catalogue;
• a quantity should not be negative unless returns are represented explicitly;
• a patient identifier should be unique where uniqueness is required.

Quality checks should be automated where possible and retained as part of the data pipeline. A warning is preferable to silent correction.

3. Missing Values

Missingness may result from nonresponse, sensor failure, system migration, privacy restrictions, or a value being genuinely inapplicable. The reason for missingness often matters more than the percentage alone.

Three classical mechanisms are:

• Missing completely at random (MCAR): missingness is unrelated to observed and unobserved values.
• Missing at random (MAR): missingness depends on observed variables, conditional on which it is independent of the unobserved value.
• Missing not at random (MNAR): missingness depends on the unobserved value itself or on unmeasured factors.

These are assumptions about a data-generating process, not properties that can be conclusively proven from the observed dataset.

3.1 Basic treatments

Deletion may be reasonable when missingness is small, plausibly random, and the remaining sample remains representative. Complete-case deletion can introduce bias when missingness is systematic. Dropping a feature may be appropriate when it has excessive missingness, weak utility, or unreliable collection, but the decision should be documented.

Simple imputation replaces missing numerical values with a training-set statistic such as the median, and categorical values with the mode or an explicit “Missing” category. Median imputation is more robust than mean imputation for skewed data. An indicator variable can record whether the original value was missing. However, an indicator is not automatically informative and may encode sensitive operational processes.

More advanced approaches include k-nearest-neighbor imputation, regression imputation, and multiple imputation. These may preserve relationships better but add assumptions, computational cost, and possible instability. In all cases, the imputation rule must be fitted on training data only. A separate imputation model may be required inside each cross-validation fold.

Worked Example: Median Imputation

Suppose training-set salaries are 30, 40, 45, 50, and one missing value, measured in thousands of currency units. The ordered observed values are 30, 40, 45, 50. Their median is (40 + 45)/2 = 42.5. The missing salary is therefore replaced by 42.5 under median imputation. This value is a modeling convention, not evidence that the individual actually earns 42.5 thousand. If the validation set has a different distribution, its missing values must still be replaced using 42.5, not a newly calculated validation median.

4. Cleaning and Managing Data Quality

Cleaning includes correcting format inconsistencies, resolving duplicate records, validating ranges, handling impossible values, and distinguishing errors from legitimate rare events. A negative bank balance may be valid, whereas a negative person’s age is generally an error. Outliers should not be removed automatically: they may indicate fraud, a new customer segment, a measurement problem, or a genuine extreme case.

Common strategies include correcting from a trusted source, capping values using domain-approved limits, transforming skewed variables, retaining an outlier flag, or excluding records with a documented reason. Every intervention should be traceable, reproducible, and reversible where feasible.

Duplicate handling requires understanding the unit of analysis. Two transactions by the same customer are not duplicates merely because their customer identifier matches. Conversely, repeated ingestion of the same transaction may create artificial evidence. Time stamps, source-system identifiers, and business keys help distinguish these cases.

5. Encoding Categorical and Structured Variables

Most algorithms require numerical input, but numeric coding can falsely imply order. For a nominal variable such as payment method with levels card, cash, and transfer, one-hot encoding creates binary indicators. With an intercept in a linear model, one category is commonly omitted to avoid perfect multicollinearity; this is called reference or dummy coding. Tree-based models may use alternative encodings, but the choice remains model-dependent.

Ordinal encoding is appropriate only when categories have a meaningful order, such as bronze, silver, and gold. The assigned numerical values should reflect order, not necessarily equal distances.

High-cardinality variables, such as thousands of product or postal codes, can create a very wide sparse matrix. Alternatives include grouping rare levels, hashing, learned embeddings, or target encoding. Target encoding replaces a category with a statistic related to the target, but it is highly vulnerable to leakage. It must be computed within training folds, often with smoothing and out-of-fold procedures.

Text can be represented using token counts, term frequency–inverse document frequency, or learned vector representations. Dates can be decomposed into year, month, weekday, hour, elapsed time, or cyclical components. For cyclical variables, sine and cosine representations preserve the fact that the end and beginning of a cycle are close, although the precise transformation depends on the period and interpretation.

6. Scaling and Transformation

Scaling is especially important for distance-based methods, nearest neighbors, support-vector machines, neural networks, and regularized linear models. It is usually less important for decision trees and their ensembles because tree splits depend on order rather than magnitude.

Standardization transforms a value x to

z = (x − μ)/σ,

where μ is the training-set mean and σ is the training-set standard deviation, assumed nonzero. The transformed variable has mean approximately zero and standard deviation approximately one on the fitting data. Standardization is sensitive to extreme values.

Min–max scaling maps x to

x' = (x − a)/(b − a),

where a and b are the training-set minimum and maximum, with b > a. A further multiplication and addition can map values to any desired interval. This method is also sensitive to future values outside the training range.

Robust scaling uses the median and interquartile range (IQR):

x' = (x − median)/(Q_3 − Q_1),

provided the IQR is nonzero. It is often useful for heavily skewed variables with outliers.

Logarithmic or power transformations can reduce right skew and stabilize variance. For nonnegative x, log(1+x) handles zeros; for strictly positive x, log(x) is conventional. Transformations alter interpretation and should be chosen with domain awareness. A transformation is not a license to conceal data-quality errors.

7. Feature Construction and Selection

Feature engineering creates variables that express useful structure more directly than raw fields. Examples include:

• total spending from transaction amounts;
• tenure from account-open and prediction dates;
• body-mass index from weight and height;
• ratios such as debt-to-income;
• rolling averages and counts over a preceding time window;
• interaction terms such as price multiplied by promotion status;
• geographic distance from latitude and longitude;
• lagged variables in time-series forecasting.

Feature construction must respect the prediction point. A rolling average may use only observations available before that point. Randomly shuffling time-dependent observations can allow future information into training.

Feature selection reduces irrelevant, redundant, or unstable variables. Filter methods use properties such as variance, correlation, or statistical association; wrapper methods evaluate subsets using model performance; embedded methods perform selection during training, for example through regularization or tree-based importance. Selection should occur inside cross-validation when it uses the target or affects model fitting. Removing a feature solely because it correlates with another can be harmful if the feature is more reliable, less expensive, or more available at deployment.

Interpretability, operational cost, latency, privacy, and fairness are legitimate selection criteria in addition to predictive accuracy.

8. Sampling and Class Imbalance

Sampling determines which observations enter training and how they represent the deployment population. Simple random sampling gives each eligible record an equal probability of selection. Stratified sampling preserves proportions of important groups, often target classes. In a time-based problem, chronological splitting is usually more valid than random splitting. Grouped splitting prevents members of the same person, household, or organization from appearing across both training and test sets.

Class imbalance occurs when one outcome is much less frequent than another. Accuracy can then be misleading. For example, if fraud represents 1% of transactions, a classifier that always predicts “not fraud” achieves 99% accuracy but detects no fraud.

Useful responses include class-weighted loss, threshold adjustment, stratified sampling, under-sampling the majority class, and over-sampling the minority class. Synthetic methods such as SMOTE create interpolated minority examples, but they should be applied only to training data and may be unsuitable for categorical, time-dependent, or highly structured data.

Evaluation should include confusion-matrix measures. For positive-class predictions:

precision = TP/(TP + FP),

recall = TP/(TP + FN),

where TP is true positives, FP false positives, and FN false negatives. Precision answers “of predicted positives, how many were positive?” Recall answers “of actual positives, how many were found?” The F1 score is the harmonic mean,

F1 = 2(precision × recall)/(precision + recall),

when both precision and recall are defined. Threshold selection should reflect the cost of errors, capacity constraints, and affected stakeholders. Precision–recall curves are often more informative than accuracy for rare positive classes.

9. Preventing Data Leakage

Data leakage occurs when information unavailable at prediction time influences model training or evaluation. It can be direct, such as using the target as a feature, or indirect, such as calculating a global mean before splitting the data.

Common examples include:

• imputing or scaling using the complete dataset before splitting;
• selecting features using all labels before cross-validation;
• oversampling before the train–test split;
• using post-outcome variables;
• allowing records from the same person into both training and test sets;
• using future observations in a time-series feature.

The remedy is to fit every data-dependent transformation on training data only. A pipeline should encapsulate preprocessing and modeling so that each cross-validation fold independently learns imputation, encoding, scaling, feature selection, and resampling. The final pipeline is refitted on all permitted training data and then evaluated once on the untouched test set.

Leakage is both a statistical and governance problem. It can cause inflated performance estimates, unsafe deployment, wasted resources, and unjustified trust in automated decisions.

10. Illustrative Case: Predicting Loan Default

Background

A lender wants to estimate whether an application will default within twelve months. Available fields include application date, income, debt, employment history, credit accounts, region, and a later field called “collections status.”

Problem and analysis

Initial profiling finds missing income values, inconsistent employment units, duplicate applications, and a default rate of 4%. The collections status is strongly associated with default, but it is recorded after collection activity begins and is therefore unavailable at application time.

Approach

The team first defines the prediction point as application submission. It removes post-submission fields, deduplicates applications using the source application identifier, and validates that income and debt use consistent units. It creates debt-to-income ratio, employment tenure, number of prior accounts, and a count of applications in the preceding 90 days. These features use only information available at submission.

The data are divided chronologically: earlier applications for training, a later period for validation, and the most recent period for testing. Within the training process, numerical missing values receive training-set median imputation and missing indicators; categorical fields receive one-hot encoding with a controlled rare-category policy. A class-weighted model and a model trained with training-only oversampling are compared. Evaluation emphasizes recall, precision, calibration, and performance across relevant demographic groups rather than accuracy alone.

Outcome

The leakage-free model performs less impressively than the original prototype but is credible under production conditions. Error analysis reveals that missing income is more common in particular regions. The lender therefore investigates the application process rather than treating missingness as merely a technical nuisance. Deployment includes threshold review, monitoring for drift, documentation of permissible variables, and human appeal procedures.

Learning points

The case demonstrates that feature engineering is constrained by time, that class imbalance changes evaluation priorities, and that data-quality patterns may reflect institutional processes. A lower but valid estimate is more valuable than a higher estimate produced through leakage.

11. Practical, Ethical, and Governance Considerations

Prepared data may contain personal, financial, health, or behavioral information. Data minimization limits collection to what is necessary. Access controls, retention policies, encryption, audit trails, and appropriate de-identification reduce risk, although pseudonymization does not eliminate re-identification risk.

Features can act as proxies for protected characteristics even when protected attributes are removed. Fairness analysis should examine error rates, calibration, access, and subgroup performance in the relevant context. There is no universally best fairness metric; trade-offs may be unavoidable. Documentation should record data sources, transformation logic, missingness decisions, training period, feature availability at prediction time, and known limitations.

Data drift occurs when input distributions change; concept drift occurs when the relationship between inputs and the target changes. Production systems require monitoring of data quality, missingness, feature distributions, class rates, performance where labels become available, and subgroup outcomes. Reproducible pipelines and versioned datasets make investigation possible.

12. Best-Practice Workflow

A professional workflow should:

1. define the target, unit, prediction time, and success criteria;

2. profile data and document provenance;

3. inspect distributions, relationships, missingness, duplicates, and validity rules;

4. split data according to time, groups, or population structure;

5. use pipeline-based preprocessing fitted only on training data;

6. make transformations appropriate to the algorithm and domain;

7. address imbalance using training-only methods and suitable metrics;

8. test subgroup performance, robustness, and sensitivity to assumptions;

9. preserve an untouched final test set;

10. document decisions and monitor the deployed system.

Chapter Summary

Data preparation converts imperfect raw data into a representation suitable for reliable learning. Exploratory data analysis identifies distributions, relationships, anomalies, and data-quality risks. Missing values require attention to their generating mechanisms; deletion, simple imputation, indicators, and model-based methods each have assumptions. Categorical data may require one-hot, ordinal, hashing, target, or text-specific representations. Scaling and transformation should match the algorithm, distribution, and operational meaning of the variables.

Feature engineering uses domain and temporal knowledge to construct informative variables, while feature selection balances predictive value with cost, stability, interpretability, privacy, and fairness. Sampling must respect classes, groups, and time. Imbalance requires metrics and training procedures beyond accuracy. Most importantly, all learned preprocessing and resampling must be fitted within the training data and cross-validation structure. Ethical governance, documentation, drift monitoring, and reproducibility are integral parts of data preparation rather than optional additions.

Key Terms

Feature: An input variable used by a machine-learning model.

Target: The outcome a supervised model is trained to predict.

Exploratory data analysis: Systematic examination of data structure, distributions, relationships, and anomalies.

Data leakage: Use of information unavailable at prediction time in training or evaluation.

Imputation: Replacement of missing values using a defined rule or model.

MCAR: Missing completely at random.

MAR: Missing at random conditional on observed variables.

MNAR: Missing not at random because missingness depends on unobserved factors or the value itself.

One-hot encoding: Representation of categorical levels as binary indicator variables.

Ordinal encoding: Numerical representation of categories that possess meaningful order.

Standardization: Transformation using the training mean and standard deviation.

Min–max scaling: Mapping values using training-set minimum and maximum.

IQR: Interquartile range, equal to the third quartile minus the first quartile.

Feature engineering: Construction or transformation of variables to represent useful predictive information.

Feature selection: Choosing a subset of available variables for modeling.

Class imbalance: Unequal representation of outcome classes.

Precision: Proportion of predicted positives that are actually positive.

Recall: Proportion of actual positives correctly identified.

Data drift: Change in the distribution of model inputs over time.

Concept drift: Change in the relationship between inputs and the target.

Review Questions

Short Answer Questions

1. Why should preprocessing transformations be fitted using training data only?

2. Distinguish MCAR, MAR, and MNAR missingness.

3. When is ordinal encoding preferable to one-hot encoding?

4. Why can accuracy be misleading for a highly imbalanced classification problem?

5. What is the difference between precision and recall?

Long Answer Questions

1. Explain a complete leakage-resistant workflow for preparing data for supervised learning.

2. Compare mean, median, indicator-based, and model-based missing-value treatments, including their assumptions and limitations.

3. Discuss how scaling and transformation choices depend on the learning algorithm and the distribution of a variable.

4. Explain several feature-engineering strategies for transactional or time-dependent data and the temporal restrictions they must satisfy.

5. Discuss the ethical, governance, and monitoring responsibilities associated with data preparation in a high-stakes application.

Analytical / Application Questions

1. A fraud dataset contains 2,000 transactions, of which 40 are fraudulent. A classifier identifies 30 of the fraudulent transactions, but also incorrectly flags 90 legitimate transactions. Calculate precision and recall, and explain which operational concern each metric represents.

2. A team computes the global average income before splitting data, imputes missing values with that average, and then evaluates on a test set. Identify the methodological problem and redesign the procedure.

3. A hospital model uses a “discharge diagnosis” field to predict whether a patient will require intensive care at admission. Analyze why this feature is problematic and propose two permissible alternatives based on information available at admission.

Multiple Choice Questions with Answers

Question 1

Which action best prevents preprocessing leakage during cross-validation?

A. Fit preprocessing once on the complete dataset

B. Fit preprocessing separately within each training fold

C. Fit preprocessing on the test set only

D. Avoid preprocessing altogether

Correct Answer: B. Fit preprocessing separately within each training fold

Question 2

Which statistic is generally most robust to extreme values when imputing a skewed numerical variable?

A. Mean

B. Median

C. Maximum

D. Range

Correct Answer: B. Median

Question 3

Which encoding is most appropriate for categories such as bronze, silver, and gold when the order is meaningful?

A. Ordinal encoding

B. Random encoding

C. Hashing without order

D. Target leakage encoding

Correct Answer: A. Ordinal encoding

Question 4

A model predicts 30 fraudulent transactions correctly and makes 90 false-positive predictions. What is its precision?

A. 20%

B. 25%

C. 30%

D. 75%

Correct Answer: B. 25%

Question 5

Which feature is most likely to cause temporal leakage in a loan-application model?

A. Income reported at application

B. Debt recorded at application

C. Collections status recorded after application

D. Employment tenure at application

Correct Answer: C. Collections status recorded after application

Question 6

Which algorithm is generally least sensitive to feature scale?

A. Nearest-neighbor classifier

B. Support-vector machine with an RBF kernel

C. Decision tree

D. Regularized linear regression

Correct Answer: C. Decision tree

Question 7

Recall is calculated as:

A. TP/(TP + FP)

B. TP/(TP + FN)

C. TN/(TN + FP)

D. (TP + TN)/(TP + TN + FP + FN)

Correct Answer: B. TP/(TP + FN)

Question 8

What is a major reason for using grouped splitting?

A. To guarantee equal feature means

B. To prevent related entities appearing in both training and test sets

C. To remove all missing values

D. To increase the number of target classes

Correct Answer: B. To prevent related entities appearing in both training and test sets

Question 9

Which procedure is appropriate for handling class imbalance?

A. Apply oversampling to the entire dataset before splitting

B. Evaluate only with accuracy

C. Apply oversampling within training folds and assess recall and precision

D. Delete all majority-class observations

Correct Answer: C. Apply oversampling within training folds and assess recall and precision

Question 10

Which statement about target encoding is correct?

A. It cannot leak target information

B. It should be calculated using test labels

C. It must be constructed within training folds to reduce leakage

D. It is identical to one-hot encoding

Correct Answer: C. It must be constructed within training folds to reduce leakage