Data Preparation and Feature Engineering

Introduction

Data preparation is the process of converting raw, heterogeneous, and potentially unreliable data into an analytical dataset that is suitable for statistical modelling, machine learning, reporting, or decision support. Feature engineering extends this process by creating informative variables from existing data. In practice, model performance is often constrained less by the choice of algorithm than by the quality, relevance, and integrity of the prepared data.

Raw data commonly contains missing values, duplicate records, inconsistent units, invalid codes, outliers, measurement errors, categorical variables, skewed distributions, and information that would not have been available at the time a prediction was made. Preparation must therefore be treated as a disciplined, auditable stage of the analytical life cycle rather than as informal spreadsheet manipulation.

A central principle is that preparation must respect the analytical objective. A transformation that is appropriate for descriptive analysis may be inappropriate for prediction. Similarly, a method that improves accuracy may reduce interpretability or introduce fairness concerns. A professional data-preparation workflow balances statistical validity, operational feasibility, reproducibility, computational efficiency, and governance.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Assess data quality and construct a reliable analytical dataset.

2. Diagnose and treat missing values, duplicates, inconsistent formats, and outliers.

3. Select suitable transformations for numerical, categorical, temporal, and text data.

4. Explain and prevent target leakage and other forms of information leakage.

5. Encode categorical variables and scale numerical features appropriately.

6. Address class imbalance using principled sampling, weighting, and evaluation methods.

7. Construct derived features and reduce dimensionality when justified.

8. Design a reproducible, versioned, and auditable preparation pipeline.

1. Data Preparation as an Analytical Design Problem

A dataset can be represented as a matrix X with n observations and p variables, together with an outcome vector y when supervised learning is used. The rows must represent comparable analytical units, such as customers, transactions, patients, or machines. The columns must have clearly defined meanings, units, time references, and measurement procedures.

Before cleaning, the analyst should answer five questions:

• What is the unit of analysis?
• What population and time period does the data represent?
• What is the target or decision variable?
• Which variables would genuinely be available at prediction or decision time?
• What errors or biases could have entered during collection?

These questions prevent technically sophisticated but conceptually invalid analysis. For example, joining customer-level demographic data to transaction-level records without controlling the grain can duplicate transactions and distort aggregate features.

Data quality dimensions

Important dimensions include:

• Completeness: the extent to which required values are present.
• Validity: whether values conform to permitted formats or ranges.
• Accuracy: whether values represent the real-world quantity correctly.
• Consistency: whether the same concept is represented uniformly across sources.
• Uniqueness: whether records are duplicated inappropriately.
• Timeliness: whether data is sufficiently current for the decision.
• Integrity: whether relationships between tables and entities are preserved.

A data dictionary should record variable name, business definition, data type, unit, valid range, source, collection time, missing-value code, and permitted transformations. This documentation is especially important in regulated or collaborative environments.

2. Profiling, Integration, and Dataset Construction

Data profiling begins with structural inspection: row counts, column types, distinct-value counts, missingness, quantiles, frequency tables, and duplicate keys. Visual tools such as histograms, box plots, missingness maps, and scatter plots help reveal patterns that summary statistics can conceal.

Combining data sources

Integration may involve joins, unions, aggregation, or temporal alignment. A join should be checked for cardinality: one-to-one, one-to-many, or many-to-many. Unexpected row multiplication is a common source of silent error. After integration, row counts, key uniqueness, aggregate totals, and representative records should be compared with source systems.

Units must be standardized before combination. For example, converting all monetary amounts to a common currency and reference date is necessary before calculating ratios. Date and time fields require attention to time zones, daylight-saving changes, and whether a timestamp describes event time, processing time, or reporting time.

Train, validation, and test partitions

For supervised learning, data should generally be divided into training, validation, and test sets. The training set is used to estimate model parameters and preprocessing parameters. The validation set supports model selection and hyperparameter tuning. The test set is reserved for final evaluation.

Random partitioning is unsuitable when observations are temporally ordered, clustered, or related. Time-based forecasting should use earlier observations for training and later observations for evaluation. Grouped data should prevent members of the same person, household, or machine from appearing in both training and test sets when that would inflate performance.

All learned preprocessing operations must be fitted using training data only. If a mean, median, vocabulary, scaling parameter, imputation rule, or principal-component basis is calculated using the full dataset before splitting, information from validation or test observations has entered the training process. This is leakage.

3. Missing Data

Missingness may arise from nonresponse, sensor failure, data-entry error, system changes, or the fact that a measurement was not applicable. A missing value is not automatically zero, false, or the average.

Three classical mechanisms are useful:

• Missing completely at random (MCAR): missingness is unrelated to observed and unobserved values.
• Missing at random (MAR): missingness depends on observed variables but not, after conditioning on them, on the missing value itself.
• Missing not at random (MNAR): missingness depends on the unobserved value or an unmeasured process.

These are assumptions about a missingness mechanism, not labels that can usually be proven from the observed dataset alone.

Treatment strategies

Deletion is reasonable when missingness is small, plausibly MCAR, and deletion does not materially reduce representativeness. Listwise deletion removes an entire row with any missing field; pairwise deletion uses available values for particular calculations but can produce inconsistent sample sizes.

Simple imputation replaces missing numerical values with a training-set mean or median and categorical values with a mode or an explicit “unknown” category. The median is more robust than the mean when the variable is skewed. An indicator variable can record whether imputation occurred, particularly when missingness itself may carry information.

More advanced methods include regression imputation, k-nearest-neighbour imputation, multiple imputation, and model-based iterative imputation. Multiple imputation creates several plausible completed datasets, analyses each, and combines estimates, thereby reflecting imputation uncertainty. It is often preferable for inferential analysis but requires stronger assumptions and more careful implementation than simple predictive imputation.

Imputation must be performed within each training fold during cross-validation. Imputing before cross-validation allows validation observations to influence the imputation parameters. For MNAR settings, sensitivity analysis, explicit missingness modelling, or collection of additional data may be necessary.

4. Duplicates, Errors, and Outliers

Duplicates can be exact, near-duplicates, or legitimate repeated events. Removing every repeated row is unsafe. The analyst must determine whether duplication represents a repeated transaction, a system replay, or an erroneous record. Entity resolution may require standardized names, addresses, identifiers, and probabilistic matching.

Validation rules should identify impossible values, such as negative age, an end date before a start date, or a percentage outside its permitted range. An invalid value should not be silently converted into a plausible value; it should be corrected from a trusted source, treated as missing, or excluded according to documented rules.

Outliers

An outlier is an observation that differs substantially from the rest of the data. It may be a data error, a rare but valid event, or evidence of a different population. Its treatment depends on cause and analytical purpose.

For a numerical variable x, the interquartile range is IQR = Q3 − Q1, where Q1 and Q3 are the first and third quartiles. A common exploratory rule flags values below Q1 − 1.5 IQR or above Q3 + 1.5 IQR. This is a screening rule, not proof that a value is erroneous. For approximately normal data, a standardized score can be calculated as z = (x − μ)/σ, where μ is the mean and σ is the standard deviation; large absolute values may warrant investigation, but normality and independence assumptions matter.

Possible responses include correction, removal with justification, transformation, winsorization, robust scaling, use of robust models, or retaining the observation. Winsorization replaces values beyond selected quantiles with boundary values. It reduces the influence of extremes but changes the data-generating distribution and should be documented. In financial fraud or safety analysis, extreme values may be the most important observations and must not be discarded merely because they are unusual.

5. Transforming Numerical Variables

Transformations can improve interpretability, stabilize variance, reduce skewness, and make relationships easier for some models to learn.

Common transformations include:

• Log transformation, often log(1+x) when zero values occur, for right-skewed non-negative measurements.
• Square-root transformation for count-like variables.
• Reciprocal transformation for certain inverse relationships.
• Power transformations such as Box–Cox, which requires strictly positive values, and Yeo–Johnson, which can accommodate zero and negative values.
• Binning, which converts a continuous variable into intervals, although it can discard information.

A transformation should be selected based on distribution, domain meaning, model assumptions, and interpretability. Linear regression may benefit from transformations that improve linearity or constant variance; tree-based models generally require less scaling and are less sensitive to monotonic transformations.

Feature construction often uses ratios, differences, rates, rolling summaries, interaction terms, and domain-specific indicators. A ratio such as revenue per employee may be meaningful only when the denominator is valid and nonzero. Every engineered feature requires a clear definition and a statement of its time reference.

6. Categorical and Other Non-numerical Variables

Categorical variables may be nominal, ordinal, or binary. Nominal categories have no natural order, whereas ordinal categories do. Treating nominal labels as integers can create a false ordering.

One-hot encoding creates one binary feature for each category. With an intercept in a linear model, one category is commonly omitted to avoid perfect multicollinearity; alternatively, a model without an intercept or a regularized implementation can use all indicators. For a new category at deployment, an explicit unknown category or a safe “ignore” policy is required.

Ordinal encoding is appropriate only when order is meaningful and the assigned numerical distances are defensible. Frequency or target encoding can be useful for high-cardinality variables, but target encoding is especially vulnerable to leakage. It should be calculated within training folds, with smoothing and careful handling of rare categories.

Hashing provides a fixed-dimensional representation for very high-cardinality categories but may cause collisions. Embeddings represent categories as learned vectors and are widely used in neural networks, but they require sufficient data and introduce additional modelling complexity.

Text may be represented using token counts, term frequency–inverse document frequency, or learned embeddings. In a corpus containing N documents, if term t occurs in d(t) documents, a common inverse document frequency form is idf(t) = log(N/d(t)); exact variants differ by implementation and smoothing. Text preprocessing must consider language, domain terminology, negation, privacy, and possible leakage through identifiers or post-outcome text.

7. Scaling and Normalization

Scaling is important for distance-based, gradient-based, and regularized algorithms. It is usually less important for decision trees and their ensembles.

Standardization uses

z = (x − μ_train) / σ_train,

where μ_train and σ_train are the training-set mean and standard deviation. The transformed variable has mean approximately zero and standard deviation approximately one on the training data, provided σ_train is nonzero. The same training parameters must be applied to validation, test, and production data.

Min–max scaling maps a variable to a selected interval, commonly [0,1]:

x' = (x − xmin,train)/(xmax,train − xmin,train).

It is sensitive to extreme values and may produce values outside [0,1] for future observations beyond the training range. Robust scaling uses the median and IQR and is less influenced by outliers. Normalization of a vector, such as dividing by its Euclidean norm, is distinct from standardizing each feature and is often used in text or similarity applications.

8. Class Imbalance

Class imbalance occurs when one outcome is much more frequent than another. Accuracy can then be misleading. If 98% of observations are negative, a classifier that always predicts negative has 98% accuracy but no ability to identify positives.

Useful metrics include precision = TP/(TP+FP), recall or sensitivity = TP/(TP+FN), and specificity = TN/(TN+FP), where TP, FP, TN, and FN are the four confusion-matrix counts. The F1 score is the harmonic mean of precision and recall:

F1 = 2(precision × recall)/(precision + recall),

provided both terms are defined. The choice of metric should reflect the cost of false positives and false negatives. Precision–recall curves are often more informative than ROC curves for highly rare positive classes.

Approaches include class-weighted loss, random oversampling, random undersampling, and synthetic methods such as SMOTE. SMOTE creates interpolated minority examples, but it can generate unrealistic points and should not be applied before data splitting. Resampling must occur inside training folds only. Threshold adjustment is another valuable option: the default probability threshold of 0.5 is not universally optimal.

Imbalance may reflect social or operational processes rather than a purely statistical nuisance. Sampling and threshold decisions can alter who receives intervention, approval, or investigation, so fairness and cost analysis are required.

9. Leakage, Bias, and Fairness

Target leakage occurs when a feature contains information that would not be available when the prediction is made. Examples include a discharge diagnosis used to predict admission risk, a post-approval repayment status used in a credit model, or a feature calculated using the full outcome-labelled dataset.

Leakage can also result from duplicate entities across partitions, temporal misordering, preprocessing fitted on all data, or feature selection using the test set. It often produces excellent offline performance and poor deployment performance.

A leakage audit should ask:

• Was the feature available at decision time?
• Does its creation use future records or the target?
• Can the same entity occur across partitions?
• Were all preprocessing and selection steps fitted only on training data?
• Does the evaluation reproduce the production information flow?

Bias may enter through nonrepresentative sampling, historical discrimination, measurement error, or unequal missingness. Removing protected attributes does not necessarily remove discrimination because proxy variables may remain. Fairness assessment should consider the application context, relevant protected groups, error rates, calibration, and legal requirements. No single fairness metric is universally correct because some criteria conflict mathematically when base rates differ.

10. Dimensionality Reduction and Feature Selection

High-dimensional data can increase computation, noise, multicollinearity, and overfitting. Feature selection retains a subset of variables; dimensionality reduction constructs a smaller representation.

Principal component analysis (PCA) transforms centred variables into orthogonal directions of decreasing variance. If the centred data matrix is X, PCA can be obtained from the eigenvectors of the covariance matrix or from singular value decomposition. The first component is the direction capturing maximum projected variance, the second captures maximum remaining variance subject to orthogonality, and so on. PCA is sensitive to scale, so standardization is often appropriate when variables have different units. Components can be difficult to interpret and high variance does not necessarily mean high predictive value.

Feature selection methods include:

• Filter methods, such as variance thresholds, correlation screening, and mutual information.
• Wrapper methods, such as recursive feature elimination.
• Embedded methods, such as L1-regularized regression or tree-based importance.

Selection must be nested within cross-validation when performance is being estimated. Selecting features using the full dataset before cross-validation leaks information from validation folds.

11. Reproducible Preparation Pipelines

A pipeline expresses preparation as an ordered, executable sequence. A typical pipeline may include schema validation, type conversion, train-test splitting, imputation, encoding, scaling, feature construction, selection, and model fitting. Different column types can be processed through separate branches and then combined.

Reproducibility requires versioned source data or snapshots, fixed random seeds where appropriate, explicit software environments, documented parameters, code review, automated tests, and stored metadata. A feature catalogue should record definitions, owners, lineage, freshness, permitted use, and known limitations.

Unit tests should check such properties as row-count preservation, key uniqueness, nonnegative durations, absence of future information, stable category handling, and consistent output schema. Data-drift monitoring compares production distributions with reference data. Drift does not automatically imply performance degradation, but it is a signal for investigation and possible retraining.

A mature architecture separates raw, validated, transformed, and analytical layers. Raw data should be preserved rather than overwritten. Transformations should be idempotent where possible: rerunning the same process on the same input should produce the same result. Access controls, encryption, retention rules, anonymisation or pseudonymisation, and audit logs are necessary when personal or sensitive data is involved.

12. Illustrative Case: Predicting Hospital Readmission

Background

A hospital wants to identify patients at elevated risk of readmission within 30 days of discharge so that follow-up support can be offered.

Problem

The source data contains demographics, diagnoses, laboratory measurements, medication records, admission dates, discharge dates, and administrative events. Missing laboratory results are common, several diagnosis codes are inconsistent, and readmission is relatively uncommon.

Analysis

The unit of analysis is the discharge episode. A patient may have multiple episodes, so a time-aware design is required. The team identifies a serious leakage risk: a post-discharge appointment cancellation had been included as a predictor, although it occurred after the prediction time. Another feature, “number of readmissions,” had been calculated using records extending beyond discharge.

Approach and solution

The team defines an explicit prediction time at discharge. Records are split chronologically, with earlier episodes used for training and later episodes for testing. Patient-level grouping prevents inappropriate overlap in a supplementary analysis. Laboratory values are imputed using training-fold medians, with missingness indicators. Diagnosis codes are standardized and rare categories grouped into an unknown or other category. Numerical variables are robustly scaled for a regularized logistic regression model.

Because the positive class is uncommon, the team uses class-weighted loss and evaluates recall, precision, area under the precision–recall curve, calibration, and subgroup error rates. It does not apply synthetic oversampling to the test data. A threshold is selected using the operational capacity of the follow-up team rather than defaulting to 0.5.

Outcome

The revised evaluation is less optimistic than the original result but remains operationally useful. The model identifies a manageable number of high-risk patients, and monitoring is established for changes in case mix and subgroup performance. Clinicians receive explanations based on clinically meaningful features rather than an unexplained risk score.

Learning points

The case demonstrates that defining the prediction moment, preventing leakage, respecting time, handling missingness within folds, choosing suitable metrics, and linking thresholds to operational capacity are more important than simply selecting a more complex algorithm. It also shows that data preparation is inseparable from governance and clinical accountability.

13. Contemporary Developments and Critical Considerations

Modern feature stores support shared, versioned features, but they can also propagate a flawed definition across many models. Automated feature engineering can discover interactions and transformations rapidly, yet it may produce unstable, opaque, or leakage-prone variables. Large language models and embedding systems enable powerful representations of text, images, and multimodal records, but they introduce concerns about privacy, memorisation, bias, licensing, and reproducibility.

Self-supervised learning can use unlabelled data to learn representations before task-specific training. Synthetic data may support development where access to sensitive data is restricted, but synthetic records do not automatically guarantee privacy or preserve minority patterns. Differential privacy provides formal privacy guarantees under specified mechanisms and privacy budgets, but it can reduce utility and requires specialist design.

The professional standard is not to use the newest technique automatically. A simpler, transparent, well-governed feature set may be preferable when the cost of error, explanation, auditability, or maintenance is high.

Best-Practice Checklist

1. Define the unit of analysis and prediction time.

2. Profile structure, quality, missingness, and relationships before modelling.

3. Preserve raw data and document every transformation.

4. Split data appropriately for time, groups, and deployment conditions.

5. Fit preprocessing only on training data and within cross-validation folds.

6. Investigate outliers rather than deleting them automatically.

7. Treat categories according to their measurement scale and deployment behaviour.

8. Select metrics and thresholds according to decision costs.

9. Audit leakage, fairness, privacy, and security.

10. Version data, code, features, parameters, and environments.

11. Test pipelines and monitor drift after deployment.

12. Prefer the least complex preparation that meets the analytical objective.

Chapter Summary

Data preparation converts raw observations into a valid analytical representation. It begins with the unit of analysis, data provenance, quality profiling, and careful integration. Missing values require assumptions about their mechanism and must be handled without allowing validation or test information to influence training. Duplicates, invalid values, and outliers require investigation based on domain meaning rather than mechanical rules.

Feature engineering includes transformations, domain-based variables, categorical encoding, text representation, scaling, and dimensionality reduction. These operations must be selected according to model requirements, interpretability, and deployment constraints. Class imbalance should be addressed through appropriate metrics, weighting, resampling within training data, and threshold decisions. Leakage, bias, and unfairness can invalidate apparently accurate models and create harmful decisions.

A reproducible pipeline uses explicit definitions, versioning, testing, lineage, access controls, and monitoring. In professional analytics, preparation is not a preliminary clerical task; it is a methodological, engineering, and governance activity that determines whether conclusions and predictions can be trusted.

Key Terms

Data profiling: Systematic examination of structure, quality, distributions, and relationships in data.

Unit of analysis: The entity or event represented by one row of an analytical dataset.

Data leakage: Use of information in model development that would not be available in the intended prediction setting.

Missing completely at random: Missingness unrelated to observed and unobserved values.

Missing at random: Missingness explainable by observed variables after conditioning on them.

Imputation: Replacement of missing values using a defined statistical or domain-based procedure.

Outlier: An observation unusually distant from the distribution of other observations.

Winsorization: Replacement of extreme values with selected boundary values.

Feature engineering: Creation or transformation of variables to improve analysis or prediction.

One-hot encoding: Representation of a categorical variable using binary indicator columns.

Target encoding: Encoding categories using outcome-related statistics, requiring leakage controls.

Standardization: Transformation using a training mean and standard deviation.

Robust scaling: Scaling based on statistics such as the median and interquartile range.

Class imbalance: Unequal representation of outcome classes.

Precision: The proportion of predicted positives that are actually positive.

Recall: The proportion of actual positives that are correctly identified.

Principal component analysis: An orthogonal transformation into directions of decreasing variance.

Feature selection: Choosing a subset of available variables.

Data drift: Change in data distributions or relationships after deployment.

Data lineage: Record of a feature’s sources, transformations, and dependencies.

Review Questions

Short Answer Questions

1. Why must the unit of analysis be defined before joining data sources?

2. Distinguish missing completely at random from missing not at random.

3. Why should scaling parameters be estimated using training data only?

4. Give two examples of target leakage.

5. When is one-hot encoding preferable to ordinal encoding?

Long Answer Questions

1. Explain a complete workflow for handling missing values in a supervised-learning project, including validation and leakage controls.

2. Compare deletion, simple imputation, multiple imputation, and model-based imputation.

3. Discuss the statistical and operational implications of class imbalance and explain how evaluation should be designed.

4. Explain PCA, its assumptions and limitations, and compare it with feature selection.

5. Design a reproducible data-preparation pipeline for a regulated organisation, including documentation, testing, governance, and monitoring.

Analytical / Application Questions

1. A fraud dataset contains 1% confirmed fraud cases. Propose an evaluation strategy and explain why accuracy alone is unsuitable.

2. A model uses a customer’s “next-month payment status” to predict whether the customer will default this month. Identify the methodological problem and redesign the feature using only information available at prediction time.

3. A dataset contains repeated measurements from the same patients across several years. Explain how you would split the data, construct temporal features, handle missing laboratory values, and assess whether the resulting model generalises to new patients.

Multiple Choice Questions with Answers

Question 1

Which action best prevents preprocessing leakage during cross-validation?

A. Fit imputation and scaling separately inside each training fold

B. Fit preprocessing once on the complete dataset before cross-validation

C. Use the test-set mean for missing-value replacement

D. Apply transformations only after reporting test performance

Correct Answer: A. Fit imputation and scaling separately inside each training fold

Question 2

What does the interquartile range equal?

A. Mean minus median

B. Q3 minus Q1

C. Maximum minus minimum

D. Standard deviation divided by the mean

Correct Answer: B. Q3 minus Q1

Question 3

Which encoding is generally appropriate for a nominal variable with no natural order?

A. Ordinal integer encoding

B. One-hot encoding

C. Rank scoring

D. Z-score encoding

Correct Answer: B. One-hot encoding

Question 4

Why can accuracy be misleading for a highly imbalanced classification problem?

A. Accuracy cannot be calculated from a confusion matrix

B. A majority-class classifier may achieve high accuracy without detecting minority cases

C. Accuracy always ignores true negatives

D. Accuracy is defined only for regression

Correct Answer: B. A majority-class classifier may achieve high accuracy without detecting minority cases

Question 5

Which metric is the harmonic mean of precision and recall?

A. Specificity

B. F1 score

C. Mean squared error

D. Adjusted R-squared

Correct Answer: B. F1 score

Question 6

A feature records an event that occurs after the prediction time. Using it for prediction is primarily an example of what?

A. Underfitting

B. Target leakage

C. Robust scaling

D. Random undersampling

Correct Answer: B. Target leakage

Question 7

Which transformation is commonly used for a strongly right-skewed non-negative variable containing zeros?

A. log(1+x)

B. 1/(x−1) without checking values

C. Subtracting the median only

D. Converting every value to a category

Correct Answer: A. log(1+x)

Question 8

What is a principal component in PCA?

A. A randomly selected original variable

B. A direction capturing variance subject to PCA’s orthogonality constraints

C. The class with the largest frequency

D. The median of all features

Correct Answer: B. A direction capturing variance subject to PCA’s orthogonality constraints

Question 9

Where should SMOTE be applied in a cross-validation workflow?

A. To the complete dataset before splitting

B. Only within the training portion of each fold

C. Only to the final test set

D. After test predictions are generated

Correct Answer: B. Only within the training portion of each fold

Question 10

Which practice most directly supports reproducibility?

A. Overwriting raw data after cleaning

B. Changing preprocessing rules manually for each run

C. Versioning data, code, parameters, and environments

D. Removing all metadata to simplify storage

Correct Answer: C. Versioning data, code, parameters, and environments