Data Preparation and Feature Engineering

Introduction

Machine-learning models learn from the representation supplied to them. A sophisticated algorithm cannot compensate for unreliable measurements, inconsistent formats, poorly encoded categories, leakage, or features that do not represent the underlying problem. Data preparation is therefore not a preliminary clerical task; it is part of model design.

Structured data commonly arrives as tables in which rows represent observations and columns represent variables. A dataset may contain numerical measurements, categorical labels, dates, text-derived indicators, identifiers, a target variable, and metadata. The data scientist must determine which variables are valid, how they should be represented, and how transformations can be performed without using information that would be unavailable at prediction time.

This chapter presents a practical workflow for acquiring, exploring, cleaning, transforming, and enriching structured data. It emphasizes reproducibility and the prevention of data leakage—the accidental use of information from the future, the test set, or the target-generating process.

Learning objectives

By the end of this chapter, you should be able to:

• inspect the structure, quality, and provenance of a tabular dataset;
• distinguish missingness, outliers, invalid values, and rare categories;
• select suitable strategies for imputation, outlier treatment, encoding, and scaling;
• engineer useful numerical, categorical, temporal, and aggregate features;
• address class imbalance using appropriate metrics and resampling strategies;
• construct a reproducible preparation pipeline; and
• identify and prevent common forms of data leakage.

1. The data-preparation workflow

A robust workflow is iterative rather than strictly linear:

1. Acquire and document data. Record source, collection period, unit definitions, permissions, and the meaning of each field.
2. Define the prediction task. Specify the target, prediction time, observation unit, and operational decision.
3. Audit the data. Inspect types, ranges, duplicates, missingness, cardinality, and possible errors.
4. Split the data. Create training, validation, and test sets using a method appropriate to the sampling and time structure.
5. Fit transformations on training data only. This includes imputers, scalers, encoders, feature selectors, and resampling procedures.
6. Transform validation and test data using the fitted objects. Do not refit them on these sets.
7. Evaluate and revise. Assess both model performance and whether features are stable, ethical, interpretable, and available at deployment.

The observation unit must be explicit. For example, a row may represent a customer, a transaction, a customer-month, or a sensor reading. Mixing units can create duplicated information and misleading aggregates. A unique identifier may be necessary for joining records but is generally not a predictive feature unless its relationship to the target is legitimate and stable.

2. Understanding and exploring structured data

Data types and semantic types

A machine-learning library may label a field as numeric, but its semantic meaning matters more than its storage type. A postal code stored as an integer is usually categorical; a date stored as text is temporal; a product code is an identifier; and a rating from 1 to 5 may be ordinal rather than continuous.

Descriptive statistics

For a numerical variable with observations x₁, x₂, …, x_n, the arithmetic mean is:

Here, n is the number of observations and x̄ is the sample mean. The mean is sensitive to extreme values. The median is the middle ordered observation (or the average of the two middle observations), and is often more robust for skewed data.

The sample variance and standard deviation are:

The denominator n − 1 is used when estimating population variability from a sample under the usual independent-sampling assumptions. Variance has squared units, whereas standard deviation has the same units as the original variable.

Exploration should include frequency tables for categorical variables, quantiles, histograms or density plots, scatterplots, grouped summaries, and target rates by category. These summaries are descriptive, not proof of causation. A strong association may result from confounding, sampling bias, or leakage.

Data-quality audit

A professional audit asks:

• Are column names, units, and definitions documented?
• Are values within plausible domain ranges?
• Are dates valid and consistently formatted?
• Are duplicate rows genuine repeated events or accidental copies?
• Are there impossible combinations, such as a discharge date before admission?
• Does the target contain errors or post-outcome information?
• Are missing values represented consistently as null, blank, “unknown,” or a sentinel such as −999?

Invalid values should not automatically be treated as missing. A value of zero may be a legitimate measurement, a structural zero, or an encoding error. Domain knowledge is required.

3. Splitting data and preventing leakage

A dataset is commonly divided into training, validation, and test sets. The training set is used to fit model parameters and preprocessing parameters. The validation set supports model selection and tuning. The test set is reserved for final, approximately unbiased assessment.

For a random split, every observation should have an appropriate chance of entering each subset, and related observations must not be separated in a way that allows information to cross the boundary. For time-dependent problems, training observations should precede validation and test observations. For repeated measurements from the same person, grouped splitting can prevent the same person appearing in both training and test sets.

Data leakage occurs when information unavailable at prediction time influences model fitting or evaluation. Important examples include:

• calculating an imputation mean using all rows before splitting;
• scaling with statistics from the test set;
• oversampling the entire dataset before splitting;
• using a variable recorded after the outcome;
• creating a customer aggregate using future transactions;
• allowing records from the same individual into both training and test sets; and
• selecting features based on their association with the target using the entire dataset.

The safe sequence is: split first, fit every data-dependent operation on training data, and apply the learned operation to other sets. A pipeline should combine these operations so that the sequence is enforced by software rather than memory.

4. Missing values

Missingness may be caused by nonresponse, sensor failure, a field not being applicable, data integration problems, or deliberate withholding. The missingness mechanism is important. In simplified statistical terminology, missingness may be:

• MCAR (missing completely at random): missingness is unrelated to observed and unobserved values;
• MAR (missing at random): missingness depends on observed variables; or
• MNAR (missing not at random): missingness depends on the unobserved value or other unmeasured factors.

These categories are assumptions about a data-generating process, not labels that can always be proven from the dataset.

Common strategies include deletion, simple imputation, model-based imputation, and an explicit missingness indicator. Mean imputation is simple but reduces variation and can distort relationships. Median imputation is more robust to skew. For a categorical field, the mode or a separate “missing” category may be appropriate. Forward filling can be sensible for time series only when the value is expected to persist and the operation respects time order.

For a feature x, a missingness indicator can be defined as:

The imputed feature and mᵢ may both be supplied when the fact that a value was missing is informative. However, the indicator can encode institutional or socioeconomic processes and should be examined for fairness.

Imputation must preserve data types and plausible ranges. A model-based imputer should be fitted only on the training data. Imputation is not a substitute for investigating why data are absent.

5. Outliers and unusual observations

An outlier is an observation that is unusually distant from the bulk of the data. It may be a recording error, a valid rare case, a member of another population, or evidence of a changed process. Removing it merely because it is inconvenient can damage the analysis.

The interquartile range is:

where Q₁ and Q₃ are the first and third quartiles. A common exploratory rule flags values below Q₁ − 1.5IQR or above Q₃ + 1.5IQR. This is a screening rule, not a universal law. It is most useful for identifying observations for investigation.

Possible treatments are correction from the source, removal when an observation is demonstrably invalid, winsorisation, a monotonic transformation such as a logarithm for positive skewed values, robust scaling, or leaving the value unchanged. Tree-based models are often less sensitive to monotonic scaling than distance-based or linear models, but extreme values can still affect splits, loss functions, and data quality.

A z-score is sometimes used:

It expresses distance from the sample mean in standard-deviation units. It is less reliable for highly skewed or heavy-tailed distributions, and it should not be interpreted as a universal outlier test.

6. Encoding categorical variables

One-hot encoding

For a nominal variable with categories such as red, green, and blue, one-hot encoding creates indicator columns. Each row receives 1 in the column corresponding to its category and 0 elsewhere. This avoids imposing an artificial numeric order.

For linear regression with an intercept, one category is commonly omitted as a reference to avoid exact linear dependence among all category indicators. With regularisation or implementations designed to handle redundant columns, retaining all indicators may be computationally acceptable, but interpretation and numerical stability must still be considered.

Unknown categories at deployment should have a defined policy, such as an “unknown” bucket or an encoder setting that ignores unseen categories. Failing on an unseen category can interrupt production scoring.

Ordinal and alternative encodings

Ordinal encoding maps ordered categories to values such as 1, 2, and 3. It is suitable only when the order is meaningful and the model will not misinterpret arbitrary differences as precise distances. Target encoding replaces a category with a target-based statistic, often a smoothed conditional mean. It can be powerful for high-cardinality variables but has a high leakage risk and must be computed within training folds or through an out-of-fold procedure.

Frequency encoding uses category frequency, while hashing maps categories into a fixed number of bins. These methods can reduce dimensionality but may lose interpretability or cause collisions. High-cardinality identifiers deserve special suspicion: they can memorize the training set without generalising.

7. Scaling and transforming numerical variables

Many algorithms depend on distances, dot products, or gradient magnitudes. Scaling prevents a variable measured in large units from dominating one measured in small units.

Standardisation uses training-set mean μ and standard deviation σ:

After transformation, the training feature has mean approximately 0 and standard deviation approximately 1, assuming σ > 0. The same μ and σ must be applied to validation, test, and production records.

Min–max scaling maps a feature to an interval, commonly [0, 1]:

This requires x_max > x_min and can be strongly affected by future extremes. Robust scaling uses the median and IQR and is less sensitive to outliers. Scaling is generally important for k-nearest neighbours, k-means, support-vector machines, neural networks, and many regularised linear models. It is usually not essential for ordinary decision trees, although consistent preprocessing may still simplify a mixed-model workflow.

Skewed positive features such as income or transaction amount may benefit from log(1 + x), where adding 1 permits zero values. The transformation is appropriate only when x ≥ 0 and must be interpreted on its transformed scale.

8. Feature engineering

Feature engineering converts raw fields into representations that expose useful structure while remaining available at prediction time.

Numerical features

Useful constructions include ratios, differences, totals, rolling statistics, counts, interaction terms, and domain-based thresholds. A ratio must have a meaningful denominator and a policy for zero or near-zero values. For example, utilization may be defined as:

The denominator must be positive and measured at the same time reference. Adding arbitrary ratios can amplify noise.

An interaction between features x₁ and x₂ is represented as x₁x₂. It allows the effect of one variable to depend on the other. In a linear model, this relationship is not captured unless the interaction term is explicitly included. Interactions should be motivated by domain reasoning or validated through appropriate evaluation.

Dates and time

A timestamp can generate year, month, day of week, hour, elapsed time since an event, and holiday or business-day indicators. Cyclical variables require care: Sunday and Monday are adjacent in a weekly cycle, although their integer codes may be far apart. A cyclical representation for a period P is:

Here, t is the position within the cycle and P is its period, such as 7 for a weekly cycle or 24 for an hourly daily cycle. Both sine and cosine are needed to represent the circular position without a false boundary.

Elapsed-time features must use only dates known at prediction time. For a forecasting task, a feature such as “days until closure” would leak the outcome if closure occurs after the prediction timestamp.

Grouped and aggregate features

Customer-level totals, mean purchase value, count of recent events, and time since the last event can be highly informative. They must be calculated using an as-of date and, for training observations, only historical records preceding that date. Otherwise, the aggregate may include future behaviour.

Feature selection

Feature selection may be filter-based, wrapper-based, or embedded in a model. Correlation screening alone can miss nonlinear relationships and can select unstable variables. Selection must occur inside the training process, particularly during cross-validation. A feature should be retained not only because it improves a score, but also because it is valid, stable, explainable, legally permissible, and operationally available.

9. Class imbalance

In binary classification, let N_+ denote the number of positive cases and N_- the number of negative cases. The positive-class prevalence is:

When π is small, accuracy can be misleading. A classifier that predicts every case as negative may achieve high accuracy while detecting no positive cases.

The confusion matrix contains true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). Important metrics include:

These formulas require nonzero denominators. Precision measures the proportion of predicted positives that are correct; recall measures the proportion of actual positives detected. The appropriate metric depends on the cost of errors. Precision–recall curves are often informative when the positive class is rare. Calibration—the agreement between predicted probabilities and observed frequencies—may also matter more than a ranking metric in risk decisions.

Strategies include class-weighted learning, threshold adjustment, under-sampling, over-sampling, and synthetic methods such as SMOTE. Resampling must occur only within the training portion of each cross-validation fold. Synthetic samples generated before splitting can cause near-duplicates to appear in validation data. Resampling does not change the real-world prevalence and should not automatically be applied to the test set.

10. Reproducible preprocessing pipelines

A pipeline is an ordered, executable specification of preparation and modelling steps. A typical tabular classification pipeline is:

raw data → split → column typing → imputation → encoding/scaling → feature construction → model → evaluation

Numerical and categorical columns often require different transformations. A column transformer can apply median imputation and scaling to numerical variables, while applying most-frequent imputation and one-hot encoding to categorical variables. The transformed output is then passed to a classifier.

Reproducibility also requires recording the data version, random seeds where relevant, software versions, feature definitions, training date, preprocessing parameters, model parameters, and evaluation protocol. A seed does not guarantee reproducibility across all hardware and library versions, but it reduces avoidable variation.

A pipeline should be tested for:

• consistent columns and data types;
• handling of missing and unseen categories;
• stable output dimensions;
• absence of future information in features;
• correct persistence and reloading; and
• behaviour on extreme, empty, or malformed inputs.

Illustrative case: predicting missed clinical appointments

Background and problem

A community clinic wants to predict whether a scheduled appointment will be missed so that staff can send reminders or offer limited overbooking. Each row represents an appointment, and the prediction is made 48 hours before the appointment. The target is binary: missed or attended.

Analysis

The raw data contain appointment time, booking time, patient age, neighbourhood, previous attendance count, previous missed count, reminder status, and appointment outcome. An initial audit finds that some age values are negative, neighbourhood is categorical, and previous attendance counts were calculated by a database query that included appointments after the prediction date. The outcome distribution is 12% missed and 88% attended.

The post-prediction aggregate is leakage and must be rebuilt using an as-of timestamp. Negative ages are investigated against the source system rather than automatically clipped. Reminder status is retained only if it is known 48 hours before the appointment.

Approach

The data are split chronologically: older appointments for training, a later period for validation, and the most recent period for testing. A grouped check ensures that the same patient does not create an unrealistic cross-period advantage if the deployment design requires patient-level separation.

Numerical variables receive training-only median imputation and robust scaling. Neighbourhood receives an explicit unknown category and one-hot encoding. Features include booking lead time, previous attendance rate, previous missed count calculated only from earlier appointments, day of week, and hour represented by sine and cosine. Because missed appointments are the minority class, the clinic compares class weighting with threshold adjustment and evaluates recall, precision, calibration, and operational cost rather than accuracy alone.

Outcome and governance

The final model improves detection of likely missed appointments, but its use is limited to supportive reminders. It is not used to deny appointments or impose penalties. The clinic documents the purpose, informs patients where required, monitors performance across neighbourhoods and age groups, and reviews whether reminder resources are distributed fairly. Feature drift and changes in booking policy are monitored after deployment.

Learning points

The case demonstrates that the most important improvement may come from correcting feature timing rather than selecting a more complex algorithm. It also shows why imbalance, temporal splitting, subgroup analysis, and governance must be considered together.

Limitations, ethics, and contemporary practice

Preparation decisions can encode institutional bias. Missingness may reflect unequal access, and location variables may act as proxies for protected characteristics. Removing a sensitive variable does not guarantee fairness because other variables may carry similar information. Teams should document intended use, assess subgroup performance, examine error costs, and provide human oversight for high-impact decisions.

Privacy principles include data minimisation, purpose limitation, access control, retention limits, and protection against re-identification. Identifiers should be separated from analytical features where possible. Regulatory obligations depend on jurisdiction and application, but health, credit, employment, education, and public-sector decisions commonly require heightened controls.

Modern feature stores and automated machine-learning systems can standardise transformations, but automation does not remove the need for semantic validation. Monitoring should detect missingness changes, category drift, distribution shift, pipeline failures, and changes in label prevalence. A feature that performs well during development may become invalid when a policy, sensor, pricing scheme, or data-collection process changes.

Best-practice checklist

1. Define the observation unit, target, prediction time, and permissible information.
2. Inspect semantics, units, ranges, duplicates, missingness, and provenance.
3. Split data appropriately before fitting data-dependent transformations.
4. Treat missingness and outliers as questions about the data-generating process.
5. Encode categories according to their meaning and deployment behaviour.
6. Scale only when the algorithm benefits, using training-derived parameters.
7. Engineer features that are available, stable, interpretable, and justified.
8. Evaluate imbalance with metrics aligned to error costs and prevalence.
9. Put preprocessing and modelling into one tested pipeline.
10. Record versions, parameters, assumptions, and monitoring responsibilities.

Chapter Summary

Data preparation is a central part of machine-learning design. Reliable work begins with a clear observation unit, prediction target, prediction time, and data audit. Missing values, outliers, invalid entries, categorical variables, skewed numerical features, and class imbalance require context-sensitive treatment rather than automatic rules. Numerical scaling supports algorithms based on distance, optimisation, or regularisation, while appropriate encoding prevents categories from receiving false meanings. Feature engineering can expose temporal, aggregate, interaction, and cyclical structure, but every feature must be available at prediction time.

The most important safeguard is leakage prevention: split data appropriately, fit transformations only on training data, and use reproducible pipelines. Evaluation must reflect class prevalence, operational error costs, calibration, subgroup effects, and deployment conditions. Ethical preparation includes privacy, fairness, documentation, monitoring, and restricting models to legitimate purposes.

Key Terms

Data leakage: Use of information in model development that would not be available at prediction time.

Observation unit: The entity or event represented by one row of data.

Imputation: Replacement or estimation of missing values.

Missingness indicator: A variable recording whether another value was missing.

Outlier: An unusually distant or atypical observation.

Interquartile range: The difference between the third and first quartiles.

Winsorisation: Capping extreme values at selected limits.

One-hot encoding: Representation of a categorical variable by binary indicator columns.

Ordinal encoding: Mapping ordered categories to numerical values.

Target encoding: Replacing categories with target-based statistics.

Standardisation: Transformation using a training mean and standard deviation.

Robust scaling: Scaling based on statistics less sensitive to outliers, commonly median and IQR.

Feature engineering: Creation or transformation of variables to represent useful predictive structure.

Interaction: A feature representing the combined effect of two or more variables.

As-of date: The time boundary defining information available for a prediction.

Class imbalance: Unequal representation of outcome classes.

Precision: The proportion of predicted positives that are true positives.

Recall: The proportion of actual positives that are detected.

Pipeline: An ordered, reproducible sequence of data transformations and modelling steps.

Distribution shift: Change in the data distribution between development and deployment.

Review Questions

Short Answer Questions

1. Why should preprocessing parameters be fitted on the training set only?
2. Distinguish an invalid value from a missing value.
3. When is one-hot encoding preferable to ordinal encoding?
4. What does recall measure in binary classification?
5. Give two examples of data leakage.

Long Answer Questions

1. Explain a complete workflow for preparing a structured dataset for supervised learning, from acquisition to evaluation.
2. Compare mean, median, model-based, and indicator-assisted imputation, including their assumptions and limitations.
3. Explain how outliers can be investigated and treated without automatically deleting valid rare observations.
4. Discuss numerical scaling, including standardisation, min–max scaling, and robust scaling, and relate each method to suitable algorithms.
5. Explain how a reproducible pipeline supports leakage prevention, deployment reliability, and auditability.

Analytical / Application Questions

1. A fraud dataset has 1% positive cases. A model achieves 99% accuracy but detects only 2% of fraud. Which additional metrics and evaluation design should be used, and why?
2. A customer-retention feature is “number of purchases in the next 30 days.” Explain why it is invalid for a churn prediction made today and propose a valid alternative.
3. A categorical field has 50,000 product codes, including many codes appearing once. Recommend an encoding and validation strategy, and discuss the risks of target encoding and memorisation.

Multiple Choice Questions with Answers

Question 1

Which action best prevents leakage when fitting a standard scaler?

A. Fit it on the complete dataset before splitting

B. Fit it on the training set and apply it unchanged to validation and test sets

C. Fit a separate scaler on each dataset

D. Fit it only on the test set

Correct Answer: B. Fit it on the training set and apply it unchanged to validation and test sets

Question 2

What does one-hot encoding primarily avoid for nominal categories?

A. Missing values

B. Outliers

C. Imposing an artificial numerical order

D. Class imbalance

Correct Answer: C. Imposing an artificial numerical order

Question 3

For a numerical feature with severe positive skew and non-negative values, which transformation may be appropriate?

A. log(1 + x)

B. Replace every value with zero

C. Assign arbitrary category codes

D. Delete the target variable

Correct Answer: A. log(1 + x)

Question 4

The interquartile range is defined as:

A. Q₁ + Q₃

B. Q₃ − Q₁

C. Q₁ ÷ Q₃

D. Q₃ − mean

Correct Answer: B. Q₃ − Q₁

Question 5

Which metric is the proportion of actual positive cases that a classifier detects?

A. Precision

B. Specificity

C. Accuracy

D. Recall

Correct Answer: D. Recall

Question 6

Why must oversampling be performed within training folds during cross-validation?

A. To increase the test-set size

B. To prevent synthetic or duplicated information from entering validation data

C. To eliminate all missing values automatically

D. To guarantee perfect calibration

Correct Answer: B. To prevent synthetic or duplicated information from entering validation data

Question 7

For a weekly cyclical feature, which representation avoids treating Sunday and Monday as distant?

A. A single arbitrary code from 1 to 7

B. The raw row number

C. Sine and cosine terms with period 7

D. The target mean for each day

Correct Answer: C. Sine and cosine terms with period 7

Question 8

A feature calculated from transactions occurring after the prediction timestamp is an example of:

A. Feature leakage

B. Robust scaling

C. One-hot encoding

D. Winsorisation

Correct Answer: A. Feature leakage

Question 9

Which model family generally does not require numerical scaling for its basic splitting mechanism?

A. k-nearest neighbours

B. Support-vector machines

C. Decision trees

D. Regularised logistic regression

Correct Answer: C. Decision trees

Question 10

If a binary dataset contains 20 positive cases and 180 negative cases, what is the positive prevalence?

A. 5%

B. 10%

C. 20%

D. 90%

Correct Answer: B. 10%