Data Preparation and Feature Engineering

Introduction

Data mining rarely fails because an algorithm is unavailable. It more often fails because the data supplied to the algorithm are incomplete, inconsistent, poorly represented, or contaminated by information that would not be available at prediction time. Data preparation converts raw observations into a reliable analytical representation. Feature engineering then constructs variables that make useful patterns easier for a learning algorithm to discover.

This chapter treats preparation as a disciplined stage of the data-mining pipeline rather than as informal spreadsheet manipulation. The central principle is that every transformation should be justified by the meaning, provenance, timing, and intended use of the data. Preparation must also be performed without data leakage: information from a validation or test set must not influence the training process.

Learning Objectives

After studying this chapter, you should be able to:

• identify data-quality problems and select suitable cleaning methods;
• integrate heterogeneous data sources while preserving semantic consistency;
• transform, normalize, discretize, and sample data appropriately;
• diagnose and treat missing values;
• construct, select, and evaluate useful features;
• address class imbalance using statistically responsible methods;
• explain and apply dimensionality-reduction techniques, especially principal component analysis (PCA);
• design a reproducible preparation pipeline and evaluate its limitations and risks.

1. The Role of Data Preparation

A dataset can be represented as a matrix X ∈ ℝ^n×p, where n is the number of observations and p is the number of features. The target variable, when supervised learning is used, is denoted by y. Rows should represent comparable analytical units—such as customers, transactions, or sensors—and columns should represent clearly defined attributes.

A useful preparation workflow is:

1. define the analytical unit, target, prediction time, and success criterion;
2. profile data types, ranges, distributions, duplicates, missingness, and provenance;
3. clean and integrate the sources;
4. split data into training, validation, and test partitions;
5. fit transformations using training data only;
6. engineer and select features;
7. address imbalance and reduce dimensionality where justified;
8. validate the final representation and document every decision.

The order matters. For example, a random split before deduplication can place nearly identical records in both training and test sets, producing an unrealistically high score. Similarly, computing a global mean before splitting allows test-set information to enter training.

2. Data Quality and Cleaning

Data quality has several dimensions:

2.1 Type, range, and logical validation

A numerical age of −4 is invalid under ordinary demographic interpretation. A percentage may be stored as 0.25 or 25; both can be valid conventions, but mixing them is dangerous. Logical rules can detect contradictions, such as a discharge date earlier than an admission date or a product marked both “discontinued” and “actively sold.” Validation rules should distinguish an impossible value from a rare but possible value.

2.2 Duplicates and outliers

Exact duplicates may result from repeated file exports. Entity duplicates are more difficult: “A. Kumar,” “Anil Kumar,” and two records with the same telephone number may refer to one person. Record linkage should use stable identifiers where available and cautious similarity rules where they are not. Merging uncertain matches can be more harmful than retaining duplicates.

An outlier is an observation unusually distant from the rest of the data. It may represent an error, a legitimate extreme, or an important minority. For a numerical variable, the interquartile range is

where Q₁ and Q₃ are the first and third quartiles. A common exploratory flag is a value below Q₁ − 1.5IQR or above Q₃ + 1.5IQR. This is a screening convention, not proof that a record is erroneous. Robust transformations, winsorization, separate modeling, or domain review may be preferable to deletion.

2.3 Cleaning text and categorical values

Text fields often contain differences in capitalization, whitespace, punctuation, spelling, and encoding. Standardization may include Unicode normalization, case conversion, controlled vocabularies, and explicit treatment of unknown values. “Unknown,” “not applicable,” and “not collected” should not automatically be collapsed into one category because they can have different meanings.

3. Data Integration

Integration combines tables, files, APIs, streams, or databases. It requires more than joining columns. The analyst must establish:

• schema correspondence: which fields represent the same concept;
• entity resolution: whether identifiers refer to the same entity;
• granularity: whether one row represents a customer, order, or order item;
• temporal alignment: whether measurements refer to comparable times;
• unit and coding standards: such as kilograms versus pounds;
• lineage: the origin, transformation history, and responsible owner of each field.

Suppose a customer table has one row per customer while a transaction table has many rows per customer. A direct join followed by modeling may repeat customer attributes for every transaction. This is not necessarily wrong, but the analytical unit must be stated. If the task is customer-level churn prediction, transaction data should be aggregated using information available before the prediction date, for example purchase count, recency, and average order value.

A temporal cutoff is essential. For an observation at time t₀, a feature must be computed from records with timestamps no later than the permitted information time. Future purchases, later diagnoses, or post-outcome status changes create target leakage.

4. Missing-Value Treatment

Missingness is not a single condition. A value may be missing completely at random, missing at random conditional on observed variables, or missing not at random because the missingness depends on the unobserved value or a sensitive process. These assumptions cannot usually be proved from the dataset, so they must be examined using context and missingness indicators.

4.1 Diagnosing missingness

For feature X_j, its missingness indicator may be defined as

The proportion missing is

where n is the number of rows. Report missingness overall and by important groups, time periods, and outcome classes. A high missing rate in one subgroup may reveal unequal access, measurement bias, or an operational problem.

4.2 Treatment methods

• Deletion: Remove rows or features only when the loss is acceptable and the missingness mechanism supports the decision.
• Simple numerical imputation: Replace missing values with a training-set mean or median. The median is more robust to skew and outliers.
• Categorical imputation: Use a training-set mode or an explicit “missing” category when absence itself may be informative.
• Group-wise imputation: Use a reference group only when the grouping variable is available at prediction time and the procedure is validated.
• Model-based imputation: Predict missing values from other variables. This can preserve relationships but may introduce model assumptions and false certainty.
• Missingness indicators: Add a binary feature indicating that the original value was missing. This is useful when the fact of nonresponse carries information.

All imputation parameters must be fitted on training data only. For example, if the training median income is 42,000, that value—not the median of the complete dataset—must be used to transform validation and test records. Imputation does not recover the true value; it creates a defensible replacement and should be reported as such.

5. Transformation and Representation

5.1 Numerical transformations

Skewed positive variables such as income or transaction amounts may benefit from a logarithmic transformation. A common form that also accommodates zero is

where x ≥ 0 and the logarithm base is fixed, commonly the natural logarithm. The transformation compresses large values; it does not make a distribution automatically normal.

Ratios and rates can be useful, but their denominators must be meaningful and nonzero. A “cost per visit” feature should specify how zero visits are handled. Features based on counts should often include an exposure period, because 20 events in one month and 20 events in five years do not have the same interpretation.

5.2 Categorical encoding

• One-hot encoding creates one binary column per category, usually omitting one reference category for linear models with an intercept to avoid perfect multicollinearity.
• Ordinal encoding is appropriate only when categories have a genuine order, such as low, medium, and high.
• Frequency encoding replaces a category with its training-set frequency; it may be compact but can conceal meaning.
• Target encoding replaces a category with an outcome statistic. It must be computed within training folds, with smoothing where appropriate, or it will leak target information.

High-cardinality identifiers, such as account numbers, are usually identifiers rather than predictive features. Including them can cause memorization and poor generalization.

6. Scaling and Normalization

Scaling is important for distance-based, gradient-based, and regularized algorithms. It is usually less important for ordinary decision trees, although preprocessing may still improve pipeline consistency.

6.1 Min–max scaling

This maps training values to approximately [0, 1], assuming x_max > x_min. It is sensitive to extreme values and future observations may fall outside the training range.

6.2 Standardization

Here μ is the training mean and σ is the training standard deviation, with σ > 0. The result has training mean approximately 0 and standard deviation 1. Standardization does not guarantee bounded values or remove outliers.

6.3 Robust scaling

This is less affected by extreme observations. The location and scale estimates must still be learned from training data.

7. Discretization and Sampling

Discretization converts a continuous variable into intervals. Equal-width binning divides the numerical range into intervals of identical width; equal-frequency binning chooses intervals containing approximately equal numbers of observations. Supervised discretization may use the target, but it must be performed inside training folds to prevent leakage. Binning improves interpretability and can represent nonlinear effects, but it loses information and may create unstable boundary effects.

Sampling is used to reduce computational cost or construct a representative dataset. Simple random sampling gives each unit an equal selection probability. Stratified sampling preserves specified group proportions, which is valuable when classes or regions differ greatly in size. Cluster sampling selects groups rather than individual units and may be less costly, although observations within a cluster are correlated.

When sampling is not self-weighting, estimates may require sampling weights. If unit i has selection probability π_i, its inverse-probability weight is

where 0 < π_i ≤ 1. Weights should be used consistently with the estimation procedure and documented. Sampling must respect the unit of analysis; sampling individual transactions from customers can unintentionally overrepresent high-frequency customers.

8. Feature Engineering

Feature engineering uses domain knowledge and raw fields to create representations that expose relevant structure. Examples include:

• calendar features such as day of week, month, holiday, or time since an event;
• aggregation features such as count, sum, mean, minimum, maximum, and recency;
• interaction features such as price × quantity or treatment × age when a combined effect is plausible;
• text features such as token counts, n-grams, or embeddings;
• spatial features such as distance to a service center;
• temporal-window features such as events in the preceding 7 or 30 days.

A feature is appropriate only if it is available at the intended decision time. The feature specification should define its source, unit, calculation window, missing-value policy, and update frequency.

Feature quality can be evaluated through predictive usefulness, stability across time and groups, interpretability, cost of collection, and risk of leakage. A highly predictive feature may be unacceptable if it is a proxy for protected status, was collected after the outcome, or is unavailable during deployment.

9. Feature Selection

Feature engineering creates candidates; feature selection limits the final set.

9.1 Filter methods

Filter methods rank features independently of the final model. Examples include variance thresholds, correlation screening, mutual information, and statistical association tests. They are computationally efficient but may miss combinations of individually weak features.

9.2 Wrapper methods

Wrapper methods evaluate subsets using a model, such as recursive feature elimination. They can capture model-specific usefulness but are computationally expensive and susceptible to overfitting unless nested validation is used.

9.3 Embedded methods

Embedded methods select during model fitting. For linear regression with a lasso penalty, the objective can be written as

where X is the feature matrix, y is the target vector, β is the coefficient vector, n is the number of training observations, and λ ≥ 0 controls regularization. The term ||β||₁ = ∑_j |β_j| encourages some coefficients to become exactly zero. Features should be scaled when coefficient penalties are intended to treat them comparably. Regularization strength must be selected using training data and validation or cross-validation.

Selection should also consider redundancy. Two strongly correlated features may carry similar information, but automatic removal can be harmful if one is more stable or more interpretable. Selection decisions should be recorded rather than made solely from a single sample.

10. Class Imbalance

In an imbalanced classification problem, one class is much less frequent than another. Accuracy can then be misleading. If 99% of transactions are legitimate, a classifier that always predicts “legitimate” achieves 99% accuracy but detects no fraud.

A confusion matrix contains true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Important metrics include:

Precision is the proportion of predicted positives that are positive; recall is the proportion of actual positives detected. F1 is the harmonic mean and is meaningful only when both denominators are defined. Specificity is TN ÷ (TN + FP). The appropriate metric depends on the cost of errors and the operational threshold.

Responses to imbalance include class-weighted loss, threshold adjustment, undersampling the majority class, oversampling the minority class, and synthetic methods such as SMOTE. Resampling must be applied inside each training fold. Never oversample before splitting, because duplicated or synthetic observations can cross into validation or test sets. The untouched test set should preserve the deployment distribution unless a different evaluation design is explicitly justified.

11. Dimensionality Reduction and PCA

High-dimensional data may contain redundant variables, increase computation, and make visualization difficult. Dimensionality reduction maps the original representation to fewer dimensions. It can be used for compression, visualization, denoising, or modeling, but reduced dimensions may be less interpretable.

11.1 PCA concept

Principal component analysis finds orthogonal directions of maximum variance. Before PCA, numerical variables are commonly centered; standardization is also appropriate when variables have different units or scales. Let the centered data matrix be X_c. The sample covariance matrix is

where n > 1 is the number of observations and X_c has column means zero. The covariance matrix has units equal to products of the original units; therefore, PCA on unstandardized variables is dominated by variables with large variance or large measurement units.

The first principal direction w₁ solves

The unit-length constraint prevents arbitrarily scaling w. Subsequent directions are orthogonal to earlier directions and maximize remaining variance. Equivalently, the directions are eigenvectors of S, and the corresponding eigenvalues λ_k are the variances of the principal-component scores.

The explained variance ratio for component k is

and the cumulative explained variance through component r is ∑_k=1^r EVR_k. This ratio describes variance retained, not predictive accuracy, fairness, or causal importance.

11.2 Worked PCA illustration

Consider two centered, standardized features whose covariance matrix is

The eigenvalues are 1.8 and 0.2, with total variance 2. The first component therefore explains 1.8 ÷ 2 = 0.90, or 90%, and the second explains 10%. A corresponding first eigenvector is proportional to (1, 1), so the first component represents a direction in which the two variables increase together. The sign of an eigenvector is arbitrary: (1,1) and (−1,−1) describe the same component. The example illustrates variance preservation, not a general rule that two correlated features should always be replaced by one.

PCA is sensitive to scaling, outliers, and the covariance structure of the training sample. Fit PCA on training data only and apply the learned centering, scaling, and component directions to new data. Components may be difficult to explain to stakeholders, and a high explained-variance ratio does not ensure that the discarded dimensions are irrelevant to a rare target or protected group.

12. Integrated Illustrative Case: Credit-Card Fraud Detection

Background

A payment company wants to identify potentially fraudulent card transactions quickly. Fraud represents a very small fraction of transactions, and the data combine transaction records, merchant information, customer history, and device signals.

Problem

The raw data contain duplicated transactions, inconsistent merchant categories, missing device information, extreme transaction amounts, and features recorded after a fraud investigation. A naive model reports very high accuracy but misses most fraud.

Analysis

The analytical unit is one transaction, and the prediction time is authorization. Therefore, investigation outcomes and later chargeback status cannot be used as predictors. The team checks duplicates using transaction identifiers and timestamps, standardizes currency, validates nonnegative amounts, and aligns all historical features to the authorization time.

Missing device identifiers are separated into “not provided” and “not applicable” where the source system supports that distinction. Numerical history features are median-imputed using training data, with missingness indicators. Transaction amount receives a log transformation. Categorical merchant codes are mapped to a controlled vocabulary and one-hot encoded.

Features include the number of transactions by the card in the preceding hour and day, time since the previous transaction, geographic distance from the recent transaction, and merchant-category frequency. Each is computed from prior transactions only. The team does not calculate a customer’s lifetime fraud rate using records after the prediction date.

Approach and solution

The data are split chronologically so that later transactions act as a realistic test set. Class weights are considered, and model thresholds are selected according to the cost of false approvals and false declines. Precision, recall, precision–recall curves, calibration, and performance by region and customer segment are reported. If resampling is tested, it occurs within the training portion of each validation fold. PCA is considered for a large correlated device-signal block, but the original variables are retained for auditability if the predictive improvement is negligible.

Outcome

The resulting pipeline produces fewer false alarms at a required fraud-recall level and avoids the inflated performance produced by leakage. Monitoring is added for feature drift, missingness changes, and subgroup performance. Analysts document that a high-risk score is a screening signal, not proof of fraud.

Learning points

The case demonstrates that preparation is inseparable from problem definition, time-aware validation, metric selection, and governance. The most valuable feature may be a carefully defined historical aggregate rather than a sophisticated algorithm. It also shows why a feature’s operational availability and social consequences matter as much as its statistical association.

13. Governance, Ethics, and Best Practices

Preparation can encode discrimination. Missingness may reflect unequal access to services; location and device variables may act as proxies for protected characteristics; and historical labels may reflect biased enforcement. Teams should assess performance and error rates across relevant groups, minimize unnecessary personal data, control access, and document lawful purpose and retention requirements.

A reproducible pipeline should record data versions, schema definitions, transformation parameters, random seeds where relevant, feature lineage, validation design, and exceptions. Privacy-preserving preparation may require aggregation, pseudonymization, access controls, or differential privacy in appropriate settings. Pseudonymization reduces direct identification but does not make data anonymous.

Contemporary practice increasingly uses automated feature stores, data-quality checks, drift monitoring, and machine-learning operations pipelines. Automation improves consistency but does not replace domain review. A feature store can also spread a leaked feature quickly if its time semantics are not explicit. The strongest practice is to treat each feature as a governed data product with an owner, definition, availability time, and test suite.

Challenges and Limitations

No preparation method is universally optimal. Deletion can reduce representativeness; imputation can understate uncertainty; scaling can obscure original units; discretization can discard information; sampling can change prevalence; feature selection can be unstable; and PCA can reduce interpretability. Validation scores are estimates subject to sampling variation and distribution shift. Preparation should therefore be revisited when the population, process, measurement system, or decision costs change.

Chapter Summary

Data preparation establishes the reliability and meaning of a data-mining representation. It includes profiling, validation, deduplication, integration, temporal alignment, missing-value treatment, transformation, scaling, discretization, and sampling. Feature engineering converts domain knowledge and raw records into variables that are available, stable, and useful at decision time. Feature selection may use filter, wrapper, or embedded methods, but all selection and transformation must be protected against leakage.

Imbalanced classification requires metrics and procedures aligned with error costs rather than accuracy alone. Precision, recall, F1, specificity, and threshold analysis provide complementary views. PCA reduces correlated numerical data into orthogonal components based on covariance and explained variance, but it is sensitive to scaling and may reduce interpretability. Across all methods, reproducibility, fairness, privacy, lineage, and deployment-time availability are essential professional requirements.

Key Terms

Data cleaning: Detecting and correcting invalid, inconsistent, duplicate, or erroneous records.

Data integration: Combining multiple data sources into a coherent analytical representation.

Data leakage: Use of information during training that would not be available when making the real prediction.

Entity resolution: Determining which records refer to the same real-world entity.

Missingness indicator: A variable recording whether another value was missing.

Imputation: Replacing a missing value with a calculated or modeled value.

Outlier: An observation unusually distant from the distribution of other observations.

Normalization: Rescaling variables according to a specified range or statistical convention.

Standardization: Centering a variable and dividing by its standard deviation.

Discretization: Converting a continuous variable into intervals or categories.

Stratified sampling: Sampling that preserves selected subgroup proportions.

Feature engineering: Constructing predictive variables from raw data and domain knowledge.

Feature selection: Choosing a subset of features for analysis or modeling.

Target encoding: Representing categories using statistics derived from the target.

Class imbalance: A substantial difference in the frequencies of target classes.

Precision: The proportion of predicted positives that are truly positive.

Recall: The proportion of actual positives that are detected.

Principal component: An orthogonal direction capturing variance in PCA.

Explained variance ratio: The proportion of total variance represented by a principal component.

Feature leakage: A form of leakage caused by a feature containing future or outcome-derived information.

Review Questions

5 Short Answer Questions

1. Why must imputation parameters be fitted on training data only?
2. Distinguish one-hot encoding from ordinal encoding.
3. What is the difference between precision and recall?
4. Why can unscaled variables distort PCA?
5. Give two examples of temporal leakage.

5 Long Answer Questions

1. Explain a complete data-preparation workflow for a supervised classification problem, including the correct order of splitting, fitting, and transforming data.
2. Compare deletion, simple imputation, model-based imputation, and missingness indicators. Discuss assumptions and limitations.
3. Explain filter, wrapper, and embedded feature-selection methods and identify situations in which each is useful.
4. Derive the meaning of the PCA explained variance ratio and discuss why a high ratio does not guarantee high predictive performance.
5. Discuss how data preparation can create unfair outcomes and propose governance controls to reduce those risks.

3 Analytical / Application Questions

1. A fraud dataset contains 0.5% fraudulent transactions. Design an evaluation and resampling strategy that avoids leakage and justify at least three performance measures.
2. A customer table is joined to a transaction table containing multiple rows per customer. Explain how you would construct customer-level features without changing the analytical unit or using future information.
3. Two numerical variables have very different units and one contains extreme values. Recommend a scaling and dimensionality-reduction strategy, including the assumptions and validation steps you would use.

Multiple Choice Questions with Answers

Question 1

Which action most directly prevents target leakage during median imputation?

A. Use the median of the complete dataset

B. Fit the median using training data only

C. Replace all missing values with zero

D. Impute after evaluating the test set

Correct Answer: B. Fit the median using training data only

Question 2

For a positive numerical variable with a long right tail and possible zero values, which transformation is commonly appropriate?

A. log(1 + x)

B. x ÷ 0

C. x − maximum(x)

D. One-hot encoding

Correct Answer: A. log(1 + x)

Question 3

Which metric is TP ÷ (TP + FN)?

A. Precision

B. Specificity

C. Recall

D. Accuracy

Correct Answer: C. Recall

Question 4

Why should oversampling normally be performed inside training folds?

A. To increase the test-set class imbalance

B. To prevent resampled information entering validation data

C. To eliminate the need for a target variable

D. To guarantee perfect calibration

Correct Answer: B. To prevent resampled information entering validation data

Question 5

In PCA, an eigenvalue associated with a component represents its:

A. Number of missing observations

B. Classification accuracy

C. Variance in the component direction

D. Number of original categories

Correct Answer: C. Variance in the component direction

Question 6

Which encoding is appropriate only when categories have a genuine order?

A. Ordinal encoding

B. Random encoding

C. Hashing of row identifiers

D. Mean imputation

Correct Answer: A. Ordinal encoding

Question 7

If TP = 80 and FN = 20, recall is:

A. 20%

B. 50%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 8

What is the principal risk of using a feature computed from events occurring after prediction time?

A. Under-scaling

B. Data leakage

C. Stratification

D. Deduplication

Correct Answer: B. Data leakage

Question 9

Which sampling method deliberately preserves selected subgroup proportions?

A. Stratified sampling

B. Unrestricted deletion

C. Principal sampling

D. Ordinal sampling

Correct Answer: A. Stratified sampling

Question 10

For covariance matrix S = [[1, 0.8], [0.8, 1]], the first PCA component explains what proportion of total variance?

A. 0.10

B. 0.20

C. 0.80

D. 0.90

Correct Answer: D. 0.90