Data Transformation and Feature Engineering

Introduction

A predictive model rarely receives data in the form most suitable for learning. Operational databases contain identifiers, free-text fields, categorical labels, skewed measurements, missing values, dates, counts, and variables recorded on incompatible scales. Data transformation converts these raw representations into forms that algorithms can interpret reliably. Feature engineering goes further: it constructs variables that express useful structure, often by combining raw observations with domain knowledge.

The central principle is that transformation should improve representation without changing the meaning of the underlying phenomenon or introducing information that would be unavailable at prediction time. A technically sophisticated transformation can still produce an invalid model if it leaks target information, encodes a protected attribute through a proxy, or applies a method outside its assumptions.

This chapter develops a systematic approach to encoding, scaling, normalization, discretization, mathematical transformation, feature construction, high-cardinality variables, and domain-informed features. It also considers validation, fairness, governance, and contemporary automated approaches.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Explain why algorithms require transformed representations of data.
2. Select appropriate encoding methods for nominal, ordinal, binary, temporal, and high-cardinality variables.
3. Distinguish scaling, normalization, standardization, and distributional transformation.
4. Construct and evaluate mathematical, interaction, aggregate, temporal, and domain features.
5. Apply transformations without target leakage or unjustified assumptions.
6. Design a reproducible feature-engineering pipeline and assess its limitations.

1. The Representation Problem

Feature engineering concerns the mapping

where X is the raw input space, Z is the model-ready feature space, and φ is a transformation pipeline. A model then estimates a function from Z to an outcome Y. The transformation is useful when it makes relevant patterns easier to learn while preserving valid predictive-time information.

A feature may be:

• Raw: directly recorded, such as age or transaction amount.
• Transformed: altered mathematically, such as log income.
• Constructed: derived from multiple fields, such as debt-to-income ratio.
• Selected: retained from a larger set because it is useful, reliable, or interpretable.

Feature engineering is not merely a technical preprocessing step. It embodies assumptions about measurement, causality, business processes, and what constitutes meaningful similarity between observations.

Measurement scale and permissible operations

The measurement scale determines which transformations are meaningful.

Assigning integers to nominal categories does not create numerical order. For example, coding red, blue, and green as 1, 2, and 3 can cause a model to interpret green as greater than blue. Such an interpretation is usually invalid.

2. Encoding Categorical Variables

One-hot encoding

For a categorical variable with categories c₁, c₂, …, c_k, one-hot encoding creates binary indicators. For category c_j, the encoded vector has value 1 in position j and 0 elsewhere. A record belongs to exactly one category when the variable is single-valued and the categories are mutually exclusive.

In a linear model with an intercept, one category is commonly omitted to avoid perfect multicollinearity. The omitted category becomes the reference group. If all k indicators and an intercept are included, their sum equals 1 for every observation, creating linear dependence. Tree-based models do not require this specific omission, although redundant columns may remain inefficient.

One-hot encoding is transparent and usually appropriate for low- or moderate-cardinality nominal variables. Its disadvantages include a wide sparse matrix and difficulty handling previously unseen categories. Production systems should define an explicit unknown-category policy, such as an “other” bucket.

Ordinal encoding

Ordinal encoding maps ordered categories to numbers, such as low = 1, medium = 2, high = 3. It is appropriate only when the order is substantively justified. The numerical distances should not automatically be interpreted as equal distances. If the distinction between categories is important but equal spacing is doubtful, one-hot encoding may be safer.

Binary and multi-label variables

A true binary variable can generally be represented as 0 and 1, provided the coding is documented. Multi-label fields, such as a customer associated with several services, require a multi-hot representation: one binary column per possible label. This is different from one-hot encoding because several columns may equal 1 for a single record.

Frequency and count encoding

For category c, frequency encoding replaces the category by its observed frequency:

where n_c is the count of records in category c and N is the number of records used to estimate the frequencies. Frequency encoding is compact and can help tree models distinguish common from rare categories. It may, however, merge categories with equal frequencies and can reflect sampling or temporal artifacts rather than meaningful properties.

Counts must be calculated using the training partition only. If frequencies are computed using the complete dataset before splitting, information from validation or test records can enter the training representation.

Target or mean encoding

Target encoding replaces a category with an estimate of its conditional target mean. For a regression target, a smoothed version is often preferable:

where μ_c is the category-specific training mean, μ is the overall training mean, n_c is the category count, and λ ≥ 0 controls shrinkage toward the global mean. Larger λ produces stronger shrinkage, particularly for rare categories. For binary classification, the target mean is the estimated event rate.

Target encoding is powerful but intrinsically leakage-prone. A category’s encoding must not be calculated from the target of the same row when that row is used to train the model. Out-of-fold encoding solves this by calculating each training row’s encoding from other folds, while validation and test encodings are calculated from the full training data only. Smoothing does not by itself eliminate leakage.

Hashing and learned embeddings

The hashing trick maps categories into a fixed number of bins using a deterministic hash function. It limits dimensionality and handles unseen categories, but collisions occur: distinct categories may share a bin. The number of bins should be selected according to model capacity, data volume, and collision tolerance.

Embeddings represent categories as learned dense vectors. They are common in neural networks and recommender systems, especially when categories are numerous and interactions are important. Embeddings can capture similarity, but they require sufficient data, careful regularization, and monitoring for drift. Their dimensions should not be interpreted as inherently meaningful without additional analysis.

3. Scaling, Standardization, and Normalization

Scaling affects distance-based, gradient-based, and regularized models particularly strongly. It is less essential for many decision-tree models, which split according to ordering rather than magnitude, although scaling can still support consistent pipelines.

Min–max scaling

For a feature x with training minimum x_min and maximum x_max, min–max scaling to [0, 1] is:

The denominator must be nonzero. The transformation is sensitive to outliers, and future observations may fall outside [0, 1] unless they are clipped—an action that should be documented because clipping discards magnitude information.

Standardization

Standardization produces a feature with training mean 0 and training standard deviation approximately 1:

Here μ is the training mean and σ > 0 is the training standard deviation. The result is dimensionless. Standardization does not make a distribution normal; it only changes location and scale. The population or sample definition of standard deviation must be consistent within an implementation, but the same fitted parameters must be applied to future data.

For a sample of n observations, the sample standard deviation is commonly:

where x̄ is the sample mean. Robust alternatives use the median and interquartile range:

This is often more stable under extreme outliers, although it may be less effective when the feature has very small dispersion or a highly discrete distribution.

Vector normalization

Vector normalization scales each observation rather than each column. For an observation vector x = (x₁, …, x_p), L2 normalization is:

provided ||x||₂ ≠ 0. L1 normalization uses ||x||₁ = ∑|x_j|. This is useful for text vectors or compositional profiles when the direction or relative composition matters more than total magnitude. It is not a substitute for standardizing columns in ordinary tabular data.

4. Mathematical and Distributional Transformations

Logarithmic transformation

For strictly positive x, the natural-log transformation is x' = ln(x). For nonnegative data with zeros, ln(1 + x) is often used. It compresses large values, can reduce right skew, and converts multiplicative changes into additive differences. It is not appropriate for negative values without a carefully justified alternative.

A log transformation changes interpretation. In a regression model with ln(x) as a predictor, a one-unit increase in ln(x) corresponds to multiplying x by e, not adding one original unit.

Power transformations

The Box–Cox family for positive values is:

The parameter λ may be estimated using a criterion such as approximate normality or likelihood, but the transformation should be fitted on training data only. Box–Cox requires strictly positive observations. The Yeo–Johnson transformation extends power transformation to zero and negative values, but its piecewise definition should be delegated to a validated implementation rather than reproduced casually.

Transformations should be selected because they support a model or a scientific interpretation, not merely because they improve a univariate normality test. Many modern models do not require normally distributed predictors, and transforming a variable can damage interpretability or distort a meaningful scale.

Ratios, rates, and percentages

Ratios can express domain relationships, such as utilization:

The denominator must be meaningful and nonzero. A ratio can be statistically unstable when the denominator is small, and it may duplicate information already present in the numerator and denominator. Consider retaining denominator size, applying a minimum exposure threshold, or using a hierarchical model where uncertainty matters.

Percentages are often bounded by 0 and 1. A logit transformation, ln(p/(1−p)), is defined only for 0 < p < 1 and may require boundary treatment. Such treatment must be substantively justified rather than applied automatically.

5. Discretization and Binning

Discretization maps a continuous variable into intervals. Equal-width bins divide the numerical range into intervals of the same width; equal-frequency bins aim for similar numbers of observations per interval. Domain-defined bins, such as age bands established by policy, may be more interpretable.

Binning can reduce sensitivity to noise, represent nonlinear thresholds, and aid communication. It also discards within-bin information, creates boundary discontinuities, and can produce unstable results when cut points are learned from small samples. Cut points must be fitted on training data and versioned. Supervised binning, which uses the target to choose boundaries, requires cross-validation or out-of-fold procedures to prevent leakage.

A continuous variable and its bins may be used together when a model needs both smooth trend and threshold effects, but collinearity and interpretability should be assessed. In many cases, splines or monotonic nonlinear models preserve more information than hard bins.

6. Feature Construction

Interactions and polynomial terms

An interaction represents the possibility that the effect of one feature depends on another. A linear model may include:

where β₃ measures the additional contribution of the product term under the model’s assumptions. Including an interaction often makes the main-effect coefficients conditional: β₁ is the effect of x₁ when x₂ = 0. Centering variables can make this reference point more meaningful, but centering does not remove the interaction.

Polynomial features such as x² allow curvature, but high degrees can overfit and create numerical instability. Regularization, validation, and domain constraints are essential.

Aggregation and event history

Useful features often summarize sequences or groups:

• number of events in the previous 7, 30, or 90 days;
• mean, median, maximum, or variability of recent measurements;
• time since the last event;
• customer-level totals and rates;
• rolling averages and exponentially weighted summaries.

Every aggregate requires an as-of time. For a prediction at time t, the feature must use only records available at or before t, subject to the operational definition of availability. A transaction posted later but backdated to t may still be unavailable at prediction time.

Date and time features

Dates can be decomposed into year, month, day of week, hour, elapsed time, season, or holiday indicator. Cyclic variables require care because ordinary integers make December and January appear far apart. A common representation for a cycle of length P is:

where t is the position in the cycle. These two features preserve adjacency around the cycle. Time-zone normalization, daylight-saving changes, calendar definitions, and business-day conventions must be specified.

Text, spatial, and graph features

Text can be represented with token counts, TF–IDF, embeddings, or language-model representations. Spatial data may yield distance, region, density, travel time, or neighborhood aggregates. Graph data may yield degree, centrality, community membership, or neighborhood statistics. Each constructed feature requires attention to data availability and leakage: a future graph state or a post-outcome review cannot be used to predict an earlier event.

7. High-Cardinality Variables

High-cardinality variables include customer IDs, product SKUs, postal codes, URLs, and device identifiers. A raw identifier may have no stable predictive meaning and can encourage memorization. One-hot encoding can create enormous sparse matrices; dropping every such variable, however, may discard legitimate information.

A practical decision framework is:

1. Clarify semantics: Is the variable an identifier, a meaningful category, or a composite code?
2. Assess persistence: Will categories recur at prediction time?
3. Measure support: How many observations exist per category?
4. Choose representation: grouped categories, frequency, hashing, target encoding, embeddings, or hierarchical features.
5. Validate by time or entity: prevent the model from exploiting repeated entities across otherwise random folds.
6. Monitor novelty and drift: track unseen-category rates and changing category frequencies.

Rare-category grouping can improve stability, but the threshold should be chosen using training data and domain reasoning. Target encoding is useful for categories with repeated observations, yet requires out-of-fold construction and smoothing. Entity-specific features, such as historical purchase count, should be computed only from prior records and should include safeguards for entities with no history.

8. Leakage, Proxies, and Transformation Governance

Target leakage

Target leakage occurs when a feature contains information generated by, or downstream from, the target or the decision process being predicted. Examples include a medical discharge code used to predict admission mortality, a refund status used to predict whether a transaction will be refunded, or a post-default collection action used to predict default.

Leakage may arise even when the feature is not literally the target. A feature must be evaluated using an information-time question: “Would this value be available, in this form, at the moment the prediction is made?”

All fitted transformations—scalers, imputers, category vocabularies, bins, encoders, feature selectors, and dimensionality-reduction maps—must be fitted within the training data in each validation split. A pipeline is safer than manually transforming the full dataset before cross-validation.

Biased proxies and fairness

A feature can be a proxy for a protected attribute even when the protected attribute is excluded. Postal code, language, device type, purchasing behavior, or browsing patterns may encode race, gender, disability, or socioeconomic status. Removing a protected column therefore does not guarantee fairness.

Feature review should ask whether a variable is legally restricted, operationally necessary, causally downstream of discrimination, or likely to reproduce unequal access. Fairness assessment should examine relevant group metrics, such as selection rates, false-positive rates, calibration, and error costs, while recognizing that these criteria can conflict. Governance requires documentation, stakeholder review, and monitoring rather than a purely statistical rule.

Reproducibility and lineage

A production feature should have a definition, owner, source, timestamp semantics, training-window specification, missing-value policy, transformation version, and monitoring rule. Feature stores can support consistency between training and serving, but they do not automatically guarantee non-leakage or fairness.

9. A Reproducible Transformation Workflow

A robust workflow is:

1. Define the prediction target, prediction time, unit of analysis, and permitted information.
2. Audit data types, missingness, ranges, duplicates, outliers, cardinality, and measurement units.
3. Split data using a design appropriate to the problem: random, stratified, grouped, or temporal.
4. Fit all preprocessing parameters on the training portion only.
5. Encode categorical variables with an explicit unknown-category strategy.
6. Scale or transform numerical variables according to model and domain requirements.
7. Construct domain features using as-of logic.
8. Validate feature distributions, leakage risks, fairness implications, and operational availability.
9. Compare against a simple baseline and use nested or appropriately separated validation for feature-selection decisions.
10. Serialize the complete pipeline and monitor drift, missingness, category novelty, and feature-serving failures.

The baseline is important. If a complex feature set barely improves a transparent baseline, its additional operational and governance cost may not be justified.

10. Illustrative Case: Predicting Late Supplier Deliveries

Background

A logistics company wants to predict whether a purchase order will arrive more than two days late at the time the order is confirmed. The data include supplier, product, order time, promised date, historical shipments, route, quantity, and later operational events.

Problem analysis

The initial dataset contains a delivery-completion timestamp, a warehouse receiving code, and a carrier exception status. These fields are highly predictive, but they are recorded after confirmation and therefore cannot be used. They are leakage variables, not legitimate predictors.

Supplier and product identifiers are high-cardinality. One-hot encoding creates thousands of columns, while raw supplier IDs encourage memorization. The team defines historical features using only shipments completed before each order’s confirmation time.

Approach

The pipeline includes:

• one-hot encoding for route type and shipment mode;
• log transformation of order quantity plus one because quantity is nonnegative and strongly right-skewed;
• a supplier historical late-rate feature with a minimum-history indicator;
• counts of supplier shipments in the previous 30 and 90 days;
• days between order and promised delivery;
• cyclic encoding of order day-of-week;
• an “unknown supplier” treatment for suppliers with no prior history;
• a time-based validation split to imitate deployment.

For a supplier with 20 prior shipments, 5 of which were late, and an overall training late rate of 0.10, suppose the smoothing parameter is λ = 10. The smoothed historical late rate is:

This value is less extreme than the unsmoothed rate of 0.25. In implementation, the encoding for each training record must be calculated from eligible prior records or through a leakage-safe out-of-fold method.

Outcome

The time-based model performs better than the baseline while showing a smaller performance gap across major supplier regions. The organization rejects a feature based on “carrier exception status” despite its high importance because it is unavailable at scoring time. Monitoring is established for category novelty, supplier-history coverage, and changes in delivery patterns.

Learning points

The case illustrates that feature engineering is a temporal and governance problem as much as a modeling problem. Historical features can be valuable, but their eligibility must be defined record by record. High-cardinality handling should preserve generalizable supplier behavior rather than memorize identifiers. A feature’s operational validity is more important than its apparent predictive power.

11. Limitations and Contemporary Developments

Automated feature engineering systems can generate transformations, interactions, aggregations, and representations at scale. Automated machine-learning tools are useful for exploration, but search spaces can produce spurious patterns, leakage, and opaque pipelines. Human review remains necessary for temporal validity, fairness, units, and domain plausibility.

Deep learning reduces manual engineering in images, text, audio, and some sequential problems by learning representations directly. It does not eliminate the need for feature design in tabular, causal, regulated, or resource-constrained settings. Learned representations can still encode proxies, exploit shortcuts, and fail under distribution shift.

Modern practice increasingly emphasizes feature contracts, point-in-time-correct data generation, data-centric testing, drift monitoring, and causal or fairness-aware feature review. The strongest systems combine domain expertise with empirical validation rather than treating feature engineering as an unlimited search for predictive correlations.

Chapter Summary

Data transformation changes the representation of observations so that statistical and machine-learning algorithms can use them effectively. Nominal variables generally require one-hot, hashing, frequency, target, or learned representations; ordinal variables require justified ordering. Scaling and standardization alter magnitude but do not guarantee normality, while vector normalization acts across features within each observation. Logarithmic and power transformations can reduce skew, but they must respect domain constraints and interpretability.

Discretization can express thresholds but loses information. Feature construction includes interactions, polynomial terms, temporal and cyclic variables, ratios, rolling summaries, and domain-specific aggregates. High-cardinality variables require semantic analysis, leakage-safe estimation, and monitoring. The central governance requirement is point-in-time correctness: every fitted parameter and constructed feature must be available without using validation, test, future, or post-outcome information. Fairness review must consider biased proxies as well as explicit protected attributes. A reproducible pipeline, meaningful baseline, appropriate validation design, and continuous monitoring are essential for professional deployment.

Key Terms

Feature engineering: Construction or transformation of variables to improve representation for analysis or modeling.

Encoding: Conversion of categorical or symbolic values into numerical or model-compatible representations.

One-hot encoding: Representation using one binary indicator per category.

Ordinal encoding: Numerical representation of categories with a justified order.

Frequency encoding: Replacement of a category by its observed frequency or count.

Target encoding: Replacement of a category by a target-related statistic, usually with leakage controls.

Smoothing: Shrinkage of unstable category estimates toward a global estimate.

Standardization: Transformation using a training mean and standard deviation.

Min–max scaling: Linear transformation of a feature to a specified range.

Vector normalization: Scaling each observation vector according to a norm.

Robust scaling: Scaling based on resistant statistics such as median and interquartile range.

Discretization: Conversion of a continuous variable into intervals or bins.

Interaction: A feature representing dependence of one variable’s effect on another.

High-cardinality variable: A categorical variable with many distinct levels.

Target leakage: Use of information unavailable at prediction time or generated downstream of the target.

Point-in-time correctness: Requirement that a feature use only information available at the prediction timestamp.

Proxy variable: A variable that indirectly represents a protected or otherwise sensitive attribute.

Feature drift: Change in a feature’s distribution or meaning over time.

Feature pipeline: Ordered, reproducible sequence of fitting and transformation operations.

Review Questions

Short Answer Questions

1. Why is integer encoding usually inappropriate for nominal categories?
2. Distinguish standardization from normalization.
3. State two risks of target encoding.
4. Why must training and validation transformations be fitted separately?
5. What is point-in-time correctness?

Long Answer Questions

1. Compare one-hot encoding, hashing, target encoding, and embeddings for high-cardinality variables.
2. Explain the assumptions, advantages, and limitations of log, power, and ratio transformations.
3. Discuss how date and time variables can be transformed without imposing false numerical relationships.
4. Explain how feature engineering can create biased proxies even when protected attributes are removed.
5. Design a complete preprocessing workflow for a temporal prediction problem and justify its validation strategy.

Analytical / Application Questions

1. A fraud model uses a merchant’s historical fraud rate. Explain how to construct this feature for training and test records without leakage, including treatment of merchants with no history.
2. A dataset contains income, debt, credit limit, and utilization. Propose transformations and constructed features, identify denominator and outlier risks, and explain which models require scaling.
3. A model’s random cross-validation score is substantially higher than its time-based score. Diagnose possible representation, leakage, entity-overlap, and drift explanations, and propose an investigation plan.

Multiple Choice Questions with Answers

Question 1

Which representation is generally most appropriate for a nominal variable with a small number of categories?

A. One-hot encoding

B. Arbitrary integer encoding

C. Logarithmic transformation

D. L2 vector normalization

Correct Answer: A. One-hot encoding

Question 2

In target encoding, what is the principal purpose of smoothing?

A. To remove all missing values

B. To shrink unreliable category estimates toward a global estimate

C. To force every category to have equal frequency

D. To guarantee that categories are ordered

Correct Answer: B. To shrink unreliable category estimates toward a global estimate

Question 3

A standardized value is computed as z = (x − μ)/σ. What does σ represent?

A. The training standard deviation

B. The number of categories

C. The target probability

D. The interquartile range only

Correct Answer: A. The training standard deviation

Question 4

Which transformation is especially suitable for representing cyclic adjacency between December and January?

A. Integer month codes only

B. One-hot encoding of every date

C. Sine and cosine features

D. Replacing month with its rank in alphabetical order

Correct Answer: C. Sine and cosine features

Question 5

A feature calculated using a customer’s purchases after the prediction timestamp is an example of:

A. Regularization

B. Target leakage

C. Robust scaling

D. Ordinal encoding

Correct Answer: B. Target leakage

Question 6

For a positive feature with training minimum 10 and maximum 30, what is the min–max-scaled value of 20 for the range [0, 1]?

A. 0.25

B. 0.50

C. 0.67

D. 1.00

Correct Answer: B. 0.50

Question 7

Which statement about standardization is correct?

A. It guarantees a normal distribution

B. It uses parameters that should be fitted on training data

C. It is identical to one-hot encoding

D. It always improves tree-based models

Correct Answer: B. It uses parameters that should be fitted on training data

Question 8

Why can raw customer IDs be problematic as predictive features?

A. They always have missing values

B. They may encourage memorization and fail for new customers

C. They cannot be stored numerically

D. They necessarily cause target leakage

Correct Answer: B. They may encourage memorization and fail for new customers

Question 9

For a nonnegative variable with many zeros and strong right skew, which transformation is often a reasonable candidate?

A. ln(1 + x)

B. ln(x) without handling zero

C. Division by zero

D. Alphabetical encoding

Correct Answer: A. ln(1 + x)

Question 10

What is the main requirement when constructing a rolling historical feature for deployment?

A. Use all records in the database regardless of date

B. Use only information available by the prediction time

C. Calculate it from the test target

D. Fit it separately after deployment for each prediction

Correct Answer: B. Use only information available by the prediction time