9. Feature Engineering and Data Representation

Introduction and Context

A machine-learning model does not learn directly from business problems, scientific observations, images, documents, or sensor streams. It learns from a numerical representation of those phenomena. The process of converting raw data into informative, reliable, and model-ready variables is known as feature engineering. The broader design of how objects, events, relationships, and signals are encoded for computation is referred to as data representation.

Feature engineering is often the difference between a technically sophisticated model and a useful model. A predictive algorithm may be unable to detect a pattern if the pattern is hidden by an unsuitable scale, an inappropriate encoding, irrelevant variables, or a representation that discards important structure. Conversely, a relatively simple model can perform strongly when the inputs express the underlying problem well.

This chapter examines the transformation of numerical, categorical, temporal, spatial, textual, and high-dimensional data. It considers encoding, scaling, feature construction, feature selection, extraction, dimensionality reduction, validation, interpretability, and governance. Particular attention is given to leakage, reproducibility, fairness, and the trade-off between compact representations and meaningful representations.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Explain the role of features and representations in machine learning.

2. Select appropriate transformations for numerical, categorical, temporal, spatial, and text data.

3. Compare scaling, encoding, feature selection, extraction, and dimensionality-reduction methods.

4. Construct a leakage-resistant feature-engineering workflow.

5. Evaluate representation choices in relation to model assumptions, interpretability, computation, and fairness.

6. Apply feature-engineering techniques to a realistic organizational problem.

7. Identify common technical, ethical, and governance risks in engineered data.

From Raw Observations to Learning Representations

A feature is a measurable attribute used by a model. In a customer dataset, account age, transaction frequency, and service plan may be features. The target or response variable is the outcome to be predicted. A feature vector represents one observation as an ordered collection of values, often written as

\[

x_i = (x_{i1}, x_{i2}, ..., x_{ip})

\]

where observation \(i\) has \(p\) features. A dataset is therefore not merely a table; it is a chosen representation of a phenomenon. Different representations can make the same problem easy or difficult to learn.

Feature engineering includes at least four activities. First, existing variables may be cleaned and transformed. Second, new variables may be constructed from domain knowledge, such as profit margin derived from revenue and cost. Third, variables may be selected or removed. Fourth, a new representation may be extracted from the original data, as in principal component analysis or a text embedding.

A useful feature generally has several properties: it is available at the time of prediction, is measured consistently, contains information about the target, is sufficiently reliable, and can be computed in production. These conditions are more important than merely achieving a high training-set association. A feature that is highly predictive because it records information from after the prediction event is not a valid predictive feature.

Representation, model, and inductive bias

Every model imposes assumptions, sometimes called inductive bias, about the structure of useful patterns. A linear model assumes that additive relationships may be adequate unless features are transformed. A distance-based method is sensitive to scale and the geometry of the feature space. A tree-based model can often discover threshold effects and interactions without explicit scaling. Neural networks may learn representations automatically, but they still depend on appropriate input normalization, tokenization, image preparation, and data quality.

Thus, feature engineering is not a preliminary clerical task. It is part of modeling itself. The same raw variable may require different treatment for logistic regression, k-nearest neighbors, gradient-boosted trees, and a neural network.

Numerical Data: Cleaning, Transformation, and Scale

Numerical variables may be continuous, discrete, bounded, count-based, or heavily skewed. A first step is to inspect distributions, units, outliers, missingness, and possible measurement errors. A salary variable, for example, may be right-skewed, whereas a satisfaction score may be bounded between 1 and 5.

Scaling methods

Standardization converts a value to a z-score:

\[

z = \frac{x - \mu}{\sigma}

\]

where \(\mu\) and \(\sigma\) are calculated from the training data. The transformed variable has approximately zero mean and unit variance. Standardization is often useful for linear models with regularization, support vector machines, principal component analysis, and neural networks.

Min–max scaling maps values to a specified interval, commonly [0, 1]:

\[

x' = \frac{x-x_{min}}{x_{max}-x_{min}}

\]

It is intuitive for bounded inputs but can be strongly affected by extreme observations.

Robust scaling uses the median and interquartile range, making it less sensitive to outliers:

\[

x' = \frac{x-\text{median}(x)}{IQR(x)}

\]

Scaling is not universally required. Decision trees partition variables by thresholds and are usually invariant to monotonic rescaling. Applying unnecessary transformations can reduce interpretability without improving performance.

Transforming skewed variables

A logarithmic transformation, such as \(\log(1+x)\), compresses large values and is useful for non-negative variables with long right tails, including transaction amounts and population counts. The square-root transformation may be suitable for count data. Power transformations, including Box–Cox and Yeo–Johnson methods, can make distributions more symmetric. Such transformations should be selected according to the measurement process and the model, not applied mechanically.

Outliers and missing values

An outlier may be an error, a rare but valid event, or evidence of a different population. Blind deletion can remove the very cases that matter operationally. Alternatives include winsorization, robust transformations, separate outlier indicators, or models designed to tolerate unusual values. Missing values should be analyzed for mechanism: missing completely at random, missing at random, or missing not at random. Median imputation is simple, but it can reduce variability. More advanced methods include model-based imputation, multiple imputation, and explicit missingness indicators.

All imputation parameters must be learned from the training partition. Computing a global median before splitting the data allows information from the validation or test set to influence the training process.

Categorical Variables and Encodings

Categorical variables represent labels rather than quantities. Nominal categories such as payment method have no natural order. Ordinal categories such as low, medium, and high do have an order, but the distance between levels may not be equal.

Common encodings

One-hot encoding creates one binary column for each category, often omitting one reference category in a regression model to avoid perfect multicollinearity. It is transparent and effective for low-cardinality nominal variables. However, it can create a very wide matrix when cardinality is high.

Ordinal encoding maps ordered categories to integers. It should not be used for nominal categories because numerical codes may falsely imply meaningful distances.

Frequency encoding replaces a category with its occurrence frequency or proportion. It is compact but may merge categories with different meanings.

Target encoding replaces a category with a statistic of the target, such as the category-specific mean. It can be powerful for high-cardinality variables, but it is highly vulnerable to target leakage. The statistic must be calculated within training folds, with smoothing or regularization for rare categories.

Hashing maps categories into a fixed number of columns using a hash function. It controls dimensionality and supports streaming, but collisions can make interpretation difficult.

Learned embeddings represent categories as dense vectors. They are especially useful when categories are numerous and relationships among them may be learned from data. They require careful validation and are less directly interpretable than one-hot variables.

A production system also needs a policy for previously unseen categories. An “unknown” bucket, hashing, or a controlled fallback prevents prediction failure when new values appear.

Temporal and Spatial Representation

Time is not simply another numeric variable. A timestamp contains multiple structures: ordering, duration, periodicity, seasonality, and recency.

Useful temporal features include hour of day, day of week, month, elapsed time since an event, rolling averages, event counts within a window, and lagged values. For cyclic variables, direct integer coding is misleading: hour 23 and hour 0 are close in time but far apart numerically. A cyclic encoding represents the variable using sine and cosine:

\[

x_{sin}=\sin(2\pi t/P), \qquad x_{cos}=\cos(2\pi t/P)

\]

where \(P\) is the period, such as 24 hours or 7 days.

Temporal feature engineering must respect chronology. Randomly shuffling time-dependent observations can allow future information to enter the past. Time-based splits, rolling-origin evaluation, and carefully defined observation windows are essential. A rolling average must use only values available before the prediction timestamp.

Spatial data can be represented through coordinates, distances, regions, neighborhoods, density, travel time, or spatial relationships. Latitude and longitude should not automatically be treated as ordinary independent numerical variables. Useful transformations include distance to a facility, geohash or grid cells, counts within a radius, and regional aggregates. Spatial autocorrelation can cause nearby observations to be unusually similar, so random train-test splitting may overstate generalization. Grouped or spatially separated validation is often more appropriate.

Spatial representations also raise privacy issues. Precise location can reveal residence, workplace, health visits, or religious activity. Aggregation and access controls may be necessary even when location improves prediction.

Text as Data

Text is unstructured from the perspective of a relational table, but it contains linguistic structure. The first decisions concern normalization, segmentation, tokenization, handling of punctuation, numbers, spelling variation, and language. Excessive cleaning can remove useful signals such as negation, emphasis, or domain terminology.

A bag-of-words representation records token counts and ignores word order. Term frequency–inverse document frequency (TF–IDF) gives greater weight to terms that are frequent in one document but uncommon across the collection:

\[

TFIDF(t,d)=TF(t,d)\times \log\left(\frac{N}{DF(t)}\right)

\]

where \(N\) is the number of documents and \(DF(t)\) is the number containing term \(t\). N-grams preserve limited local order and can capture phrases such as “late delivery.”

Dense language embeddings represent words, sentences, or documents in a continuous vector space. Transformer-based models generate context-sensitive representations and can be adapted to classification, retrieval, summarization, and question answering. These methods may improve semantic performance, but they introduce computational cost, opacity, bias, and data-governance concerns.

Text preprocessing must also avoid leakage. For example, constructing a vocabulary or selecting terms using the entire dataset before cross-validation allows validation documents to influence training. The vectorizer and feature selector should be fitted only on the training fold.

Selecting and Extracting Features

Feature selection retains a subset of the original variables. Feature extraction creates new variables from existing data. The distinction matters for interpretation: selected features retain their original meaning, whereas extracted components may be combinations that require additional explanation.

Filter, wrapper, and embedded methods

Filter methods rank variables independently of a chosen model using correlation, mutual information, chi-square statistics, variance thresholds, or statistical tests. They are computationally efficient but may miss interactions.

Wrapper methods evaluate subsets by repeatedly training a model. Recursive feature elimination is an example. These methods can capture model-specific utility but are computationally expensive and prone to overfitting if validation is not nested correctly.

Embedded methods perform selection during training. L1 regularization can drive some coefficients to zero, while tree-based importance measures can identify useful variables. Importance measures should be interpreted cautiously because correlated variables can divide or exchange importance.

A feature should not be removed solely because its univariate association is weak. It may contribute through an interaction, nonlinear effect, or subgroup pattern. Conversely, a statistically significant variable may have negligible practical value.

Dimensionality reduction

Principal component analysis (PCA) transforms correlated variables into orthogonal components. The first component captures the greatest possible variance, the second captures the greatest remaining variance, and so on. PCA is useful for compression, visualization, denoising, and some distance-based methods. Its components are linear combinations, so interpretation may be difficult. PCA also emphasizes variance, which is not necessarily the same as predictive relevance.

Linear discriminant analysis seeks projections that separate known classes and is supervised, unlike PCA. Nonlinear methods such as t-SNE and UMAP are useful for exploratory visualization, but distances and cluster sizes in a two-dimensional map should not automatically be treated as proof of real structure. Autoencoders learn nonlinear low-dimensional representations through neural networks and may be useful when large datasets and complex patterns justify their cost.

Dimensionality reduction should be fitted within the training data. A component calculated using all observations can leak information into evaluation data. The number of dimensions should be selected using validation performance, reconstruction error, computational needs, and interpretability—not by a single universal rule.

A Leakage-Resistant Feature-Engineering Workflow

A disciplined workflow begins with the prediction question and the moment at which a prediction must be made. The analyst defines the unit of analysis, target, observation window, prediction horizon, data availability constraints, and success criteria.

The next stage is data auditing: checking types, units, missingness, duplicates, category drift, outliers, and possible proxies for sensitive attributes. Exploratory analysis is conducted on the training data or within a strictly controlled exploratory protocol. The data is then divided into training, validation, and test sets using random, temporal, grouped, or spatial splitting as appropriate.

Transformations are implemented as a reproducible pipeline. The pipeline fits scalers, imputers, encoders, vectorizers, selectors, and reducers on training data and applies the learned parameters to validation, test, and production data. Cross-validation is used to compare alternatives, while nested cross-validation is valuable when feature selection or hyperparameter tuning is extensive.

Evaluation should compare a baseline with increasingly sophisticated representations. A baseline may use simple imputation and one-hot encoding. More advanced alternatives should demonstrate improvement that is stable across folds and relevant subgroups. The final pipeline is then assessed once on the untouched test set and monitored after deployment.

Applied Case: Predicting Subscription Churn

Background

Consider an illustrative case involving a digital learning platform with individual and institutional subscribers. Management wants to identify accounts at risk of cancelling during the next 30 days so that support teams can intervene.

Situation and problem

The raw data contains plan type, account creation date, login events, course completion, support tickets, payment history, device type, country, and free-text support conversations. The first prototype achieves high validation performance, but the model fails after deployment and performs unevenly across regions.

Analysis

Investigation reveals three problems. First, the team calculated “number of support tickets in the next 30 days,” which includes future information and creates leakage. Second, country was one-hot encoded, producing sparse but manageable variables, while hundreds of institution identifiers were target-encoded using the entire dataset. Third, the random split placed events from the same account in both training and validation, making performance look better than it was for genuinely new accounts.

The data science team defines the prediction timestamp as the end of each week. Only information available by that timestamp can be used. Churn is defined as cancellation during the following 30 days. Accounts are divided by customer identity so that records from one account do not cross partitions. A temporal holdout is also retained to evaluate later behavior.

Approach and solution

Numerical variables such as weekly sessions and payment amount are inspected for skew. A log transformation is applied to payment amount, and robust scaling is used for a regularized logistic-regression benchmark. Missing payment information is imputed within the training folds, with a missingness indicator.

Plan type and device type use one-hot encoding with an unknown category. Account age is derived from the prediction timestamp rather than from the current date. Recency, frequency, and rolling engagement features use only historical events. Cyclic encoding represents day-of-week activity. Institution identifiers are excluded from the first production model because they may act as unstable proxies; a carefully cross-fitted, smoothed encoding is evaluated separately.

Support text is represented first with TF–IDF unigrams and bigrams. A later experiment uses a pretrained language representation, but the simpler representation is retained for the operational model because it is easier to audit. Feature selection removes extremely rare terms and uses training-fold information only.

The team compares logistic regression, a tree-based model, and a calibrated gradient-boosting model. Evaluation includes area under the precision–recall curve, calibration, recall among the highest-risk accounts, intervention capacity, and performance by region and customer type.

Outcome

The corrected evaluation produces a lower but credible estimate of performance. More importantly, performance remains stable on the temporal holdout, probability estimates are usable by the retention team, and the model no longer depends on future support activity. The organization introduces monitoring for category drift, changes in text vocabulary, missingness, and subgroup calibration.

Learning points

The case demonstrates that feature engineering is inseparable from problem definition. Leakage can arise from an apparently sensible variable, while an encoding that is statistically effective may be operationally unstable or ethically problematic. Validation design, not only algorithm selection, determines whether a representation is trustworthy.

Challenges, Risks, and Professional Judgement

The most common error is optimizing features against a random validation score without checking whether the representation will exist at prediction time. Other mistakes include fitting preprocessing before splitting, encoding nominal variables as ordered integers, ignoring unknown categories, scaling variables unnecessarily, treating missingness as purely technical, and selecting features using the test set.

High-dimensional representations create risks of overfitting, storage cost, slow inference, and unstable explanations. Correlated predictors make coefficient interpretation difficult. Automated feature generation can produce thousands of plausible but spurious interactions. Feature stores and reusable pipelines improve consistency, but a centrally stored feature can still be incorrectly defined or stale.

Ethical concerns arise when features encode protected characteristics or their proxies. Postal codes, browsing behavior, language, and device information may reproduce socioeconomic or demographic disparities. Removing a sensitive variable does not guarantee fairness if correlated proxies remain. Teams should document feature purpose, provenance, access rights, retention, and known limitations. Privacy-preserving aggregation, minimization, anonymization where appropriate, and impact assessments should be considered.

Feature engineering must also respect regulation and organizational governance. Data protection rules may restrict the use of personal data, automated decision-making, or sensitive inferences. In high-impact settings, the organization should be able to explain what information was used, why it was relevant, how it was transformed, and how errors can be challenged.

Current Developments and Best Practice

Modern systems increasingly combine manually engineered features with learned representations. A retail model may combine transaction aggregates, categorical embeddings, product text embeddings, and temporal behavior. Multimodal systems combine text, images, audio, and structured data, but alignment of timestamps, identities, and quality standards becomes more difficult.

Feature stores support reuse, versioning, lineage, and online/offline consistency. Automated machine-learning systems can search transformations and model pipelines, but automation does not remove the need for domain judgment or governance. Retrieval-augmented systems and foundation models have expanded the use of embeddings, although vector similarity should be validated for the intended task and monitored for drift.

Strong practice can be summarized as follows:

• Begin with the prediction moment, target definition, and domain mechanism.
• Use validation that reflects deployment: temporal, grouped, or spatial when necessary.
• Fit every learned transformation inside the training process.
• Prefer the simplest representation that achieves the required performance.
• Preserve provenance, versioning, documentation, and reproducibility.
• Test robustness across time, data sources, and relevant subgroups.
• Monitor drift, missingness, category changes, latency, and post-deployment outcomes.
• Treat interpretability, privacy, and fairness as design requirements rather than final-stage repairs.

Chapter Summary

Feature engineering converts raw observations into variables that a learning algorithm can use. Numerical data may require imputation, robust treatment of outliers, transformations, or scaling. Categorical data can be represented through one-hot, ordinal, frequency, target, hashing, or embedding approaches, each with different assumptions and risks. Temporal and spatial data demand representations that preserve ordering, cyclicity, proximity, and appropriate validation. Text can be encoded using counts, TF–IDF, n-grams, or learned embeddings.

Feature selection retains original variables, whereas feature extraction creates new representations. PCA, supervised projections, nonlinear visualization methods, and autoencoders reduce dimensionality but involve trade-offs between compression, predictive utility, and interpretation. A reliable workflow defines the prediction point, prevents leakage, fits transformations within training folds, evaluates realistic deployment conditions, and monitors data and model behavior after release. Technical performance must be balanced with stability, fairness, privacy, governance, and operational usefulness.

Key Terms

Feature: A measurable input variable used by a model.

Feature vector: An ordered numerical representation of one observation.

Data representation: The computational form used to encode an object, event, or signal.

Feature engineering: The construction, transformation, and preparation of variables for learning.

Standardization: Transformation using a variable’s mean and standard deviation.

Robust scaling: Scaling based on the median and interquartile range.

One-hot encoding: Representation of categories as separate binary indicator variables.

Ordinal encoding: Mapping ordered categories to numerical codes.

Target encoding: Replacing a category with a target-derived statistic.

Data leakage: Use of information during training that would not be available at prediction time.

Cyclic encoding: Sine–cosine representation of periodic variables.

TF–IDF: Text weighting that emphasizes terms frequent in a document but uncommon in the corpus.

Feature selection: Choosing a subset of existing variables.

Feature extraction: Creating new variables from existing data.

Embedded method: Feature selection performed as part of model training.

Principal component analysis: An unsupervised linear transformation ordered by explained variance.

Dimensionality reduction: Representing data with fewer variables or dimensions.

Embedding: A learned dense vector representation of an object or category.

Feature drift: Change in the distribution or meaning of a feature over time.

Feature pipeline: A reproducible sequence of transformations applied to data.

Review Questions

Short Answer Questions

1. Why does the choice of data representation influence model performance?

2. Distinguish nominal and ordinal categorical variables.

3. State one reason for using sine–cosine encoding for time.

4. What is the principal difference between feature selection and feature extraction?

5. Why must imputation parameters be learned only from training data?

Long Answer Questions

1. Compare standardization, min–max scaling, and robust scaling, including situations in which each may be appropriate.

2. Explain one-hot, target, hashing, and embedding representations for high-cardinality categorical variables.

3. Discuss how temporal and spatial structure should influence data splitting and feature construction.

4. Explain PCA and contrast it with supervised dimensionality reduction.

5. Design a complete, leakage-resistant feature-engineering workflow for a classification problem.

Analytical / Application Questions

1. A bank uses a customer’s average balance over the month following a loan application to predict loan default. Identify the methodological problem and propose a valid alternative.

2. A hospital dataset contains thousands of diagnosis codes, irregular timestamps, missing laboratory values, and patient records repeated across visits. Recommend representations and a validation strategy, justifying each choice.

3. A text-based hiring model performs well overall but poorly for applicants using a particular dialect. Analyze possible representation and governance causes and propose an evaluation and remediation plan.

22. Multiple Choice Questions with Answers

Question 1

A feature is most appropriate for a deployed prediction model when it is:

A. Highly correlated with the target, regardless of when it is measured

B. Available at prediction time and computed consistently

C. Selected only because it has a large numerical range

D. Derived from the test set to maximize accuracy

Correct Answer: B. Available at prediction time and computed consistently

Question 2

Which transformation is generally most appropriate for a heavily right-skewed non-negative monetary variable?

A. Logarithmic transformation such as log(1+x)

B. Nominal one-hot encoding

C. Random permutation

D. Replacing every value with its rank only

Correct Answer: A. Logarithmic transformation such as log(1+x)

Question 3

Why can integer encoding be inappropriate for a nominal variable such as payment method?

A. It always creates missing values

B. It prevents all tree models from training

C. It may falsely imply order and distance between categories

D. It necessarily increases the number of columns

Correct Answer: C. It may falsely imply order and distance between categories

Question 4

Which procedure prevents leakage when using target encoding during cross-validation?

A. Calculate category means using the complete dataset before splitting

B. Calculate category means separately within each training fold

C. Use the test target to smooth rare categories

D. Encode every category as zero

Correct Answer: B. Calculate category means separately within each training fold

Question 5

A variable represents hour of day. Which representation best reflects the closeness of 23:00 and 00:00?

A. Use the raw integer hour only

B. Use one arbitrary label for all hours

C. Use sine and cosine of the hour’s periodic position

D. Replace hour with the record identifier

Correct Answer: C. Use sine and cosine of the hour’s periodic position

Question 6

Which statement about PCA is correct?

A. It selects original variables without changing them

B. It creates orthogonal components ordered by explained variance

C. It guarantees the best classification accuracy

D. It is supervised because it always uses the target

Correct Answer: B. It creates orthogonal components ordered by explained variance

Question 7

A rolling average used to predict demand at time t should be calculated from:

A. Values occurring after time t

B. All values in the complete dataset

C. Values available no later than time t

D. Only the test-set values

Correct Answer: C. Values available no later than time t

Question 8

Which method is especially sensitive to feature scale?

A. A distance-based nearest-neighbor method

B. A decision tree using threshold splits

C. A rule that checks one categorical label

D. A database sort operation

Correct Answer: A. A distance-based nearest-neighbor method

Question 9

In the churn case, splitting records by account identity was important because it:

A. Guarantees that churn never occurs

B. Prevents related records from the same account crossing partitions

C. Eliminates the need for a test set

D. Converts text into numerical vectors

Correct Answer: B. Prevents related records from the same account crossing partitions

Question 10

Why can removing a sensitive attribute fail to eliminate unfairness?

A. Models cannot use any remaining variables

B. Correlated variables may act as proxies for the sensitive attribute

C. Scaling automatically reintroduces the attribute

D. One-hot encoding always creates demographic labels

Correct Answer: B. Correlated variables may act as proxies for the sensitive attribute