Feature Engineering and Variable Selection

Introduction and Context

Predictive models learn patterns from the variables supplied to them. However, raw business data is rarely arranged in the form most suitable for modelling. A transaction date may contain information about seasonality, a customer address may need geographical encoding, and a revenue measure may require transformation because its distribution is highly skewed. The disciplined process of creating useful predictors from raw data is called feature engineering. The process of identifying a smaller, more informative subset of predictors is called variable selection or feature selection.

These activities are central to data-driven decision making because model quality depends not only on the algorithm but also on the representation of the problem. Good features can improve predictive accuracy, interpretability, computational efficiency and operational usefulness. Poor features can introduce bias, instability, unfairness or target leakage. Feature engineering is therefore both a technical and a business activity: it requires statistical judgement, knowledge of the data-generating process and understanding of how decisions will be made.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish observations, variables, features and targets;
• transform numerical and categorical data appropriately;
• impute missing values and create informative missingness indicators;
• encode categorical variables and scale numerical variables;
• construct date, time, interaction, ratio and aggregation features;
• apply filter, wrapper and embedded selection methods;
• control dimensionality while avoiding overfitting and data leakage;
• assess feature quality, fairness, stability and operational feasibility.

1. From Raw Variables to Model Features

An observation is one unit being analysed, such as a customer, invoice or daily store record. A variable is a measurable attribute of an observation. A feature is a model-ready representation of a variable or a combination of variables. The target, often written as y, is the outcome to be predicted. The feature vector for observation i may be written as:

Here, p is the number of candidate features. A feature matrix X has one row per observation and one column per feature. Feature engineering maps raw data R into a usable representation:

The function φ represents the engineering pipeline. It should be learned using the training data and then applied unchanged to validation, test and production data. This prevents information from the future or from held-out observations entering the modelling process.

A useful feature generally has four properties:

1. Predictive relevance: it contains information related to the target.
2. Availability: it is available at the time the prediction is required.
3. Reliability: its definition and measurement are sufficiently stable.
4. Operational usefulness: its use is affordable, interpretable and compatible with business processes.

Correlation with the target is not enough. A feature may correlate strongly because it records an event that occurs after the outcome, or because it is a proxy for a protected characteristic.

2. Data Quality and Numerical Transformation

Missing values

Missingness can arise from non-response, system failure, inapplicability or deliberate business choices. Common approaches include deletion, constant replacement, mean or median imputation, group-based imputation and model-based imputation.

For a numerical variable, median imputation replaces a missing value with the training-set median. The median is often preferable to the mean when extreme values are present. A missingness indicator can be added:

The imputed value and indicator must be calculated from the training partition only. A missing value may itself carry meaning—for example, an absent credit history may indicate a new customer—so replacing it without an indicator can remove useful information.

Deletion is appropriate only when missingness is limited, plausibly random and unlikely to change the population being analysed. Imputation does not make data accurate; it creates assumptions that should be documented.

Outliers and transformations

An outlier is an observation unusually distant from the rest of a distribution. It may represent an error, a legitimate rare event or a change in operating conditions. Blindly removing outliers can eliminate important fraud or failure cases.

Logarithmic transformation is useful for positive, right-skewed variables such as income or transaction value. A common version that accommodates zero is:

where x must be non-negative and the logarithm is natural unless otherwise stated. The transformation compresses large values; it does not make every distribution normal and should not be used for negative values without a suitable alternative.

Winsorisation replaces values below or above selected quantiles with boundary values. Standardisation and robust scaling are described below. All outlier rules should be defined using training data and justified by domain knowledge.

Scaling numerical variables

Standardisation converts a variable to a mean of zero and standard deviation of one:

Here, μ and σ are the training-set mean and standard deviation. The result has no units. Standardisation is particularly important for distance-based methods, regularised regression, principal component analysis and neural networks. It is usually unnecessary for tree-based methods because split decisions depend on order rather than magnitude.

Min–max scaling maps a variable to a chosen interval, commonly [0, 1]:

This requires x_max > x_min. It can be sensitive to future values outside the training range. Robust scaling uses the median and interquartile range:

where IQR = Q₃ − Q₁. Robust scaling is less affected by extreme observations.

Binning and nonlinear representation

Binning replaces a continuous value with intervals such as age bands or spending categories. It can improve communication and capture threshold effects, but it loses information and may create artificial discontinuities. Polynomial terms, splines or monotonic transformations can represent nonlinear relationships more flexibly. These methods increase dimensionality and should be validated rather than assumed to improve performance.

3. Encoding Categorical Variables

Categorical variables represent groups rather than numerical magnitudes. Treating nominal categories such as region codes as numbers can create a false order. Appropriate encoding depends on cardinality, model type and the operational purpose.

One-hot encoding

For a variable with k categories, one-hot encoding creates indicator variables. With an intercept in an ordinary linear model, one category is normally omitted to avoid perfect multicollinearity; this is called reference or dummy coding. For example, a three-category region variable may become two columns, with the omitted category represented when both columns are zero.

One-hot encoding is transparent and effective for low-cardinality nominal variables, but high-cardinality variables can produce many sparse columns.

Ordinal and binary encoding

Ordinal encoding assigns ordered integers only when categories genuinely have an order, such as low, medium and high. Binary encoding is suitable for two categories, such as yes and no. Numerical codes for nominal categories should not be interpreted as quantities.

Frequency and target encoding

Frequency encoding replaces a category with its observed frequency or proportion. It is simple but may conflate categories with equal frequencies. Target encoding replaces a category with an estimate of its average target. For regression, this could be a category-specific mean; for classification, a category-specific event rate.

Target encoding is powerful but highly vulnerable to leakage. It must be computed within training folds, with smoothing or out-of-fold procedures, and never by using the target of the row being encoded. A simple smoothed estimate can be expressed as:

where n_c is the number of observations in category c, μ_c is its observed training target mean, μ is the overall training target mean and λ ≥ 0 controls shrinkage toward the overall mean. The formula is applicable when the target mean is meaningful; classification requires care in defining the event rate and handling class imbalance.

Rare categories may be grouped into an “Other” category, but the threshold should be established using training data and reviewed for business meaning.

4. Time-Based and Domain-Informed Features

Dates are not merely labels. From a timestamp, analysts may derive year, month, day of week, hour, quarter, holiday status, elapsed time and time since a previous event. For a record at time t, recency may be defined as:

Both times must use the same time zone and units. The previous event must occur before the prediction time. A rolling average of a measure v over the previous k periods can be written as:

The exclusion of v_t is essential when v_t is not known at prediction time. Similar lag features include previous-period sales, prior complaint count and days since last purchase.

Cyclical variables require special treatment. Month 12 and month 1 are close in time, although their integer codes are far apart. A cyclical representation is:

where c is the cycle position and P is its period—for example, P = 12 for months or P = 7 for days of the week. Both sine and cosine are needed to preserve the circular geometry.

Domain-informed features combine subject knowledge with data. Examples include:

• revenue per customer;
• defect rate rather than defect count alone;
• debt-to-income ratio;
• stock coverage in days;
• utilisation = used capacity ÷ available capacity;
• customer tenure = prediction date − start date.

Ratios require careful denominators. A zero or near-zero denominator can create undefined or unstable values. The numerator and denominator must refer to comparable periods and populations. Features should be calculated from information available before the decision point.

Aggregations are often useful. Customer-level totals, averages, counts and maximums can summarise transaction histories. However, an aggregation must be performed within the correct time window. A lifetime total calculated after the outcome occurs is leakage when used to predict the earlier outcome.

5. Feature Interactions and Dimensionality

An interaction represents the possibility that the effect of one variable depends on another. In a linear model, an interaction between x₁ and x₂ may be represented by x₁x₂. The coefficient of the interaction is interpreted alongside the main effects; it is not generally correct to interpret it in isolation.

Feature engineering can rapidly increase dimensionality. High dimensionality raises computational cost, increases the opportunity for spurious patterns and may reduce interpretability. The relevant issue is not simply the number of variables but the relationship between the number of candidate features, sample size, noise level and model complexity.

Dimensionality reduction differs from feature selection. Feature selection retains original variables, whereas methods such as principal component analysis create new combinations. Principal component analysis is unsupervised and maximises variance, not necessarily predictive value. Consequently, components should not automatically be assumed to be better predictors or easier for managers to interpret.

6. Variable Selection Methods

Filter methods

Filter methods rank variables before fitting the final predictive model. Examples include variance thresholds, correlation screening, mutual information, chi-square tests and information gain. They are fast and model-independent, making them useful for an initial reduction.

A low variance does not necessarily mean low predictive value, and a low marginal correlation does not rule out a useful nonlinear or interaction effect. Filters should therefore be treated as screening tools, not definitive evidence.

Wrapper methods

Wrapper methods evaluate subsets by repeatedly fitting a model. Forward selection begins with no variables and adds candidates; backward elimination begins with all variables and removes candidates. Recursive feature elimination repeatedly fits a model and removes the least important features.

Wrappers can capture model-specific usefulness but are computationally expensive and can overfit the validation process if too many alternatives are tried without proper nested validation.

Embedded methods

Embedded methods perform selection during model estimation. Lasso regression uses an L₁ penalty:

Here, β_j is the coefficient for feature j, p is the number of features, Loss(β) measures predictive error and λ ≥ 0 controls the penalty. Larger λ generally shrinks coefficients more strongly and may set some exactly to zero. Features should normally be scaled before penalised regression so that the penalty treats coefficients fairly.

Ridge regression uses the L₂ penalty, λ∑β_j², which usually shrinks correlated coefficients but does not set them exactly to zero. Elastic net combines L₁ and L₂ penalties and can be useful when predictors are numerous and correlated. Tree ensembles provide model-based importance measures, but impurity importance can favour high-cardinality variables; permutation importance is often more informative but can still be unstable when predictors are correlated.

7. Leakage, Bias and Validation

Data leakage occurs when information unavailable at prediction time enters the training process. Common sources include fitting a scaler on the entire dataset, imputing with full-data statistics, using post-outcome variables, randomly splitting time-dependent data and calculating target encoding before cross-validation.

A sound workflow is:

1. define the prediction target and prediction time;
2. split data into training, validation and test sets using a design appropriate to the problem;
3. fit imputers, encoders, scalers and selectors on training data only;
4. apply the fitted transformations to validation and test data;
5. tune feature choices using cross-validation within the training data;
6. evaluate once on the untouched test set;
7. monitor feature availability and drift after deployment.

For time-ordered data, use earlier observations for training and later observations for validation. Random splitting can allow future patterns to influence the apparent performance. For grouped data, such as multiple transactions per customer, groups should not be split across partitions when that would allow identity information to transfer.

Selection can also create bias. Removing variables only because they are inconvenient may produce omitted-variable bias in explanatory work. Conversely, including a protected attribute or a strong proxy may reproduce discriminatory historical decisions. A variable can be legally permissible yet ethically problematic if it creates unjustified disparate impact. Governance requires documentation of source, definition, transformation, availability, purpose and known risks for every production feature.

8. A Practical Feature Engineering Pipeline

A professional pipeline should be reproducible, version-controlled and testable. A typical sequence is:

1. Business framing: specify the decision, target, horizon and acceptable error.
2. Data audit: inspect types, missingness, duplicates, ranges, timestamps and unit definitions.
3. Temporal alignment: ensure every feature is available before the prediction timestamp.
4. Baseline: fit a simple model using a small, defensible feature set.
5. Transformation: impute, encode, scale and transform as appropriate.
6. Feature creation: add time, aggregation, domain and interaction features.
7. Selection: remove leakage, duplicates, impossible variables and unstable or redundant features.
8. Validation: compare alternatives using an appropriate metric and resampling design.
9. Interpretation: check coefficients, importance, partial effects and business plausibility.
10. Deployment monitoring: track missingness, distributions, category changes, drift and performance.

A pipeline object in analytics software should store the learned parameters and transformation order. Manual spreadsheet transformations are difficult to reproduce and are especially prone to applying future information accidentally.

Applied Case: Illustrative Retail Churn Prediction

Background

A fictional subscription retailer wants to identify customers likely to cancel within the next 30 days so that retention teams can offer appropriate support. The available data includes customer profiles, monthly invoices, service contacts and cancellation records.

Problem

The initial dataset contains raw signup dates, current invoice status, contact counts, plan type and transaction histories. Analysts must create features without using information generated after the prediction date.

Analysis

Useful candidates include tenure, number of support contacts in the previous 90 days, average payment delay during the previous six months, plan type and recent usage trend. A raw cancellation-processing code is excluded because it is recorded after cancellation. “Total contacts ever” is replaced with contacts in a defined historical window, because future contacts would leak information.

Tenure is calculated as prediction date minus signup date. Payment delay is aggregated only over invoices issued before the prediction date. Plan type is one-hot encoded, and rare plan categories are grouped after examining training data. Missing payment-delay values are accompanied by an indicator because some customers have no prior invoice history.

The data is split chronologically: earlier months form training data, a later month forms validation data and the final month is retained as a test set. The imputer, encoder and scaler are fitted only on training data. A regularised logistic regression provides an interpretable baseline; a tree-based model is then compared using the same partitions.

Outcome

The selected model performs better than the baseline because recency, payment delay and recent usage change capture behaviour not represented in the raw fields. However, the retention team rejects a feature based on a customer’s postcode because it acts as a proxy for socioeconomic characteristics and offers no clear service rationale. The final feature list is slightly less accurate in offline testing but more defensible and easier to explain.

Learning points

The case demonstrates that feature engineering is not a contest to create the greatest number of variables. It is a controlled process of representing timely, relevant and ethically acceptable information. Chronological validation, leakage controls, domain review and operational interpretation are as important as algorithm selection.

Challenges and Contemporary Developments

Feature stores now support the central management and reuse of engineered variables, but they do not automatically guarantee correct time alignment. Automated feature engineering can generate transformations and interactions rapidly, yet it may create unstable or uninterpretable variables. Large language models and other AI systems can assist with data dictionaries, coding and candidate generation, but human review remains necessary for leakage, fairness, privacy and business validity.

Privacy-preserving approaches may restrict the use of granular identifiers or require aggregation. Regulations and organisational policies commonly require purpose limitation, access controls, retention limits, explainability and audit trails. Feature selection should therefore consider not only predictive gain but also data minimisation, security and the consequences of incorrect decisions.

Chapter Summary

Feature engineering converts raw data into model-ready representations, while variable selection controls redundancy, cost and overfitting. Important operations include imputation, transformation, scaling, categorical encoding, date decomposition, cyclical encoding, lagged and rolling features, ratios, aggregations and interactions. Every transformation must respect the prediction time and be fitted using training data only.

Filter methods are fast screening tools, wrapper methods evaluate subsets directly and embedded methods select variables during model fitting. Penalised regression, tree-based importance and dimensionality reduction each have assumptions and limitations. A reliable workflow combines domain knowledge, leakage-safe validation, reproducible pipelines, fairness review and production monitoring. The best feature set is not necessarily the largest or most predictive in an offline experiment; it is the set that is accurate, available, stable, explainable and appropriate for the decision.

Key Terms

Feature: A model-ready representation of information.

Target: The outcome a model is designed to predict.

Feature engineering: Creating or transforming predictors from raw data.

Variable selection: Choosing a subset of candidate predictors.

Imputation: Replacing missing values using a defined rule.

Missingness indicator: A binary feature showing whether an original value was missing.

Standardisation: Rescaling using the training mean and standard deviation.

One-hot encoding: Representing categories with binary indicator columns.

Target encoding: Replacing a category with a target-based estimate.

Feature interaction: A representation in which the effect of one variable depends on another.

Lag feature: A past value used to represent historical behaviour.

Rolling feature: An aggregation over a preceding time window.

Data leakage: Use of information that would not be available at prediction time.

Filter method: Model-independent variable screening.

Wrapper method: Selection based on repeated model evaluation.

Embedded method: Selection performed during model estimation.

Regularisation: Penalising model complexity to control overfitting.

Dimensionality: The number of features used to represent observations.

Feature drift: Change in feature distributions or meaning after deployment.

Review Questions

Short Answer Questions

1. What is the difference between a raw variable and a model feature?
2. Why must imputation statistics be learned from the training set only?
3. When is one-hot encoding preferable to ordinal encoding?
4. What is data leakage, and why does it produce misleading performance?
5. Distinguish filter, wrapper and embedded selection methods.

Long Answer Questions

1. Explain how a complete feature engineering pipeline should be designed for a supervised learning problem.
2. Discuss the advantages and limitations of standardisation, min–max scaling and robust scaling.
3. Explain how time-based features can improve prediction while creating leakage risks.
4. Compare Lasso, Ridge and Elastic Net for variable selection and correlated predictors.
5. Discuss the ethical and governance issues involved in selecting customer features.

Analytical / Application Questions

1. A retailer predicts tomorrow’s demand using daily sales data. Propose three leakage-safe lag or rolling features and explain the required time windows.
2. A categorical variable has 500 rare categories. Compare two encoding strategies and identify the validation precautions required for target encoding.
3. A credit model’s postcode feature improves accuracy but produces substantially different approval rates across groups. Recommend an analytical and governance process before deployment.

Multiple Choice Questions with Answers

Question 1

Which transformation is generally most appropriate for a nominal variable with several categories and no natural order?

A. One-hot encoding

B. Ordinal encoding

C. Standardisation

D. Logarithmic transformation

Correct Answer: A. One-hot encoding

Question 2

In standardisation, what does σ represent in z = (x − μ) / σ?

A. The median

B. The standard deviation

C. The maximum value

D. The number of categories

Correct Answer: B. The standard deviation

Question 3

Why must a rolling average used for prediction at time t normally exclude the current value when that value is not yet known?

A. To reduce the number of features

B. To avoid multicollinearity only

C. To prevent data leakage

D. To make the average equal to zero

Correct Answer: C. To prevent data leakage

Question 4

Which method uses a penalty involving ∑|β_j| and may set coefficients exactly to zero?

A. Ridge regression

B. Lasso regression

C. Principal component analysis

D. Min–max scaling

Correct Answer: B. Lasso regression

Question 5

For a three-category nominal variable in a linear model with an intercept, why is one dummy category commonly omitted?

A. To create perfect correlation with the target

B. To avoid perfect multicollinearity

C. To standardise the categories

D. To increase the number of parameters

Correct Answer: B. To avoid perfect multicollinearity

Question 6

Which is an example of a filter method?

A. Forward selection based on repeated model fitting

B. Lasso estimation

C. Variance-threshold screening

D. Recursive feature elimination

Correct Answer: C. Variance-threshold screening

Question 7

A target-encoding value calculated using the target of the same row being encoded can cause what problem?

A. Underfitting

B. Data leakage

C. Unit conversion

D. Class balancing

Correct Answer: B. Data leakage

Question 8

Which model family is generally least dependent on feature scaling for its split decisions?

A. Decision trees

B. K-nearest neighbours

C. Lasso regression

D. Principal component analysis

Correct Answer: A. Decision trees

Question 9

For monthly cyclical encoding, what value is commonly used for the period P?

A. 2

B. 7

C. 12

D. 365

Correct Answer: C. 12

Question 10

What is the main distinction between feature selection and dimensionality reduction?

A. Selection retains original variables; reduction creates new representations

B. Selection always uses neural networks; reduction never does

C. Selection is supervised; reduction is always supervised

D. Selection increases dimensionality; reduction removes the target

Correct Answer: A. Selection retains original variables; reduction creates new representations