4. Exploratory Analysis and Feature Engineering

Introduction and Context

A predictive model can be mathematically sophisticated and still perform poorly if the data supplied to it are poorly understood or badly represented. Exploratory analysis and feature engineering address this problem. Exploratory analysis investigates the structure, quality, distribution, relationships, and unusual behaviour of data before formal modelling. Feature engineering transforms raw observations into variables that enable an algorithm to recognise useful patterns.

These activities are not merely preliminary housekeeping. They influence the definition of the modelling problem, the choice of algorithms, the interpretation of results, and the reliability of decisions made from predictions. In many practical projects, improvements in feature representation produce greater gains than replacing one predictive algorithm with another.

The chapter assumes that data have been collected and assembled, as discussed in earlier stages of a machine-learning workflow. It focuses on the analytical work required before and during model development, while maintaining a strict distinction between information available at prediction time and information that becomes available only afterwards.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Explain the purpose and principles of exploratory data analysis (EDA).

2. Use visualisation and summary statistics to identify distributions, outliers, missingness, imbalance, and possible data-quality problems.

3. Interpret correlation and association without confusing them with causation.

4. Distinguish feature representation, transformation, selection, extraction, and construction.

5. Apply suitable methods for encoding categorical variables, scaling numerical variables, transforming skewed data, and representing dates and text.

6. Design interaction and domain-informed features while avoiding leakage and unnecessary complexity.

7. Select features using statistical, model-based, and embedded approaches.

8. Evaluate feature-engineering decisions within a reproducible and leakage-resistant modelling pipeline.

Exploratory Analysis as Analytical Reasoning

EDA is an iterative process of asking questions about data and testing those questions through summaries, visualisations, and domain knowledge. It is not a search for attractive graphs. A useful exploration connects each result to a modelling decision.

A first inspection should establish the unit of analysis, the target variable, the observation period, the data-generating process, and the meaning of each field. For example, a row may represent a customer, a transaction, a monthly account, or a customer-month combination. Confusing these levels can create duplicated information or invalid comparisons.

A practical EDA sequence begins with structural checks: dimensions, data types, unique identifiers, duplicate records, missing values, and target availability. It then examines univariate distributions, bivariate relationships with the target, and multivariate structure. Finally, the analyst investigates time order, subgroup differences, and possible leakage.

Univariate examination

For numerical variables, relevant summaries include count, mean, median, standard deviation, quartiles, minimum, maximum, and selected percentiles. The mean is sensitive to extreme values, whereas the median is more robust. A histogram or density plot reveals skewness, multiple modes, gaps, and possible recording errors. Box plots provide a compact view of spread and unusual observations, although a point beyond the conventional 1.5-interquartile-range rule is not automatically erroneous.

For categorical variables, frequency tables and bar charts show dominant categories, rare levels, unexpected labels, and inconsistent spelling. Categories such as “UK”, “U.K.”, and “United Kingdom” may describe the same state but be treated as different values unless standardised.

The target requires special attention. In regression, its distribution may reveal extreme values or heteroscedasticity. In classification, class proportions indicate imbalance and affect the interpretation of accuracy. A model that always predicts the majority class may achieve high accuracy while being useless for the minority class.

Bivariate and multivariate examination

Scatter plots can reveal linear, curved, clustered, or heteroscedastic relationships between numerical variables. Grouped plots and conditional summaries help compare distributions across target classes or business segments. A relationship visible in aggregate data may disappear or reverse within subgroups; this is one reason domain stratification matters.

Correlation matrices are useful screening tools, but they require careful interpretation. Pearson correlation measures linear association and is sensitive to extreme values. Spearman correlation measures monotonic association using ranks and may identify nonlinear monotonic relationships. Neither measure establishes causation, and a low correlation does not prove that a variable is irrelevant to a nonlinear model.

For categorical variables, association can be examined using contingency tables, conditional rates, chi-square statistics, or measures such as Cramér’s V. A numerical correlation matrix is not sufficient for mixed-type datasets. Visual tools such as heat maps, pair plots, box plots by category, and grouped summaries often provide more meaningful evidence.

Missingness and data quality

Missing values may be missing completely at random, missing conditionally on observed variables, or related to the unobserved value itself. The fact that a field is missing can itself carry information. For example, failure to submit an income field may reflect a particular application pathway. Consequently, an analyst should examine missingness by time, target class, source system, and relevant groups before choosing an imputation method.

Common strategies include deletion, constant imputation, mean or median imputation, group-wise imputation, model-based imputation, and explicit missing-category treatment. Each strategy changes the data-generating assumptions. Median imputation is often robust for skewed numerical data, but it should be calculated using the training data only. An imputation indicator can be added when the fact of missingness may be predictive.

Duplicates, impossible values, unit inconsistencies, and temporal contradictions are equally important. A negative age, a transaction dated before account creation, or a temperature recorded in the wrong unit is not simply an outlier; it may indicate a defect in the data pipeline.

From Raw Fields to Predictive Features

A feature is a measurable input used by a predictive model. Feature engineering is the deliberate process of selecting, transforming, combining, or constructing such inputs. A raw field is not necessarily a useful feature. A timestamp may be converted into hour, weekday, month, elapsed time, or a holiday indicator. A transaction amount may be expressed as a ratio to a customer’s average amount or as a count over a recent window.

Feature engineering should satisfy four conditions. First, the feature must be available at the time a prediction is made. Second, it must be computed consistently in training and deployment. Third, it should represent information plausibly related to the target. Fourth, its complexity should be justified by improvement in validated performance, interpretability, or operational value.

Numerical transformations

Scaling places numerical variables on comparable numerical ranges. Standardisation transforms a value by subtracting the training mean and dividing by the training standard deviation. Robust scaling uses the median and interquartile range and is less affected by extreme values. Min–max scaling maps values to a specified interval, often [0,1]. Scaling is particularly important for distance-based algorithms, nearest-neighbour methods, support vector machines, neural networks, and regularised linear models. Tree-based algorithms are generally less sensitive to monotonic scaling.

Skewed variables may be transformed using logarithms, square roots, or power transformations. A log transformation can compress large values and make multiplicative relationships more nearly additive, but it requires care with zero and negative values. The transformation must be selected according to the measurement scale and modelling objective, not applied mechanically.

Quantile transformations and discretisation can sometimes improve robustness or interpretability. Binning a continuous variable may help communicate risk bands, but it discards information and introduces arbitrary boundaries. Continuous representations should normally be retained unless there is a clear statistical or operational reason to bin.

Categorical encoding

Nominal categories have no intrinsic order. One-hot encoding represents each category with a binary indicator, usually omitting one reference level in linear models to avoid perfect multicollinearity. It works well for low- and moderate-cardinality variables but can create many columns.

Ordinal encoding assigns ordered numerical values and is appropriate only when the categories possess a meaningful order, such as low, medium, and high. Assigning numbers to unordered categories can create an artificial relationship.

For high-cardinality variables, alternatives include frequency encoding, hashing, carefully regularised target encoding, and learned embeddings. Target encoding replaces a category with a statistic related to the target, but it is especially vulnerable to leakage. The statistic must be calculated within training folds, with smoothing and, where necessary, out-of-fold procedures. Rare categories may be grouped into an “Other” level when this is substantively reasonable.

Dates, time, and cyclic structure

Dates should rarely be passed as arbitrary integers. Useful derived variables include year, quarter, month, day of week, hour, time since an event, and elapsed time between events. For periodic variables, sine and cosine encodings preserve circular proximity: Sunday should be close to Monday, and 23:00 should be close to 00:00. A simple integer encoding may incorrectly treat the end and beginning of a cycle as far apart.

Time-dependent data also require temporal validation. Random splitting can allow future patterns to influence the training set and produce an unrealistic estimate of performance. Features based on historical aggregates must use only records available before the prediction timestamp.

Text and unstructured fields

Text can be represented through token counts, term-frequency inverse-document-frequency (TF–IDF) values, n-grams, topic representations, or embeddings. The appropriate representation depends on the amount of data, computational resources, language, and required interpretability. Text preprocessing should not erase meaningful information such as negation or domain-specific terminology. Modern language embeddings can capture semantic relationships, but they introduce model-management, bias, privacy, and reproducibility considerations.

Interaction, Aggregation, and Domain-Informed Construction

A model may require a feature that expresses the joint effect of two variables. Interaction features include products, ratios, differences, conditional indicators, and cross-categories. For example, the effect of advertising expenditure may differ by region, while the significance of income may depend on household size. A linear model cannot represent such conditional effects unless suitable interaction terms are supplied. Tree-based methods can discover many interactions automatically, though explicit construction may still improve interpretability or performance.

Ratios must be designed carefully. A ratio can encode a meaningful rate, such as claims per policy-year, but it can also become unstable when its denominator is near zero. Differences can represent change from a baseline, while rolling statistics can represent recent behaviour. Aggregations such as customer-level transaction count, mean purchase value, maximum delay, and number of distinct products are powerful in transactional applications.

Domain knowledge guides which transformations are credible. In manufacturing, vibration summaries and temperature gradients may be more meaningful than raw sensor readings. In finance, utilisation, payment regularity, and recent delinquency may be more informative than a single balance. In healthcare, rates and trends may be more useful than isolated measurements, but clinical and ethical validation is essential.

Feature construction must not encode post-outcome information. If a hospital readmission model uses the total length of stay, and length of stay is known only after discharge, the feature is unavailable when the decision is required. This is target leakage even if the variable has strong predictive power in historical data.

Feature Selection and Dimensional Reasoning

Feature selection retains a subset of existing variables, whereas feature extraction creates new variables, often in a lower-dimensional space. Selection improves efficiency and interpretability; extraction may address high dimensionality or correlated inputs but can reduce direct interpretability.

Filter methods rank features independently of a particular predictive model. Examples include variance thresholds, univariate tests, mutual information, and correlation screening. They are computationally efficient but may miss interactions and can select redundant variables.

Wrapper methods evaluate subsets using a predictive model. Recursive feature elimination is one example. Such methods can be expensive and must be nested within validation to prevent optimistic estimates.

Embedded methods perform selection during model fitting. Lasso regression can shrink some coefficients to zero, while tree-based importance measures can identify useful variables. Importance scores should be interpreted cautiously: correlated variables may share importance, and impurity-based tree importance can favour variables with many possible split points. Permutation importance, calculated on held-out data, often provides a more relevant assessment of predictive contribution.

Principal component analysis (PCA) creates orthogonal linear combinations of centred and usually scaled variables. Components are ordered by explained variance, not necessarily by predictive relevance. PCA can reduce noise and dimensionality, but the resulting components may be difficult to explain. It must be fitted on the training data only, and the same fitted transformation must be applied to validation and production data.

The number of features should be governed by validation performance, stability, cost, fairness, latency, and interpretability—not by a desire to maximise dimensionality reduction or retain every available field.

A Leakage-Resistant Workflow

A sound workflow separates exploration, transformation, validation, and deployment while preserving the information boundary. The target is defined first, including the prediction horizon and decision point. Data are then split into training and testing sets using random, grouped, or temporal methods appropriate to the problem.

All learned transformations—imputation values, category mappings, scaling parameters, target encodings, feature-selection rules, and PCA components—are fitted on the training data. Cross-validation is used for model and feature decisions, and the final test set is reserved for a one-time estimate of generalisation. In software, a pipeline combines preprocessing and modelling so that the same operations occur in the same order during training and inference.

A useful feature-engineering record documents the feature name, definition, source fields, time availability, transformation, missing-value treatment, expected range, owner, and validation checks. Data and feature versions should be tracked. Monitoring should detect changes in distributions, missingness, category frequencies, and relationships with outcomes. A feature that was valid at development time may become unreliable after a process or policy change.

Applied Case: Predicting Subscription Cancellation

Background

Consider an illustrative telecommunications company seeking to identify customers at risk of cancelling within the next 30 days. The data contain customer attributes, service details, monthly invoices, support contacts, and dated usage records.

Problem and initial analysis

The target is defined as cancellation during the 30-day period following a prediction date. Initial exploration shows a moderately imbalanced target, several missing contract fields, highly skewed support-contact counts, and duplicated customer records caused by monthly snapshots. A random row-level split would place records from the same customer in both training and testing sets, creating overly optimistic results.

The analytical team therefore uses a customer-level, time-aware split. It examines cancellation rates by contract type, tenure band, payment method, and recent service interruptions. Correlation analysis shows that monthly charge and premium-plan status are related, but neither alone describes recent dissatisfaction. Visualisations reveal that cancellation risk increases more sharply with repeated service interruptions than with total lifetime support contacts.

Feature strategy

The team constructs tenure in months, recent interruption count during the previous 60 days, change in monthly usage from the preceding period, payment failures during the previous 90 days, and the ratio of support contacts to months active. Missing contract duration is represented with an indicator and an appropriate imputed value. Contract type is one-hot encoded, while high-cardinality device identifiers are excluded because they have little stable business meaning.

A potentially powerful feature, “number of days until cancellation,” is rejected because it is available only after the outcome. Similarly, post-cancellation retention offers are excluded. These decisions reduce apparent historical performance but preserve real-world validity.

Modelling and evaluation

Preprocessing is implemented in a pipeline. Imputation, encoding, scaling, and feature selection are fitted within each training fold. A regularised logistic model provides an interpretable baseline, while a gradient-boosted tree model captures nonlinearities and interactions. Evaluation uses time-based validation and metrics suited to the business objective, including precision among customers contacted, recall at a fixed intervention capacity, calibration, and expected campaign cost.

The results show that recent service interruptions, payment failures, and declining usage contribute more useful information than several demographic fields. The company uses calibrated risk scores to prioritise retention activity rather than automatically labelling customers as certain to cancel.

Outcome and learning points

The principal improvement comes from defining the prediction horizon correctly, preventing customer-level leakage, and constructing temporally valid behavioural aggregates. The case demonstrates that feature engineering is inseparable from business process knowledge. It also shows why a feature should be judged by operational availability, stability, fairness, and cost—not merely by its association with the target in a historical dataset.

Challenges, Risks, and Professional Considerations

Overfitting is a central risk. Trying many transformations and selecting those that perform best on a test set converts the test set into part of the training process. Feature selection, target encoding, and hyperparameter tuning must occur inside cross-validation.

Another risk is unstable representation. A category that appears frequently during development may disappear, while a new category may arise in production. Pipelines require explicit unknown-category handling. Extreme values may reflect legitimate rare cases, fraud, or errors; automatic deletion can remove precisely the cases that matter.

Fairness requires examining whether features act as proxies for protected characteristics and whether missingness or measurement quality differs across groups. Removing a protected attribute does not guarantee fairness if correlated variables remain. Evaluation should therefore include subgroup performance, calibration, error rates, and the consequences of false positives and false negatives. In regulated settings, feature definitions, data provenance, and decision logic may need to be auditable.

Privacy is also relevant. Detailed behavioural aggregates can reveal sensitive personal information. Data minimisation, access control, retention limits, and purpose limitation should guide feature construction. Human review is important when predictions influence credit, employment, healthcare, education, or access to essential services.

Emerging Directions and Best Practices

Automated feature engineering systems can search transformations, interactions, and aggregations, while feature stores support consistent reuse between training and serving. Representation learning and deep embeddings reduce the need for manually designed features in images, text, audio, and complex sequences. However, automation does not remove the need for domain reasoning, leakage controls, governance, or interpretability.

Best practice is to begin with a transparent baseline, define the information boundary, use appropriate validation, and add features incrementally. Every feature should have a documented rationale and a measurable contribution. Prefer stable, available, and interpretable features where performance is comparable. Inspect errors by subgroup and time period, monitor drift after deployment, and periodically remove features that no longer reflect the operational environment.

Chapter Summary

Exploratory analysis establishes what the data represent, how they are distributed, where they are incomplete or inconsistent, and which relationships may support prediction. Visualisation, descriptive statistics, association analysis, and temporal investigation provide evidence for modelling decisions. Feature engineering converts raw fields into valid, informative representations through encoding, scaling, transformation, aggregation, interaction design, and domain-informed construction.

Feature selection may use filter, wrapper, or embedded methods, while feature extraction methods such as PCA reduce dimensionality at the cost of interpretability. All learned preprocessing must be fitted within the training process, and temporal or grouped validation must reflect how predictions will be used. Leakage, overfitting, drift, unfairness, privacy risks, and unstable production pipelines are major professional concerns. Effective feature engineering combines statistical reasoning, computing practice, domain knowledge, and responsible governance.

Key Terms

Feature: A measurable input supplied to a predictive model.

Exploratory data analysis: Systematic investigation of data structure, quality, distributions, and relationships.

Data leakage: Use of information unavailable at prediction time or improperly exposed across validation boundaries.

One-hot encoding: Representation of a categorical variable using binary indicator columns.

Ordinal encoding: Numerical representation of categories with a genuine meaningful order.

Target encoding: Replacement of a category by a statistic derived from the target.

Standardisation: Transformation using the training mean and standard deviation.

Robust scaling: Scaling based on statistics such as the median and interquartile range.

Skewness: Asymmetry in the distribution of a variable.

Interaction feature: A variable representing the joint or conditional effect of two or more inputs.

Aggregation: Summary of multiple records, often across time or entities.

Feature selection: Retaining a subset of existing variables for modelling.

Feature extraction: Creating new variables from existing variables, often in a reduced space.

Regularisation: Penalisation of model complexity to improve generalisation.

Principal component analysis: A method producing orthogonal components ordered by explained variance.

Permutation importance: Change in held-out performance after randomly permuting a feature.

Concept drift: Change over time in the relationship between inputs and the target.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Missingness indicator: A variable recording whether another value was missing.

Pipeline: Ordered, reproducible sequence combining preprocessing and modelling operations.

Review Questions

Short Answer Questions

1. Why should exploratory analysis examine the unit of analysis before modelling?

2. Distinguish Pearson correlation from Spearman correlation.

3. When is ordinal encoding more appropriate than one-hot encoding?

4. Why should scaling parameters be learned from training data only?

5. What is the purpose of a missingness indicator?

Long Answer Questions

1. Explain a complete EDA workflow for a classification problem, including visual, statistical, and temporal analysis.

2. Compare filter, wrapper, and embedded feature-selection methods, including their strengths and risks.

3. Discuss how dates, time windows, ratios, and interactions can be converted into predictive features.

4. Explain target leakage using practical examples and describe methods for preventing it.

5. Evaluate the professional and ethical issues involved in feature engineering for high-impact decisions.

Analytical / Application Questions

1. A fraud dataset contains transaction amount, merchant category, customer identifier, and a field recording whether the transaction was later investigated. Which features would you construct, which would you reject, and why?

2. A model performs well under random cross-validation but poorly after deployment, where data arrive month by month. Diagnose possible causes and propose a revised validation and feature-engineering strategy.

3. A categorical variable has 50,000 levels, many occurring only once. Compare possible encoding strategies and recommend an approach for an interpretable production model.

22. Multiple Choice Questions with Answers

Question 1

A dataset contains monthly records for the same customers. Which validation strategy best prevents information leakage caused by repeated customers?

A. Randomly split individual rows without restrictions

B. Split customers so that each customer appears in only one partition

C. Use only the rows with complete values

D. Shuffle the target labels before splitting

Correct Answer: B. Split customers so that each customer appears in only one partition

Question 2

Which transformation is generally most appropriate for a nominal categorical variable with no meaningful order?

A. Ordinal encoding

B. One-hot encoding

C. Ranking by target mean using the full dataset

D. Replacing categories with arbitrary integers

Correct Answer: B. One-hot encoding

Question 3

Why must target encoding be performed within training folds?

A. To increase the number of categories

B. To prevent target information from entering validation data

C. To ensure every category receives a unique number

D. To eliminate all missing values

Correct Answer: B. To prevent target information from entering validation data

Question 4

Which feature is an example of target leakage in a model predicting hospital readmission at discharge?

A. Age at admission

B. Diagnosis recorded at admission

C. Length of stay known after discharge

D. Number of previous admissions

Correct Answer: C. Length of stay known after discharge

Question 5

Which algorithm family is usually least sensitive to the numerical scale of input features?

A. Nearest-neighbour methods

B. Support vector machines

C. Regularised linear regression

D. Decision-tree-based methods

Correct Answer: D. Decision-tree-based methods

Question 6

A variable represents hour of day. Which representation best preserves the fact that 23:00 and 00:00 are close in time?

A. An arbitrary integer from 0 to 23 only

B. Two variables using sine and cosine

C. A unique identifier for each hour

D. A missing-value indicator

Correct Answer: B. Two variables using sine and cosine

Question 7

What is the main purpose of permutation importance calculated on held-out data?

A. To measure the change in performance when a feature's information is disrupted

B. To replace all correlated variables automatically

C. To scale features to the interval 0 to 1

D. To calculate the target mean for each category

Correct Answer: A. To measure the change in performance when a feature's information is disrupted

Question 8

Which statement about PCA is correct?

A. It always produces features that are easy to interpret

B. It selects variables solely according to their causal importance

C. It creates orthogonal combinations ordered by explained variance

D. It can be fitted on the complete dataset before cross-validation without risk

Correct Answer: C. It creates orthogonal combinations ordered by explained variance

Question 9

A transaction amount is highly right-skewed and contains zeros. Which approach is most defensible?

A. Apply a logarithm directly without checking zeros

B. Consider a suitable zero-aware transformation and validate its effect

C. Delete every value above the mean

D. Convert all values to arbitrary categories

Correct Answer: B. Consider a suitable zero-aware transformation and validate its effect

Question 10

Which feature is most likely to be domain-informed and temporally valid for predicting customer cancellation?

A. A retention offer made after cancellation

B. The final cancellation date

C. Number of service interruptions during the previous 60 days

D. The model's predicted cancellation probability from a future model

Correct Answer: C. Number of service interruptions during the previous 60 days