Data Understanding, Preparation and Feature Engineering

Introduction

Machine-learning performance depends at least as much on data quality and representation as on algorithm selection. A sophisticated model trained on duplicated, biased, leaked, inconsistent, or poorly represented data can produce unreliable predictions. Data preparation is therefore not a clerical preliminary step; it is a form of model design.

This chapter presents an end-to-end approach to acquiring, understanding, cleaning, transforming, sampling, and representing data. Particular attention is given to exploratory data analysis (EDA), feature construction, class imbalance, data leakage, reproducibility, and lineage. The central principle is that every transformation must be justified by the problem context, learned only from appropriate data, and recorded so that it can be reproduced and audited.

Learning Objectives

After studying this chapter, a learner should be able to:

1. Identify suitable data sources and assess their quality, provenance, and legal status.
2. Perform univariate, bivariate, and multivariate exploratory analysis.
3. Detect and manage missing values, duplicates, outliers, inconsistent formats, and erroneous records.
4. Select appropriate encoding, scaling, transformation, and sampling procedures.
5. Construct useful features while preserving semantic validity and avoiding leakage.
6. Address class imbalance using principled methods and suitable evaluation metrics.
7. Design reproducible preprocessing pipelines with data lineage and governance controls.

1. Data Acquisition and Data Quality

1.1 Sources and data contracts

Data may be acquired from transactional databases, application programming interfaces, sensors, surveys, documents, experiments, public repositories, or third-party providers. Before acquisition, define a data contract: the expected fields, types, units, permissible ranges, update frequency, ownership, security classification, and meaning of the target variable.

A data dictionary should specify, for each variable, its name, business meaning, storage type, measurement unit, allowed values, missing-value convention, and source system. For example, “income” is incomplete without knowing whether it means annual gross income, monthly net income, or an imputed estimate.

Important quality dimensions include:

Data quality is contextual. A missing postal code may be tolerable for a recommendation task but unacceptable for a location-based emergency model.

1.2 Sampling and representativeness

A dataset is a sample from a target population. If the sample differs systematically from that population, model performance may not generalize. Sampling error concerns random variation, whereas sampling bias arises from the selection process itself. A very large non-representative sample can remain seriously biased.

For a population proportion estimated by a simple random sample, an approximate standard error is:

Here, p is the population proportion, p̂ is its estimate, and n is the sample size. The approximation assumes independent observations and is most useful when the sample is reasonably representative. In practice, data scientists should also examine coverage, non-response, temporal drift, and subgroup representation.

Stratified sampling divides the population into groups and samples within each group. It can improve representation of small but important segments. Random sampling is often suitable for independent records, whereas time series require chronological splitting to prevent future information from entering the past.

2. Exploratory Data Analysis

EDA combines numerical summaries, visualizations, and domain reasoning. It is not merely a search for correlations; it is an investigation of how the data was generated and whether its structure is compatible with the intended task.

2.1 Univariate analysis

For numerical variables, inspect count, missingness, minimum, maximum, mean, median, standard deviation, quantiles, and skewness. The arithmetic mean is:

The sample standard deviation is:

The divisor n − 1 provides the usual unbiased estimator of population variance under independent, identically distributed sampling. For highly skewed data, the median and interquartile range (IQR) may be more informative than the mean and standard deviation. The IQR is Q₃ − Q₁, where Q₁ and Q₃ are the first and third quartiles.

Categorical variables should be examined using frequency and proportion tables. Rare categories may indicate genuine minority groups, data-entry errors, or excessive granularity.

2.2 Relationships among variables

Bivariate analysis compares a predictor with the target or with another predictor. Useful tools include scatter plots, grouped box plots, cross-tabulations, conditional means, and correlation measures. Pearson correlation measures linear association:

Its value lies between −1 and 1. It is sensitive to outliers and does not establish causation. A correlation near zero can coexist with a strong nonlinear relationship. Spearman correlation, based on ranks, is often more appropriate for monotonic but non-linear relationships or ordinal data.

Multivariate analysis examines interactions, redundancy, clusters, and subgroup differences. A variable may appear harmless globally but become highly predictive within one subgroup. EDA should therefore be conducted not only overall but also across time, geography, demographic groups, and operational segments.

Figure: A leakage-aware data preparation pipeline.

3. Cleaning and Data Quality Management

3.1 Missing values

Missingness may be completely at random, related to observed variables, or related to the unobserved value itself. These mechanisms are commonly described as MCAR, MAR, and MNAR. The mechanism matters: replacing all missing values with a mean can conceal informative non-response.

Common strategies include:

• deletion, when missing records are few and plausibly random;
• constant-value or statistical imputation;
• group-wise imputation using domain-relevant strata;
• model-based or multiple imputation;
• an explicit missingness indicator when absence itself may carry meaning.

For a numeric feature, training-set median imputation is robust to extreme values. The imputation value must be calculated using training data only and then applied unchanged to validation and test data.

3.2 Duplicates, inconsistency, and invalid records

Duplicates can arise from repeated exports, retries in distributed systems, or legitimate repeated events. Deduplication requires a definition of record identity; blindly dropping identical rows may remove valid repeated transactions. Standardize dates, units, spelling, and categorical labels before aggregation. Validate domain rules such as age ≥ 0, an end date not preceding a start date, and total amount equalling the permitted sum of components.

3.3 Outliers

An outlier may be an error, a rare but valid event, or evidence of a new regime. Detection methods include robust IQR rules, z-scores, isolation-based methods, and domain thresholds. A common descriptive rule flags values below Q₁ − 1.5 IQR or above Q₃ + 1.5 IQR, but it is not a universal deletion rule. Alternatives include correction, winsorization, log transformation, robust scaling, separate modelling, or retention with an outlier indicator.

4. Transformation and Encoding

4.1 Numerical transformations and scaling

Standardization transforms a feature as:

where μ and σ are the training-set mean and standard deviation. It produces unitless values with mean approximately zero and standard deviation approximately one, assuming σ > 0. It is especially important for distance-based, gradient-based, and regularized models.

Min–max scaling is:

It maps training values to approximately [0, 1], but is sensitive to extreme values and can produce values outside that interval for future observations. Robust scaling uses the median and IQR and is less affected by outliers. Tree-based models generally do not require scaling because their splits depend on order rather than distance.

Logarithmic transformation can reduce right skew, but ordinary log(x) requires positive values. For non-negative data containing zeros, log(1 + x) is often used; negative values require a separately justified transformation.

4.2 Categorical encoding

Nominal categories can be represented using one-hot encoding. For k categories, one-hot encoding creates binary indicators. With an intercept in an ordinary linear regression, one category is commonly omitted to avoid perfect multicollinearity (the dummy-variable trap). Regularized models may tolerate redundant columns, but the modelling objective and implementation should determine the choice.

Ordinal encoding is appropriate only when category order is meaningful. Target encoding replaces a category with a target-related statistic and can be powerful for high-cardinality variables, but it is highly vulnerable to leakage. It must use out-of-fold or strictly training-only estimates, with smoothing where appropriate.

High-cardinality identifiers such as customer IDs usually identify entities rather than explain transferable behaviour. They should not be used as ordinary predictive features without a clear generalization argument.

5. Feature Engineering

Feature engineering converts raw observations into variables that expose relevant structure to a learning algorithm. Examples include age from date of birth, tenure from start date, rolling transaction totals, ratios, interaction terms, text representations, and domain-based aggregations.

A feature should satisfy four tests:

1. Availability: Is it known at prediction time?
2. Validity: Does it represent the intended concept?
3. Stability: Will its meaning and distribution persist?
4. Value: Does it improve out-of-sample performance or interpretability?

Ratios require special care. A ratio with a near-zero denominator can be unstable, and a mathematically defined ratio may not have a meaningful business interpretation. Interaction terms such as x₁x₂ can represent conditional effects, but they increase dimensionality and may need regularization.

For time-dependent data, derive lagged or rolling features only from prior observations. If a moving average over the previous w periods is used, it may be written as:

The exclusion of x_t is essential when predicting the value at time t. Feature construction should be implemented in a pipeline so that training and production apply identical logic.

6. Data Leakage and Validation Design

Leakage occurs when information unavailable at prediction time influences model training or evaluation. It can be direct, such as including a post-outcome field, or indirect, such as calculating a global mean before splitting the data.

Typical leakage sources include:

• imputing or scaling using the entire dataset;
• selecting features using the test target;
• random splitting of longitudinal records;
• duplicate entities appearing across train and test sets;
• target encoding without out-of-fold discipline;
• features generated after the decision event;
• preprocessing performed before cross-validation.

A robust workflow first defines the prediction timestamp, then splits data according to deployment conditions. Random splits may be suitable for independent observations; temporal splits are needed for forecasting; group splits are needed when the same person, household, patient, or device could otherwise occur in multiple partitions.

Cross-validation estimates generalization by repeatedly fitting on training folds and evaluating on held-out folds. Every learned operation—including imputation, scaling, feature selection, and encoding—must be fitted separately within each training fold. A pipeline formalizes this requirement and reduces operational inconsistency.

7. Imbalanced Data

In binary classification, let TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives. Accuracy is:

For rare events, accuracy may be misleading. Precision and recall are:

The F1 score is their harmonic mean:

These measures assume the relevant denominators are non-zero. Specificity is TN/(TN+FP). The choice of metric should reflect the costs of errors. Precision–recall curves are often informative when the positive class is rare; ROC curves remain useful but can appear optimistic under extreme imbalance.

Methods for imbalance include class-weighted learning, random undersampling, oversampling, and synthetic methods such as SMOTE. Oversampling must occur inside training folds, never before the train–test split. Synthetic examples may be implausible in sparse or mixed-type data, and undersampling can discard important majority examples. Threshold adjustment is often preferable when the model ranks cases well but the default decision threshold is inappropriate.

8. Data Lineage, Reproducibility, and Governance

Data lineage records where data originated, how it was transformed, which code and parameters were used, and which model consumed it. A production-grade dataset should have a version, schema, timestamp, owner, quality report, and transformation history. Reproducibility additionally requires controlled random seeds where possible, environment specifications, dependency versions, and immutable evaluation data.

Governance includes privacy, access control, retention, consent, and responsible use. Personal data should be minimized and protected. Sensitive attributes may be legally restricted, yet excluding them does not guarantee fairness because proxy variables can remain. Fairness analysis should examine performance and error rates across relevant groups, while respecting privacy and statistical reliability. Data preparation must also document whether labels reflect historical institutional decisions that may encode discrimination.

9. Applied Case: Credit-Card Fraud Detection

Background

Consider an illustrative payment platform processing millions of card transactions daily. Fraud represents a small minority of transactions, but missed fraud is costly and excessive false alarms inconvenience customers.

Problem

The platform wants to score each transaction at authorization time. Available fields include amount, merchant category, device information, account history, and transaction time. Labels become available only after investigation, sometimes days later.

Analysis

Initial EDA reveals strong class imbalance, repeated authorization messages, missing device identifiers, and different transaction volumes by country. A field called “investigation outcome” has high predictive power but is unavailable at authorization and must be excluded. Randomly splitting transactions would also place transactions from the same account in both partitions, overstating generalization.

Approach

The team first removes technical duplicates using transaction and event identifiers. It defines the prediction timestamp and creates lagged account features from historical transactions only: count of transactions in the preceding 24 hours, amount relative to the account’s historical median, and time since the previous transaction. Missing device identifiers receive an indicator and a training-set-derived imputation where suitable.

The evaluation uses a temporal holdout and a group-aware diagnostic by account. Class-weighted learning is compared with training-fold oversampling. Performance is reported using precision, recall, PR-AUC, calibration, and an operational cost matrix rather than accuracy alone. The decision threshold is selected according to review capacity and the relative cost of missed fraud and false alerts.

Outcome

The final pipeline is stored with its schema, feature definitions, training period, code version, and monitoring rules. Production monitoring tracks missingness, feature drift, alert rate, subgroup error rates, and delayed-label performance. Any feature whose availability changes after deployment triggers a review.

Learning points

The case demonstrates that feature availability, temporal correctness, duplicate control, imbalance handling, threshold selection, and lineage are inseparable from model quality. A high offline score is not credible unless it reproduces the information conditions of deployment.

10. Best Practices and Limitations

A disciplined preparation process should:

• define the prediction unit, target, and prediction time;
• profile schema, missingness, duplicates, distributions, and subgroup coverage;
• split data according to deployment reality;
• fit all learned transformations only on training data;
• preserve raw data while creating versioned derived datasets;
• test pipelines with known edge cases;
• monitor drift, missingness, validity, and performance after deployment;
• document assumptions, exclusions, and ethical risks.

No preprocessing method is universally optimal. Imputation can introduce bias, outlier rules can remove legitimate rare cases, synthetic sampling can distort class boundaries, and feature selection can privilege historical patterns that later change. Preparation is therefore iterative: domain experts, data engineers, statisticians, and model developers must evaluate both statistical performance and operational consequences.

Chapter Summary

Data understanding begins with a clear data contract, provenance assessment, and definition of the target population. Exploratory analysis uses summaries and visualizations to reveal distributions, relationships, errors, subgroup differences, and temporal structure. Cleaning addresses missingness, duplicates, invalid values, inconsistent representations, and outliers while preserving valid rare events.

Transformation includes numerical scaling, skew reduction, and categorical encoding. Feature engineering creates representations aligned with the prediction task, but every feature must be available at prediction time and semantically defensible. Leakage prevention requires deployment-realistic splitting and training-only fitting of preprocessing operations. Imbalanced classification requires metrics and decision thresholds linked to error costs rather than reliance on accuracy. Finally, lineage, reproducibility, privacy, fairness, and monitoring make data preparation suitable for professional machine-learning systems.

Key Terms

Data contract: Formal specification of fields, types, units, quality rules, and ownership.

Data dictionary: Description of variables, meanings, formats, and permissible values.

Data lineage: Record of data origins and transformations.

Exploratory data analysis: Systematic investigation of data using summaries, visualizations, and domain reasoning.

Missingness: Absence of an expected value.

Imputation: Replacement of missing values using a defined method.

Outlier: Observation unusually distant from a distribution or expected domain range.

Standardization: Transformation using training mean and standard deviation.

Min–max scaling: Rescaling based on observed minimum and maximum.

One-hot encoding: Representation of categories by binary indicator variables.

Target encoding: Encoding based on target-related statistics.

Feature engineering: Construction or transformation of predictive variables.

Data leakage: Use of information unavailable at prediction time.

Cross-validation: Repeated training and evaluation over data partitions.

Class imbalance: Unequal representation of target classes.

Precision: Proportion of predicted positives that are true positives.

Recall: Proportion of actual positives that are detected.

Concept drift: Change in the relationship between inputs and the target over time.

Pipeline: Ordered, reproducible sequence of preprocessing and modelling operations.

Review Questions

Short Answer Questions

1. Why should preprocessing parameters be learned from training data only?
2. Distinguish data validity from data accuracy.
3. When is ordinal encoding inappropriate?
4. Why can accuracy be misleading for rare-event classification?
5. What information should a data dictionary contain?

Long Answer Questions

1. Explain a leakage-aware workflow for preparing data for supervised learning.
2. Compare deletion, statistical imputation, and model-based imputation, including their assumptions and risks.
3. Discuss how temporal, group, and random splitting differ and when each is appropriate.
4. Explain the advantages and limitations of standardization, min–max scaling, and robust scaling.
5. Evaluate oversampling, undersampling, class weighting, and threshold adjustment for imbalanced classification.

Analytical / Application Questions

1. A hospital model uses a feature recording whether a patient received a specialist referral within 30 days after admission. Explain why this may constitute leakage and propose a valid alternative.
2. A fraud dataset has 1% positive cases. A classifier produces TP = 80, FP = 120, FN = 20, and TN = 9,780. Calculate precision, recall, and F1, and explain which operational information is still needed before selecting a threshold.
3. Design a feature-engineering and validation strategy for predicting monthly customer churn when customers appear repeatedly over time. Specify the prediction timestamp, split method, leakage controls, and monitoring plan.

Multiple Choice Questions with Answers

Question 1

Which operation must be fitted using training data only?

A. Computing a global mean for imputation

B. Assigning a fixed column name

C. Removing an impossible negative age

D. Reading a schema description

Correct Answer: A. Computing a global mean for imputation

Question 2

Which metric is the harmonic mean of precision and recall?

A. Accuracy

B. F1 score

C. Specificity

D. Mean squared error

Correct Answer: B. F1 score

Question 3

Which encoding is appropriate when categories have a genuine order?

A. Ordinal encoding

B. Random encoding

C. Pixel encoding

D. Hashing the target

Correct Answer: A. Ordinal encoding

Question 4

What is the principal danger of target encoding without fold discipline?

A. Underflow

B. Data leakage

C. Unit conversion

D. Duplicate column names

Correct Answer: B. Data leakage

Question 5

Which split is generally most appropriate for forecasting future outcomes?

A. Purely random split

B. Temporal split

C. Alphabetical split

D. Split by feature magnitude only

Correct Answer: B. Temporal split

Question 6

For a binary classifier with TP = 80 and FP = 20, what is precision?

A. 0.20

B. 0.50

C. 0.80

D. 0.90

Correct Answer: D. 0.90

Question 7

Which model family generally does not require feature scaling for split selection?

A. Decision trees

B. Nearest-neighbour models

C. Many gradient-based linear models

D. Support-vector models with distance-based kernels

Correct Answer: A. Decision trees

Question 8

Why is a post-outcome investigation field unsuitable for authorization-time fraud prediction?

A. It has too few categories

B. It is unavailable at prediction time

C. It is always numerical

D. It prevents one-hot encoding

Correct Answer: B. It is unavailable at prediction time

Question 9

What does data lineage primarily document?

A. Only model accuracy

B. Data origin and transformations

C. Only user passwords

D. The number of neural-network layers

Correct Answer: B. Data origin and transformations

Question 10

Which method can reduce the effect of extreme values while retaining observations?

A. Blind row deletion

B. Robust scaling

C. Random label replacement

D. Test-set oversampling

Correct Answer: B. Robust scaling