Data Quality, Preparation and Feature Engineering

Introduction

Data science models learn from representations of observations, not from business concepts directly. A predictive system therefore depends on two linked capabilities: producing data that are trustworthy and constructing variables that expose useful structure. Poor-quality data can create biased estimates, unstable models, invalid conclusions and operational failures. Conversely, well-designed preparation and feature engineering can improve accuracy, interpretability, computational efficiency and fairness—often more than changing the modelling algorithm.

Data preparation is not a purely mechanical preprocessing stage. It is an analytical process involving measurement, provenance, assumptions, domain knowledge and decisions about what information may legitimately be available at prediction time. The central principle is that every transformation must be reproducible, justified and fitted only on information available in the appropriate training context.

Learning Objectives

By the end of this chapter, the reader should be able to:

• profile data quality using statistical and operational diagnostics;
• distinguish missingness mechanisms and select defensible treatments;
• detect and manage duplicates, inconsistent values, outliers and anomalous records;
• integrate heterogeneous datasets while preserving keys, grain and provenance;
• construct numerical, categorical, text and temporal features;
• address class imbalance, target leakage and unwanted bias;
• build reproducible preprocessing pipelines and evaluate their limitations;
• assess data preparation decisions from technical, ethical and governance perspectives.

1. Data Quality as Fitness for Use

Data quality is multidimensional. A dataset is not simply “clean” or “dirty”; its suitability depends on the intended use. Common dimensions include:

A useful distinction is between syntactic validity and semantic validity. “2025-02-30” fails a calendar validity test, whereas “age = 250” may be syntactically numeric but semantically implausible. A value can also be accurate yet inappropriate: a customer’s address may be correct historically but not current enough for delivery prediction.

Data profiling

Profiling begins before transformation. For each column, inspect data type, distinct-value count, missingness, quantiles, frequency distribution, units, minimum and maximum values, and representative records. Profiling should also be stratified by important groups, time periods and data sources, because global summaries can conceal local failures.

For a variable X with n observations, the missingness rate is:

where m is the number of missing observations. This descriptive quantity has no causal interpretation by itself. A high rate may result from a sensor failure, a question not asked, a value being structurally irrelevant, or a subject deliberately withholding information.

Quality rules should be explicit. Examples include “transaction amount must be non-negative,” “end time must not precede start time,” and “each order line must reference an existing order.” Rules should produce exception reports rather than silently altering values.

2. Missing Data and Imputation

Missingness must be analysed in relation to the data-generating process. Three conventional mechanisms are useful:

• MCAR (Missing Completely At Random): missingness is unrelated to observed and unobserved values.
• MAR (Missing At Random): missingness depends on observed variables, after conditioning on them.
• MNAR (Missing Not At Random): missingness depends on the unobserved value itself or another unmeasured factor.

These are assumptions about a mechanism, not labels that can be proven from a missingness table alone. For example, income may be missing more often among high-income respondents, suggesting a plausible MNAR process.

Treatments include deletion, simple imputation, model-based imputation and explicit missingness indicators. Mean imputation is easy but reduces variance and can distort relationships. Median imputation is more robust to skew. Groupwise imputation can use a legitimate reference group, but small groups may produce unstable estimates. Multiple imputation represents uncertainty by creating several completed datasets and combining estimates; it is generally preferable when inferential uncertainty matters.

A missingness indicator can be defined as:

It may be predictive when the fact that a value is missing carries information. However, it can also encode historical inequity or a workflow artifact. Imputation statistics must be learned from the training partition only. Computing a median from the full dataset before splitting is a form of leakage.

3. Errors, Duplicates and Outliers

Duplicate handling requires understanding the unit of analysis. Two identical rows may be duplicate events, or they may be legitimate repeated purchases. Exact duplicates can be detected mechanically, but near-duplicates require entity resolution using identifiers, normalized names, addresses, timestamps or probabilistic matching. Merging records without a reliable key can create false customers and inflate counts.

Outliers are observations unusually distant from the remainder of a distribution. They may represent errors, rare but valid cases, fraud, a different population or a measurement-unit problem. Removing them solely because they reduce model error is poor practice.

For a univariate variable, the interquartile range is:

A common exploratory flag is a value below Q₁ - 1.5IQR or above Q₃ + 1.5IQR. This is a screening rule, not a universal deletion rule; it is sensitive to skew and does not establish that an observation is erroneous. A robust z-score can be based on the median and median absolute deviation (MAD):

where \tilde{x} is the sample median and MAD = median(|x_i - \tilde{x}|). The factor 1.4826 makes MAD approximately comparable to standard deviation under a normal distribution. Robust methods are useful for detection, but domain validation should determine the response: correct, cap, transform, retain with an indicator, or exclude with a documented reason.

Transformations such as log(1+x) are useful for non-negative right-skewed variables. Standardization uses:

where μ and σ are the training-set mean and standard deviation. Standardization changes scale, not substantive information, and is especially relevant for distance-based, regularized and gradient-based models. Tree-based models generally do not require it.

4. Integration, Granularity and Reproducibility

Integration combines data from multiple sources. Before joining tables, establish:

1. Entity definition: What real-world object does each row represent?
2. Grain: Is a row a customer, order, order line, account-day or sensor reading?
3. Key quality: Are keys unique, stable and complete?
4. Time semantics: Does the joined record use information available at the prediction time?
5. Provenance: Which source, extraction date and transformation produced each field?

A one-to-many join can silently multiply rows. If customer data are joined to transactions and then modelled at customer level, transaction variables should usually be aggregated first—for example, count, mean, recency or total amount. The join should be audited by comparing row counts, match rates and unmatched-key samples before and after integration.

A reproducible pipeline separates raw, validated, transformed and model-ready data. It should version code, configuration, schemas, source snapshots where permitted, feature definitions and training metadata. Deterministic transformations should be idempotent: applying them twice should not produce a different result. Random operations require recorded seeds, although a seed does not compensate for uncontrolled data or software versions.

5. Feature Engineering Principles

A feature is a measurable representation supplied to a model. Good features are relevant to the prediction target, available at decision time, sufficiently stable, and interpretable enough for their use. Feature engineering should begin with a causal and operational question: what information would a decision-maker genuinely possess when the prediction is made?

Numerical features

Common transformations include scaling, ratios, differences, logarithms, ranks, bins, winsorization and interaction terms. Ratios require careful denominators: a rate such as cost per unit is undefined or unstable when the unit count is zero. A small constant may be added only when substantively defensible, not merely to hide a data problem.

Polynomial and interaction features can express non-linear relationships. For variables X and Z, an interaction XZ allows the effect of one variable to vary with the other. Such expansions increase dimensionality and can produce multicollinearity. Regularization, domain constraints and validation are therefore important.

Categorical features

One-hot encoding represents a category with binary indicators. It is suitable for low- or moderate-cardinality nominal variables. Ordinal encoding is appropriate only when categories possess a genuine order; assigning numbers to unordered labels creates artificial distances. High-cardinality variables may use frequency encoding, carefully cross-fitted target encoding, hashing, learned embeddings or domain aggregation.

Target encoding replaces a category with an estimate related to the target. Naive target encoding is highly leakage-prone because an observation can influence its own encoded value. Out-of-fold encoding and smoothing are required. A smoothed category mean can be expressed as:

where n_c is the category count, \bar{y}_c its sample target mean, μ the global target mean, and α ≥ 0 a smoothing strength. This formula is applicable to training data only through cross-fitting when used for model evaluation.

Text features

Text preprocessing may include normalization, tokenization, handling of punctuation, language detection and removal of boilerplate. Stop-word removal and stemming are task-dependent and can discard meaning. The bag-of-words model represents documents by token counts; TF–IDF reduces the influence of terms common across documents. For term t in document d:

where tf(t,d) is a term-frequency measure, N is the number of documents and df(t) is the number containing t. Exact variants differ in normalization and smoothing. Sparse linear models with TF–IDF remain strong baselines; transformer embeddings can capture context but require attention to domain shift, computational cost, privacy and representation bias.

Temporal features

Time must be treated as ordered information. Useful features include year, month, weekday, elapsed time, recency, rolling averages, lagged values and seasonality indicators. Cyclical variables such as hour of day can be represented as:

where h is the hour on a 24-hour cycle. This representation makes 23:00 and 00:00 close in feature space. Rolling statistics must use only prior observations. Random train-test splitting is inappropriate when it permits future information to influence training or when performance is intended for future deployment; temporal or rolling-origin validation is safer.

6. Imbalance, Leakage and Bias

Class imbalance occurs when one outcome is much less frequent than another. Accuracy can be misleading: a classifier predicting the majority class for every case may achieve high accuracy while detecting no positives. Report confusion matrices and metrics such as precision, recall, F1 score, balanced accuracy, area under the precision–recall curve and calibrated probabilities according to the decision context.

Let TP, TN, FP and FN denote true positives, true negatives, false positives and false negatives. Then:

The F1 score is the harmonic mean:

These denominators must be non-zero. Resampling methods include random oversampling, undersampling and synthetic methods such as SMOTE. Resampling must occur inside each training fold, never before the split, and synthetic examples should be generated only from training data. Class weights change the loss or decision emphasis without fabricating observations.

Target leakage occurs when a feature contains information unavailable at prediction time or information generated after the target event. Examples include a post-discharge diagnosis used to predict admission, a “closed account” flag used to predict future churn, and a full-dataset aggregate that includes the target period. Leakage often produces spectacular validation results and disappointing deployment results. A feature availability timestamp and a point-in-time data store are strong controls.

Bias can enter through sampling, measurement, labels, missingness, proxy variables and deployment decisions. Removing a protected attribute does not remove bias if correlated proxies remain. Evaluation should examine error rates, calibration and data quality across relevant groups, while recognizing that fairness criteria can conflict mathematically and operationally. Fairness assessment requires legal, social and domain interpretation, not only a metric.

7. A Reproducible Preparation Workflow

A professional workflow is iterative:

1. Define the prediction task, unit, target, decision time and success criteria.
2. Acquire data with documented provenance and permissions.
3. Profile distributions, schemas, missingness, keys, time coverage and subgroup patterns.
4. Validate rules and quarantine exceptions rather than silently discarding them.
5. Split data according to deployment reality.
6. Fit imputers, encoders, scalers and feature selectors on training data only.
7. Apply the fitted transformations to validation, test and production data.
8. Evaluate predictive performance, stability, calibration, fairness and operational cost.
9. Monitor drift, missingness, schema changes and feature availability after deployment.
10. Version and document every material decision.

A pipeline should distinguish fit from transform. For example, the training median is estimated during fit; the same median is then applied to validation and production. This prevents information from crossing evaluation boundaries. Automated data contracts can specify types, ranges, allowed categories, freshness and null thresholds. Monitoring should alert on both data drift and concept drift: the former concerns input distributions, while the latter concerns changes in the relationship between inputs and outcomes.

8. Illustrative Case: Predicting Subscription Churn

Background and problem

Consider a fictional streaming service seeking to identify customers likely to cancel within the next 30 days. The modelling table is intended to contain one row per customer at a weekly scoring date. The target is cancellation during the subsequent 30 days.

Analysis

The initial data combine customer profiles, billing events, application logs and support tickets. A raw join creates several rows per customer because usage and tickets are event-level data. Some accounts lack support records, and a “cancellation reason” field is completed only after cancellation.

Approach

The team first defines a point-in-time snapshot. Event tables are aggregated using only records occurring on or before the scoring date. Features include days since last session, sessions in the preceding 7 and 30 days, failed-payment count, recent support-contact count and subscription tenure. Missing support events are represented as zero only after confirming that the event table captures all support channels; otherwise they are represented as unknown or accompanied by a data-coverage indicator.

A temporal split is used: earlier weeks for training, later weeks for validation and the latest period for testing. Numeric imputation and scaling are fitted on training data. High-cardinality plan and region fields are encoded using training-only procedures. The post-cancellation reason is excluded as leakage. Because churn is relatively rare, the team reports precision–recall performance, calibration and recall at the capacity of the retention team rather than accuracy alone.

Outcome

The resulting model performs less impressively than the leaked prototype but remains stable on the future test period. Review shows that performance is weaker for customers using a newly launched application version because the event pipeline is incomplete for that version. The organization therefore improves instrumentation before expanding automated offers.

Learning points

The most important improvement came from defining the prediction time and grain, not from selecting a more complex algorithm. Data coverage itself became a monitored feature. The case also illustrates that a model can be statistically adequate yet operationally unsafe if its features are unavailable, unstable or unfairly measured for a subgroup.

9. Limitations, Governance and Contemporary Practice

Preparation decisions can alter the population represented by the data. Deleting incomplete cases may disproportionately remove vulnerable groups. Imputation can create false precision. Embeddings may encode social stereotypes, while automated anomaly removal can suppress legitimate minority behaviour. Privacy law and organizational policy may restrict collection, linkage and retention of personal data. Data minimization, purpose limitation, access controls, audit trails and retention schedules should therefore accompany technical preprocessing.

Modern feature stores and declarative pipeline systems support reuse, lineage and point-in-time correctness, but they do not guarantee valid features. Automated feature engineering can search many transformations and increase multiple-testing and overfitting risks. Large language models can assist with schema documentation or text extraction, but generated transformations require validation, security controls and human accountability. The enduring standard is not novelty but defensible evidence that a feature is valid, available, useful and appropriately governed.

Chapter Summary

Data quality is fitness for a defined purpose across accuracy, completeness, validity, consistency, uniqueness, timeliness and provenance. Effective preparation begins with profiling and explicit validation rules. Missingness mechanisms must be considered before imputation, and imputation must be fitted within the training boundary. Duplicates, outliers and inconsistent records require domain-aware treatment rather than automatic deletion.

Integration depends on grain, keys, temporal semantics and lineage. Feature engineering transforms numerical, categorical, textual and temporal information into representations suitable for modelling, but every feature must be available at prediction time. Class imbalance calls for decision-appropriate metrics and training-only resampling. Leakage is a central threat to valid evaluation, while bias can arise in collection, measurement, labels and deployment. Reproducible pipelines, temporal validation, monitoring and governance convert preparation from an ad hoc activity into an auditable engineering discipline.

Key Terms

Data quality: Fitness of data for a specified use.

Data profiling: Systematic examination of schema, distributions, missingness and validity.

Missingness mechanism: Process determining whether values are absent.

MCAR: Missingness unrelated to observed and unobserved values.

MAR: Missingness explainable by observed variables after conditioning.

MNAR: Missingness related to unobserved values or factors.

Imputation: Estimation or replacement of missing values.

Outlier: An unusually distant observation relative to a reference distribution.

MAD: Median absolute deviation, a robust measure of dispersion.

Data grain: Meaning of one row in a dataset.

Entity resolution: Identification and matching of records referring to the same entity.

Feature engineering: Construction or transformation of variables for analysis or modelling.

Target encoding: Encoding a category using target-related statistics.

TF–IDF: Text weighting based on term frequency and inverse document frequency.

Point-in-time correctness: Use of only information available at the prediction time.

Class imbalance: Unequal prevalence of outcome classes.

Target leakage: Use of information unavailable or inappropriate at prediction time.

Data drift: Change in the distribution of input data over time.

Concept drift: Change in the relationship between inputs and the target.

Data lineage: Record of data origins and transformations.

Review Questions

Short Answer Questions

1. Distinguish data validity from data accuracy.
2. Why must imputation statistics be fitted on training data only?
3. What is the difference between MCAR, MAR and MNAR?
4. Why is random splitting often inappropriate for temporal prediction?
5. State two ways in which target leakage can occur.

Long Answer Questions

1. Explain a complete workflow for profiling, validating and preparing a multi-source dataset.
2. Compare deletion, simple imputation and multiple imputation, including their assumptions and limitations.
3. Discuss numerical, categorical, text and temporal feature engineering techniques.
4. Explain why accuracy is inadequate for many imbalanced classification problems and recommend suitable evaluation practices.
5. Analyse how data preparation decisions can create or mitigate bias and privacy risk.

Analytical / Application Questions

1. A customer table has one row per customer and a transaction table has many rows per customer. Design an integration strategy that preserves the modelling grain and avoids row multiplication.
2. A churn model has excellent random-split performance but poor performance on the next quarter. Identify possible causes and propose a diagnostic and validation plan.
3. A medical dataset has missing income values concentrated in one demographic group. Propose a preparation and fairness assessment strategy, explaining what assumptions would need to be tested.

Multiple Choice Questions with Answers

Question 1

Which data-quality dimension asks whether values conform to defined formats, ranges or domain rules?

A. Validity

B. Timeliness

C. Uniqueness

D. Provenance

Correct Answer: A. Validity

Question 2

Why should an imputer be fitted separately within each training fold?

A. To increase the number of features

B. To prevent information from the validation fold entering training

C. To guarantee normally distributed variables

D. To remove all outliers

Correct Answer: B. To prevent information from the validation fold entering training

Question 3

Which situation is the clearest example of target leakage?

A. Using customer age at prediction time

B. Using a post-cancellation reason to predict cancellation

C. Using training-set median income

D. Using the number of sessions in the previous week

Correct Answer: B. Using a post-cancellation reason to predict cancellation

Question 4

For a 24-hour cyclical feature, which pair of transformations is appropriate?

A. h/24 and h²

B. sin(2πh/24) and cos(2πh/24)

C. log(h) and √h

D. rank(h) and one-hot(h) only

Correct Answer: B. sin(2πh/24) and cos(2πh/24)

Question 5

If TP = 30 and FP = 10, what is precision?

A. 0.25

B. 0.50

C. 0.75

D. 0.90

Correct Answer: C. 0.75

Question 6

Which method is specifically designed to reduce the influence of very frequent terms across a document collection?

A. TF–IDF weighting

B. Ordinal encoding

C. Median imputation

D. Winsorization

Correct Answer: A. TF–IDF weighting

Question 7

What is the principal risk of naive target encoding?

A. It always removes category information

B. It can allow target information from an observation into its own feature

C. It requires numerical targets only

D. It cannot represent high-cardinality variables

Correct Answer: B. It can allow target information from an observation into its own feature

Question 8

Which validation design is generally most appropriate when predicting future outcomes from historical time-ordered data?

A. Random split with no time information

B. Temporal or rolling-origin split

C. Splitting after duplicating all records

D. Using the test target during feature construction

Correct Answer: B. Temporal or rolling-origin split

Question 9

Which statement about the IQR rule is correct?

A. It proves that every flagged observation is erroneous

B. It is a screening rule that requires contextual interpretation

C. It is valid only for categorical variables

D. It guarantees removal of all influential observations

Correct Answer: B. It is a screening rule that requires contextual interpretation

Question 10

What does point-in-time correctness require?

A. Using every available historical and future record

B. Using only information available at the scoring or prediction time

C. Deleting all timestamp fields

D. Replacing temporal validation with random validation

Correct Answer: B. Using only information available at the scoring or prediction time