Problem Formulation, Data and Feature Engineering

Introduction and Context

Machine learning projects rarely fail because an algorithm cannot be executed. They more often fail because an organizational problem has been translated incorrectly, the data do not represent the decision of interest, or features contain information that would not be available at the time of prediction. Problem formulation and data engineering therefore precede model selection.

A successful project connects five elements: a business or scientific decision, a precisely defined prediction or action, an appropriate learning paradigm, a time-valid dataset, and an evaluation procedure aligned with the cost of errors. For example, “use AI to reduce customer churn” is incomplete. A usable formulation might be: “Using information available on 1 June, estimate whether a subscription customer will cancel during the following 30 days, so that the retention team can contact high-risk customers.” This statement identifies the unit of analysis, prediction time, outcome window and intended action.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Translate organizational problems into supervised, unsupervised or reinforcement-learning tasks.
2. Define observations, targets, features, prediction horizons and decision points.
3. Design a reproducible data pipeline from source systems to model-ready data.
4. Detect and control missing data, noise, outliers, imbalance and data leakage.
5. Select suitable encoding, scaling and feature-construction methods.
6. Evaluate whether engineered features are valid, useful, ethical and maintainable.

1. From Organizational Question to Learning Task

1.1 The formulation template

A robust formulation answers the following questions:

The distinction between a target and a business objective is important. A model may predict cancellation accurately while producing little financial value if interventions are expensive or ineffective. Thus the technical metric must be connected to an operational decision.

1.2 Supervised learning

In supervised learning, examples contain input variables X and an observed target y. The model estimates a function f(X) ≈ y. Regression predicts a numerical quantity, such as delivery time or energy demand. Classification predicts a category, such as fraudulent/not fraudulent or three disease risk levels.

A binary classification target can be represented as y ∈ {0,1}. A multiclass target has more than two mutually exclusive categories. In multilabel classification, one observation can possess several labels simultaneously, such as a document tagged “finance” and “regulation.”

The target must have a clear labeling rule. “High-value customer” requires a threshold, time period and calculation method. Ambiguous labels introduce what is often called label noise: the recorded target does not reliably represent the intended phenomenon.

1.3 Unsupervised learning

Unsupervised learning uses observations without a supplied target. Clustering can segment customers or identify operating modes. Dimensionality reduction can compress correlated variables for visualization or downstream modelling. Anomaly detection seeks observations that differ substantially from a learned pattern.

Unsupervised outputs are not automatically meaningful discoveries. A cluster may reflect a measurement artifact, a region-specific coding practice or a scale difference rather than a real behavioral group. Domain interpretation and stability checks are essential.

1.4 Reinforcement learning

Reinforcement learning concerns sequential decisions. An agent observes a state, chooses an action, receives a reward and transitions to another state. A simplified interaction is:

The objective is generally to maximize expected discounted return:

Here, G_t is return at time t, R_t+k+1 is a future reward, and γ is the discount factor, with 0 ≤ γ < 1 in the usual infinite-horizon setting. This framework is appropriate when actions influence future states, as in inventory control or adaptive recommendations. It is inappropriate merely because a problem contains a “reward” in ordinary language.

1.5 Choosing among paradigms

A useful diagnostic is whether a reliable outcome label exists and whether the system makes sequential interventions. If labels exist and prediction is the goal, supervised learning is suitable. If labels are absent and structure must be explored, unsupervised learning may help. If actions alter future opportunities and feedback arrives over time, reinforcement learning may be considered.

Many projects combine paradigms. For instance, clustering may create exploratory segments, followed by supervised prediction within each segment. Such combinations should be justified by improved decisions rather than novelty.

2. Data Representation and Pipeline Design

2.1 The analytical dataset

A model-ready dataset usually contains one row per analytical unit and columns for identifiers, timestamps, features, target and sometimes sample weights. The row definition must prevent accidental duplication. If a customer appears once per transaction but the target is customer churn, transaction rows may give frequent purchasers disproportionate influence.

A data dictionary should record field name, business meaning, data type, permitted values, unit, source, refresh frequency, missing-value convention, owner and sensitivity classification. Dates should include time zone where relevant. Currency values should specify currency and whether they are nominal or inflation-adjusted.

2.2 Pipeline stages

A dependable pipeline commonly includes:

1. Source extraction from databases, files, APIs or event streams.
2. Validation of schema, types, ranges, uniqueness and referential integrity.
3. Joining sources using documented keys and time rules.
4. Cleaning and feature construction.
5. Target creation using a defined observation and outcome window.
6. Time-aware splitting into training, validation and test sets.
7. Fitting preprocessing only on training data.
8. Transformation, modelling, evaluation and monitoring.

The pipeline should be reproducible. Version data definitions, code, model artifacts and configuration. Record lineage so that a prediction can be traced to its source values and transformation logic.

2.3 Splitting data correctly

Random splitting is acceptable only when observations are approximately independent and identically distributed for the intended deployment setting. Time-dependent problems generally require chronological splits. If records from the same patient, household or machine occur in both training and test sets, group-based splitting may be necessary.

A test set should be protected until final evaluation. Repeatedly optimizing decisions against the test set converts it into a development set and produces optimistic performance estimates.

3. Data Quality and Validity

Data quality has several dimensions: accuracy, completeness, consistency, timeliness, uniqueness and validity. A value can be syntactically valid but semantically wrong; for example, a temperature of 25,000 may fit a numeric type but not a sensor's physical range.

Useful checks include:

• row-count and duplicate-key checks;
• allowable ranges and category checks;
• date-order checks, such as discharge occurring after admission;
• cross-field rules, such as total equaling the sum of components;
• unexpected distribution changes;
• referential integrity between tables;
• null-rate and freshness monitoring.

Outliers require interpretation. An extreme value may be an error, a rare but genuine event or an important failure signal. Blind removal can destroy precisely the cases a model must detect. Options include correction from a trusted source, robust transformations, winsorization, separate flags or retention with a suitable algorithm. Any intervention should be documented.

4. Missing Values

Missingness is not a single condition. A value may be missing completely at random, missing at random conditional on observed variables, or missing not at random because the absence itself relates to the unobserved value or process. These assumptions cannot be established solely from a null-count table.

Common treatments are:

• deletion, when missingness is rare and plausibly harmless;
• simple imputation, such as median for numeric variables or a designated category;
• model-based or multiple imputation for more complex analytical settings;
• explicit missing indicators when absence carries information;
• process correction, when the source system is defective.

For a numeric variable x, training-set median imputation replaces missing values by median(x_train). The median must be calculated using training data only. Computing it on all data allows information from validation or test observations to influence preprocessing. This is a form of leakage.

Imputation values should be reasonable in context. “Unknown” is not always equivalent to zero: zero may mean no activity, whereas unknown means the measurement was unavailable.

5. Data Leakage and Temporal Validity

Data leakage occurs when information unavailable at prediction time influences training or evaluation. It leads to performance estimates that cannot be reproduced in deployment.

Typical examples include:

• using a post-outcome field, such as a cancellation reason, to predict cancellation;
• calculating a customer’s lifetime total using transactions after the prediction date;
• fitting a scaler or imputer on the full dataset before splitting;
• allowing duplicate individuals across training and test sets;
• selecting features using the complete labelled dataset before cross-validation.

A temporal feature must satisfy:

For an outcome window, target construction must also satisfy:

A practical leakage audit asks: “Could an operator have known this value at the exact moment the prediction would have been made?” If the answer is uncertain, exclude the variable or reconstruct it historically.

6. Class Imbalance and Evaluation

Class imbalance occurs when one target class is much more frequent than another. Accuracy can be misleading. If only 1% of transactions are fraudulent, a classifier that predicts “not fraud” for every case has 99% accuracy but zero fraud detection.

For binary classification, let TP, TN, FP and FN denote true positives, true negatives, false positives and false negatives. Important measures include:

Precision measures the proportion of predicted positives that are correct; recall measures the proportion of actual positives detected. The F1 score is the harmonic mean of precision and recall:

These formulas assume binary classification and nonzero denominators. When error costs differ, threshold selection should reflect costs, capacity and consequences rather than defaulting to 0.5.

Approaches to imbalance include class-weighted loss, under-sampling, over-sampling and synthetic methods such as SMOTE. Resampling must occur inside each training fold, never before the train-test split. The evaluation set should retain the deployment class distribution unless a carefully justified alternative is used.

If the cost of a false negative is C_FN and the cost of a false positive is C_FP, an expected error-cost estimate can be written as:

This is useful only when the costs are defined on a comparable monetary or decision scale and when other benefits and constraints are considered.

7. Encoding and Scaling

7.1 Categorical variables

Nominal categories have no natural order. One-hot encoding represents k categories with indicator variables. With an intercept in a linear model, one category is commonly omitted to avoid perfect multicollinearity; this is called reference or dummy coding. Tree-based models may support categorical values directly or require an implementation-specific representation.

Ordinal encoding is appropriate only when category order is meaningful and the numerical spacing is not being interpreted unjustifiably. For example, “low,” “medium” and “high” have order, but the distance from low to medium need not equal the distance from medium to high.

High-cardinality categories may require hashing, frequency encoding, target encoding or learned embeddings. Target encoding must be computed without using the validation target for the same observation. Cross-fold or out-of-fold construction is required to avoid leakage.

7.2 Numerical scaling

Standardization transforms a value x using training-set mean μ and standard deviation σ:

It produces mean approximately zero and standard deviation approximately one in the training sample, assuming σ > 0. Standardization is important for distance-based, gradient-based and regularized models, but often less important for decision trees.

Min-max scaling maps a variable to a chosen interval, commonly [0,1]:

This requires x_max ≠ x_min and is sensitive to extreme values. Robust scaling based on median and interquartile range can be preferable for heavy-tailed data. Scaling parameters must be fitted on training data only.

8. Feature Construction and Selection

Feature engineering converts raw observations into variables that expose relevant structure. Examples include:

• ratios, such as claims per policy;
• counts, sums and averages over a defined time window;
• recency, frequency and monetary-value measures;
• date parts, elapsed time and seasonality indicators;
• interaction terms, such as price × promotion;
• log transformations for positively skewed quantities;
• rolling statistics for time series;
• text representations such as n-grams or embeddings.

Every aggregate needs a window and an anchor time. “Number of complaints in the previous 90 days” is valid only if the window ends at prediction time. A rolling mean for values x over the previous w observations is:

This formula assumes equally weighted observations and a sufficient history; missing or irregularly timed observations require additional treatment.

Feature selection can reduce noise, computation and maintenance burden. Filter methods use statistics independent of a fitted model; wrapper methods evaluate subsets through model performance; embedded methods select during training, as with some regularization methods. Selection must be nested within cross-validation when performance is being estimated.

Feature usefulness is not only predictive. A feature should be available at inference, stable under distribution changes, interpretable enough for its use case, legally permissible and affordable to compute. A powerful but unstable proxy can be worse than a slightly weaker robust feature.

9. Worked Example: A Leakage-Safe Churn Pipeline

Consider an illustrative subscription company with 100,000 active customers on 1 January. The proposed task is to predict cancellation during the next 30 days using information available on 1 January.

The unit is customer-date, and the target is 1 if cancellation occurs from 2 January through 31 January, otherwise 0. Candidate features include sessions in the previous 30 days, failed payments in the previous 90 days, plan type, tenure and support contacts in the previous 60 days. A cancellation-reason field is excluded because it is recorded after cancellation.

The company uses January–March data for training, April for validation and May for final testing, with each feature window ending at the relevant prediction date. Median imputation and standardization parameters are learned from training data. One-hot categories are fitted on training categories, while unseen deployment categories are mapped to an “unknown” representation.

Suppose the test set contains 10,000 customers and the model flags 400 as high risk. Of these, 120 cancel; there are 300 cancellations among customers not flagged. Then TP = 120 and FP = 280, while FN = 300. Therefore:

The result should not be judged in isolation. If the retention team can contact only 400 customers, precision at that capacity may be the operational metric. The team should also compare treatment outcomes: a prediction model identifies risk, but a randomized intervention test is needed to determine whether contacting customers reduces cancellation.

10. Case Study: Predictive Maintenance in Manufacturing

The following is an illustrative case.

Background. A manufacturer experiences unexpected motor failures. Managers want to schedule maintenance before failure without replacing healthy components unnecessarily.

Problem. The initial proposal is “predict failures from sensor data.” Engineers refine this into: using sensor readings available at each hourly decision point, predict whether a motor will fail within the next 48 hours.

Analysis. Motors produce repeated observations, so random row splitting would place readings from the same motor in both training and test data. This could make the model memorize motor-specific behavior. A grouped, chronological split is more defensible. Engineers discover that a field called “maintenance completed” is updated after a failure and therefore leaks future information.

Approach. The team constructs lagged temperature, vibration and current features; rolling means and standard deviations over the previous 6 and 24 hours; and a count of recent alarms. Missing sensor readings receive both an imputed value and a missingness indicator. Features are generated by a version-controlled pipeline, with all windows ending at the prediction timestamp. Because failures are rare, evaluation emphasizes recall at the maintenance team’s capacity and false-alarm cost.

Outcome. The model identifies many failures earlier, but deployment monitoring shows sensor calibration drift at one plant. The organization adds data-quality alerts and retraining criteria rather than assuming the initial performance will persist.

Learning points. The main achievement is not merely selecting an algorithm. It is defining the prediction horizon, respecting temporal availability, preventing group leakage, representing missingness and aligning evaluation with maintenance decisions.

11. Governance, Ethics and Professional Practice

Feature engineering can reproduce unfairness even when protected attributes are excluded. Postal code, language, device type or employment history may act as proxies. Teams should examine performance and error rates across relevant groups, document intended use, and investigate whether features reflect historical discrimination.

Privacy principles include purpose limitation, data minimization, access control, retention limits and secure handling. Sensitive data should not be collected simply because future modelling might benefit from them. Consent and legal basis depend on jurisdiction and context; professional teams should consult applicable privacy and sector regulations.

Documentation should include a data sheet or data card, feature definitions, label-generation logic, known limitations, evaluation population, subgroup results and monitoring plan. Human oversight is essential where predictions affect credit, employment, healthcare, education or access to essential services. Explanations should be appropriate to the audience and should not imply causal evidence when the model is merely predictive.

12. Contemporary Developments and Best Practices

Modern pipelines increasingly use feature stores, automated validation, data contracts and monitoring for drift. Representation learning and pretrained embeddings can reduce manual construction for text, images and complex event streams, but they do not eliminate the need for temporal validity, governance and domain evaluation. Automated feature generation may create thousands of candidates; without strict validation it can amplify leakage and spurious correlations.

Recommended practice is to:

1. define the decision, unit, target and time horizon before modelling;
2. create a data dictionary and lineage record;
3. split data according to deployment reality;
4. fit every learned preprocessing step only on training data;
5. audit leakage, duplicates, missingness, imbalance and subgroup behavior;
6. prefer features that are stable, available and actionable;
7. evaluate both statistical performance and operational value;
8. monitor data quality, drift, performance and intervention outcomes after deployment.

Challenges and Limitations

No preprocessing method guarantees valid inference. Missing-data assumptions may be wrong, labels may encode institutional bias, and historical relationships may change after deployment. Synthetic oversampling does not create new real-world evidence. Scaling cannot repair a misdefined unit of analysis. Feature selection may improve test performance by chance if validation is repeatedly consulted. Finally, predictive association does not establish that changing a feature will change the outcome; causal questions require different designs.

Chapter Summary

Problem formulation translates an organizational objective into a precise decision, unit, input set, target, time horizon and evaluation criterion. Supervised learning uses labelled outcomes, unsupervised learning explores structure without labels, and reinforcement learning addresses sequential action and feedback. Reliable projects require a documented, reproducible data pipeline with schema validation, lineage and deployment-realistic splitting.

Data quality involves accuracy, completeness, consistency, timeliness, uniqueness and validity. Missing values require attention to their generating process, while outliers must be interpreted rather than automatically removed. Leakage is prevented by ensuring that every feature and preprocessing parameter could have been available at prediction time. Imbalance requires suitable metrics and, when appropriate, carefully controlled weighting or resampling. Encoding and scaling must match variable type and model assumptions. Feature construction can expose useful temporal, aggregate and interaction structure, but every feature must be valid, stable, ethical and maintainable. Governance, subgroup evaluation, privacy and post-deployment monitoring are integral parts of professional machine learning.

Key Terms

Problem formulation: Translation of a real-world objective into a formal learning and decision task.

Analytical unit: The entity represented by one dataset row.

Feature: An input variable supplied to a model.

Target: The outcome a supervised model is trained to predict.

Prediction horizon: The future period over which an outcome is defined.

Supervised learning: Learning from examples with observed targets.

Unsupervised learning: Learning patterns without supplied targets.

Reinforcement learning: Learning sequential actions through rewards and state transitions.

Data leakage: Use of information unavailable at prediction time.

Imputation: Replacement or estimation of missing values.

Class imbalance: Unequal prevalence of target classes.

Precision: Proportion of predicted positives that are true positives.

Recall: Proportion of actual positives detected.

Encoding: Conversion of categorical or symbolic values into model inputs.

Standardization: Transformation using training mean and standard deviation.

Feature engineering: Construction or transformation of variables for modelling.

Data drift: Change in input distributions over time.

Label noise: Error or ambiguity in recorded target values.

Data lineage: Record of data origins and transformations.

Concept drift: Change in the relationship between inputs and the target.

Review Questions

Short Answer Questions

1. What four time-related elements should be specified when defining a predictive task?
2. Why can accuracy be misleading for a highly imbalanced classification problem?
3. Distinguish nominal encoding from ordinal encoding.
4. Why must imputation parameters be fitted on training data only?
5. Give two examples of data leakage.

Long Answer Questions

1. Explain how an organizational objective can be translated into a supervised-learning formulation, using a suitable example.
2. Describe a production-grade data pipeline from source extraction to monitoring.
3. Compare deletion, simple imputation and model-based imputation, including their assumptions and risks.
4. Explain precision, recall, specificity and F1, and discuss how threshold selection should reflect operational costs.
5. Discuss the ethical, privacy and governance implications of feature engineering.

Analytical / Application Questions

1. A hospital wants to predict readmission using patient records. Design a leakage-safe unit of analysis, target window, feature window and evaluation split.
2. A fraud model has 2% positive cases. Recommend an evaluation strategy and explain why accuracy alone is inadequate.
3. A manufacturer has repeated sensor readings from 50 machines. Propose a splitting strategy and three temporally valid engineered features, explaining how you would monitor them after deployment.

Multiple Choice Questions with Answers

Question 1

Which situation is most appropriate for supervised learning?

A. Grouping customers when no outcome labels exist

B. Predicting next-month cancellation from historical labelled cancellations

C. Learning actions through sequential rewards

D. Visualizing high-dimensional data without a target

Correct Answer: B. Predicting next-month cancellation from historical labelled cancellations

Question 2

What is the primary reason for using a chronological split in a time-dependent prediction problem?

A. It guarantees higher accuracy

B. It removes all missing values

C. It reflects the information available at deployment time

D. It eliminates class imbalance

Correct Answer: C. It reflects the information available at deployment time

Question 3

Which example is data leakage?

A. Using tenure measured at prediction time

B. Using the median calculated from training data for imputation

C. Using cancellation reason recorded after the customer cancels

D. Using a one-hot representation fitted on training categories

Correct Answer: C. Using cancellation reason recorded after the customer cancels

Question 4

A classifier has TP = 30 and FP = 70. What is its precision?

A. 0.30

B. 0.43

C. 0.70

D. 0.75

Correct Answer: A. 0.30

Question 5

Which transformation is standardization?

A. x' = x ÷ max(x)

B. z = (x − μ) ÷ σ

C. x' = x − min(x)

D. z = x × σ + μ

Correct Answer: B. z = (x − μ) ÷ σ

Question 6

Which encoding is generally most suitable for categories with a meaningful order?

A. Ordinal encoding

B. Random noise encoding

C. One-hot encoding in every case

D. Image normalization

Correct Answer: A. Ordinal encoding

Question 7

Why should oversampling be performed inside training folds rather than before cross-validation?

A. To increase the test-set size

B. To prevent synthetic or duplicated information from influencing validation folds

C. To guarantee perfect recall

D. To remove the need for scaling

Correct Answer: B. To prevent synthetic or duplicated information from influencing validation folds

Question 8

What does recall measure in binary classification?

A. The proportion of predicted positives that are correct

B. The proportion of actual positives that are detected

C. The proportion of actual negatives that are detected

D. The proportion of all predictions that are correct

Correct Answer: B. The proportion of actual positives that are detected

Question 9

Which feature is temporally valid for a prediction made at 1 June?

A. Complaints recorded on 5 June

B. A cancellation reason entered on 10 June

C. The number of sessions during the previous 30 days

D. A payment failure occurring after the prediction

Correct Answer: C. The number of sessions during the previous 30 days

Question 10

Why might group-based splitting be required for repeated observations from the same machine?

A. To ensure every group appears in both training and test sets

B. To prevent the model from benefiting from the same machine appearing in both sets

C. To convert numerical features into categories

D. To increase the number of target labels

Correct Answer: B. To prevent the model from benefiting from the same machine appearing in both sets