12. Machine Learning Workflows in Python

Introduction and Context

A machine-learning model is only one component of a reliable data-science solution. In professional work, data must be inspected, transformed, divided into appropriate datasets, used to train a model, evaluated with suitable measures, and maintained after deployment. These activities form a machine-learning workflow. Python, and particularly the scikit-learn ecosystem, provides abstractions that make such workflows reproducible, testable, and reusable.

A workflow is more than a sequence of code cells. It is a controlled process that protects the integrity of evaluation, makes preprocessing consistent between training and production, and allows alternative algorithms to be compared fairly. A model that achieves high accuracy because information from the test set entered the training process is not a successful model. Similarly, a sophisticated algorithm cannot compensate for inappropriate target definitions, poorly selected metrics, or unstable data preparation.

This chapter develops a practical framework for building workflows in Python. It focuses on `Pipeline`, `ColumnTransformer`, feature engineering, supervised and unsupervised learning, cross-validation, hyperparameter tuning, evaluation metrics, explainability, and leakage prevention. The emphasis is on decisions that affect the validity and operational usefulness of machine-learning systems.

Learning Objectives

After studying this chapter, a learner should be able to:

1. Describe the stages of a dependable machine-learning workflow.

2. Construct preprocessing and modelling pipelines with scikit-learn.

3. Apply different transformations to numerical and categorical features.

4. Distinguish training, validation, and test procedures and explain why leakage is harmful.

5. Select cross-validation strategies, metrics, and hyperparameter-search methods appropriate to a problem.

6. Build workflows for supervised and unsupervised learning.

7. Interpret model behaviour using global and local explainability techniques.

8. Diagnose common workflow failures and propose professional safeguards.

From Raw Data to a Reproducible Workflow

A typical workflow has the following conceptual stages: problem formulation, data acquisition, exploratory analysis, data splitting, preprocessing and feature engineering, model fitting, validation and tuning, final evaluation, interpretation, and deployment monitoring. These stages are iterative rather than strictly linear. Exploratory analysis may reveal that the target is incorrectly defined; validation may show that a feature is unavailable at prediction time; monitoring may reveal that the data distribution has changed.

The first task is to define the prediction unit and the prediction moment. For example, a credit-risk model may make one prediction per loan application at the time of application. A feature calculated six months after approval is therefore invalid, even if it improves historical performance. This temporal definition is as important as the choice of algorithm.

A dataset is commonly divided into a training set and a held-out test set. The training set is used for fitting preprocessing parameters, model parameters, and tuning decisions. The test set is used once, or very sparingly, for an unbiased estimate of final performance. When a separate validation set is not available, cross-validation within the training set supports model selection. A useful abstraction is:

• Training data: estimates parameters.
• Validation procedure: compares models and hyperparameters.
• Test data: estimates expected performance on unseen data.

The split must reflect how the model will encounter future observations. Random splitting may be reasonable for independent observations, but it is inappropriate for time-dependent data, repeated measurements from the same person, or groups whose records are correlated. Group-based or temporal splits may be required.

The scikit-learn Estimator Model

Scikit-learn uses a consistent estimator interface. A transformer generally implements `fit` and `transform`; a predictor implements `fit` and `predict`; many predictors also implement `predict_proba` or `decision_function`. Calling `fit` estimates quantities from data, while `transform` applies learned quantities to new observations. For example, `StandardScaler` learns a mean and standard deviation during fitting and uses them later to standardize records.

This distinction is central to sound workflow design. A transformation must be fitted only on the relevant training portion. Calling `fit_transform` on the complete dataset before splitting allows information from validation or test observations to influence the learned transformation. A pipeline automates the correct order during cross-validation.

A `Pipeline` connects a sequence of transformations and a final estimator. Its parameters are exposed using names such as `model__C` or `preprocess__numeric__imputer__strategy`, allowing the complete sequence to be tuned as a unit. The pipeline is also the object that should normally be persisted and deployed, rather than saving a model separately from undocumented preprocessing code.

Designing the Preprocessing Layer

Real datasets frequently contain mixed data types, missing values, categorical variables, skewed distributions, and irrelevant identifiers. Preprocessing should be designed according to the statistical assumptions of the algorithm and the operational meaning of each field.

For numerical variables, missing values may be imputed using a median, mean, constant, or a more carefully designed model. Median imputation is often less sensitive to outliers than mean imputation. Scaling is important for distance-based and regularized models. Standardization produces approximately zero-centred variables with unit variance, whereas robust scaling uses statistics less affected by extreme observations. Tree-based models are generally less dependent on scale, although consistent preprocessing may still simplify a shared workflow.

Categorical variables can be represented using one-hot encoding, ordinal encoding, target encoding, or learned embeddings in other frameworks. One-hot encoding is suitable when categories have no natural order. Unknown categories should be handled explicitly, often with `handle_unknown='ignore'`, so that a new category at inference does not cause a failure. Ordinal encoding should not be used merely for convenience when the numerical ordering would create a false relationship.

`ColumnTransformer` applies separate transformations to selected columns. A representative classification workflow is:

```python

from sklearn.compose import ColumnTransformer

from sklearn.impute import SimpleImputer

from sklearn.pipeline import Pipeline

from sklearn.preprocessing import OneHotEncoder, StandardScaler

from sklearn.linear_model import LogisticRegression

numeric_pipe = Pipeline([

("imputer", SimpleImputer(strategy="median")),

("scale", StandardScaler())

])

categorical_pipe = Pipeline([

("imputer", SimpleImputer(strategy="most_frequent")),

("encode", OneHotEncoder(handle_unknown="ignore"))

])

preprocess = ColumnTransformer([

("num", numeric_pipe, numeric_columns),

("cat", categorical_pipe, categorical_columns)

])

workflow = Pipeline([

("preprocess", preprocess),

("model", LogisticRegression(max_iter=1000))

])

```

The example separates data preparation from prediction while retaining them in one estimable object. It also makes the treatment of new records explicit. In a production system, column names, data types, and category-handling rules should be validated before prediction.

Feature Engineering as Part of the Model

Feature engineering converts raw observations into representations that make useful structure more accessible to an algorithm. Examples include extracting hour, day of week, or month from timestamps; calculating ratios; aggregating transaction records by customer; representing text with n-grams or TF-IDF; and transforming strongly skewed variables with a logarithm.

Feature engineering must respect availability. A customer’s total spending over the entire year cannot be used to predict whether the customer will churn in March unless the total is calculated only from information available by the prediction date. Aggregations should normally be performed within the appropriate temporal and group boundaries.

Custom transformations can be implemented using a transformer class or a carefully designed function. A custom transformer should learn all data-dependent quantities in `fit` and apply them in `transform`. It should preserve the expected input-output contract and be included in the pipeline. This ensures that feature engineering is repeated consistently during validation and deployment.

Feature selection is also part of workflow design. Removing redundant or noisy variables may improve interpretability, speed, and generalization. However, selection based on the complete dataset can leak information. Methods such as univariate selection, recursive feature elimination, and model-based selection should be placed inside the pipeline when they use the target or estimate data-dependent statistics.

Supervised Learning Workflows

In supervised learning, the target variable guides estimation. Classification predicts discrete classes, while regression predicts numerical values. The same workflow principles apply to both, but the target structure affects splitting, metrics, and error analysis.

For imbalanced classification, a stratified split or stratified cross-validation preserves class proportions. Accuracy may be misleading when one class dominates. A classifier that predicts the majority class for every case can have high accuracy but no practical value for detecting rare events. Precision, recall, F1 score, balanced accuracy, average precision, and area under the receiver operating characteristic curve answer different questions.

Regression workflows commonly use mean absolute error (MAE), mean squared error (MSE), root mean squared error (RMSE), and coefficient of determination (R²). MAE is easier to interpret in the original unit and is less influenced by large errors than MSE or RMSE. RMSE penalizes large errors more strongly. R² measures explained variation relative to a baseline, but it should not be treated as a complete measure of business usefulness.

A metric should be selected before examining competing models wherever possible. If missing a positive medical case is especially costly, recall may receive priority. If false alarms consume expensive investigative resources, precision may be more important. In threshold-based classification, the probability model and the decision threshold are separate choices. A threshold should be selected using validation data and the cost structure, not adjusted on the test set.

Cross-Validation and Hyperparameter Tuning

Cross-validation estimates how a model may perform on unseen data and supports comparison of alternative configurations. In k-fold cross-validation, the training data are divided into k folds. Each fold is used once for validation while the other folds are used for fitting, and the scores are aggregated. Stratified k-fold is commonly used for classification; group and time-series variants are necessary when observations are not independent.

Cross-validation does not remove bias caused by a flawed split or leakage. If records from the same patient occur in both training and validation folds, performance may be inflated because the model recognizes patient-specific patterns. If future records are used to predict the past, a random split creates an unrealistic evaluation.

Hyperparameters are settings chosen before or around model fitting, such as regularization strength, tree depth, number of estimators, or neighbourhood size. `GridSearchCV` evaluates a specified Cartesian grid, while `RandomizedSearchCV` samples configurations and can explore large spaces more efficiently. Successive-halving methods allocate increasing resources to promising configurations. The search should be performed on a pipeline, not on preprocessed data prepared using all observations.

Nested cross-validation provides a stronger estimate when the same dataset is used both to tune and evaluate models. An inner loop selects hyperparameters; an outer loop estimates generalization. It is computationally more expensive but useful for small datasets or comparative research. After a model is selected, it can be refitted on the full training data, with the test set retained for final reporting.

A result should include uncertainty, not only a single score. Reporting fold-level variation, confidence intervals where appropriate, and performance across meaningful subgroups supports better decisions. Large variation across folds may indicate limited data, heterogeneous populations, or an unsuitable validation design.

Unsupervised Workflows

Unsupervised learning has no target variable used for fitting. Clustering, dimensionality reduction, anomaly detection, and representation learning are common examples. Workflow discipline remains necessary even though conventional predictive accuracy is unavailable.

K-means clustering is sensitive to feature scale and requires a chosen number of clusters. A pipeline can impute and scale features before clustering. The silhouette coefficient and within-cluster sum of squares can support analysis, but neither proves that the clusters are meaningful for a business or scientific purpose. Domain interpretation and stability across samples are essential.

Dimensionality-reduction methods such as principal component analysis (PCA) learn directions of variation from data. PCA must be fitted only on the training portion when it precedes a supervised model. Otherwise, the test set influences the learned representation. PCA may improve computational efficiency, but it can reduce interpretability because components combine original variables.

Anomaly detection methods, including isolation-based and one-class approaches, require careful consideration of what constitutes an anomaly. Rare does not always mean erroneous, and a detector trained on contaminated data may learn that important abnormal cases are normal. Evaluation may involve expert review, temporal back-testing, or a limited labelled set rather than a single generic score.

Evaluation, Calibration, and Explainability

Evaluation should combine a numerical metric with error analysis. Confusion matrices reveal the distribution of false positives and false negatives. Residual plots can reveal nonlinearity, heteroscedasticity, or systematic bias in regression. Performance should be examined across relevant subgroups, time periods, and operating conditions.

Probability calibration concerns whether predicted probabilities correspond to observed frequencies. A model that assigns a probability of 0.8 to a group of cases should be correct approximately 80 percent of the time under suitable conditions. Calibration curves, Brier score, and calibration methods such as Platt scaling or isotonic regression can be used, provided calibration is fitted without contaminating evaluation data.

Explainability operates at several levels. Global explanations describe overall feature influence or model structure. Coefficients in a suitably specified linear model are one form of global interpretation, though correlated variables complicate their meaning. Permutation importance measures how performance changes when a feature is shuffled, but correlated features may share or obscure importance. Local explanations describe why a particular prediction differs from a reference, using techniques such as partial dependence, accumulated local effects, or SHAP-style contributions.

Explanations should not be treated as proof of causality. A feature may be predictive because it is a proxy for a protected attribute, a consequence of a decision already made, or an indicator of data collection practices. Explanations should therefore be combined with domain review, fairness analysis, and documentation of data provenance.

An Illustrative Case: Predicting Subscription Cancellation

Background. Consider a fictional telecommunications provider that wants to identify customers at risk of cancelling within the next 30 days. The organisation has customer tenure, contract type, monthly charges, support contacts, recent usage, and cancellation records.

Situation and problem. The initial analyst reports 94 percent accuracy. Further inspection shows that only 6 percent of customers cancel, so a majority-class model achieves approximately the same accuracy. In addition, the data include a field called `cancellation_reason`, entered after a customer has cancelled. This field is unavailable at prediction time.

Analysis. The target is defined as cancellation within the following 30 days, and the prediction date is fixed for each customer. Records are ordered by date. A temporal split is selected: earlier observations form the training period and later observations form the test period. Features created from customer activity are computed only up to each prediction date. The post-cancellation field is removed. Because the positive class is rare, the team reports recall, precision, average precision, and a confusion matrix rather than accuracy alone.

Approach. Numerical missing values are imputed and scaled; categorical fields are imputed and one-hot encoded. These operations and a logistic-regression classifier are placed in a `Pipeline` with `ColumnTransformer`. A balanced random forest is also evaluated. Hyperparameters are selected through time-aware validation on the training period. The operating threshold is chosen according to the retention team’s capacity: it can contact only a fixed proportion of customers each week. Permutation importance and case-level reviews are used to understand model behaviour.

Outcome. The corrected evaluation produces lower headline performance than the original report, but it is operationally credible. The retention team receives a ranked list and a documented threshold. Monitoring is established for class prevalence, feature distributions, calibration, and performance when later outcomes become available.

Learning points. The case illustrates that leakage prevention, temporal design, suitable metrics, and operational constraints are more valuable than an attractive but invalid accuracy figure. It also demonstrates why a deployed artefact should include preprocessing, model, threshold, feature definitions, and evaluation documentation.

Failure Modes and Professional Safeguards

Several mistakes recur in practical projects. Scaling, imputing, selecting features, or applying PCA before the data split causes leakage. Selecting a model after repeatedly inspecting the test score turns the test set into a hidden validation set. Randomly splitting related records produces overly optimistic estimates. Ignoring missingness mechanisms can create systematic subgroup errors. Fitting a preprocessing object in a notebook and later recreating it manually can cause training-serving skew.

Other failures are organisational. A model may be accurate but unusable if its predictions arrive too late, if the intervention cost exceeds its benefit, or if users do not understand how to act on its output. A model may also degrade when user behaviour, policy, economic conditions, or data collection systems change. Monitoring should therefore cover data quality, drift, latency, calibration, error rates, subgroup performance, and business outcomes.

Reproducibility requires fixed random seeds where appropriate, versioned code and data specifications, recorded package versions, deterministic or documented split procedures, and saved experiment metadata. Reproducibility does not mean that every future result must be identical; it means that the conditions and decisions behind a result can be reconstructed.

Ethical, Governance, and Regulatory Considerations

Machine-learning workflows can affect access to credit, employment, insurance, healthcare, education, and public services. Teams should establish a legitimate purpose, minimise data collection, protect sensitive information, and document retention and access controls. Personal data should not be repurposed casually merely because it improves predictive performance.

Fairness assessment should consider the context, protected or sensitive attributes, relevant definitions of harm, and trade-offs among error rates. Removing a sensitive attribute does not guarantee fairness because proxy variables may remain. Automated predictions should have appropriate human oversight, appeal mechanisms, and clear accountability. Explanations provided to affected individuals must be meaningful rather than merely technical.

Governance artefacts may include a data sheet, model card, feature catalogue, validation report, risk assessment, decision log, and monitoring plan. These documents support auditability and responsible handover between data scientists, engineers, domain experts, and managers.

Emerging Directions

Current workflow development increasingly emphasises automated experiment tracking, reproducible environments, feature stores, model registries, and continuous integration tests for data and models. These tools can improve consistency, but automation does not eliminate the need for statistical judgement. Automated machine-learning systems still require a valid target, an appropriate split, meaningful constraints, and human review.

Modern systems also combine scikit-learn components with text, image, deep-learning, and large-language-model pipelines. The same principles remain applicable: transformations must be fitted without leakage, evaluation must reflect deployment, and versioned components must be monitored. Responsible machine learning is becoming a lifecycle practice rather than a final compliance step.

Best Practices for Reliable Python Workflows

1. Define the prediction unit, target, and information-availability time before modelling.

2. Split data according to independence, groups, and chronology.

3. Put every data-dependent transformation inside a pipeline.

4. Use `ColumnTransformer` for heterogeneous tabular data and handle unknown categories explicitly.

5. Select metrics according to error costs and operational objectives.

6. Tune hyperparameters using cross-validation on the training data only.

7. Keep the test set isolated until the final evaluation.

8. Inspect subgroup performance, calibration, errors, and data drift.

9. Persist the complete workflow and record versions, assumptions, and thresholds.

10. Treat interpretability, privacy, fairness, and monitoring as design requirements.

Chapter Summary

A machine-learning workflow is a reproducible process linking problem definition, data splitting, preprocessing, feature engineering, model fitting, validation, evaluation, interpretation, and deployment monitoring. Scikit-learn’s estimator interface, `Pipeline`, and `ColumnTransformer` provide practical foundations for implementing this process. Cross-validation and hyperparameter search support model comparison, but only when the split reflects the data-generating process and all data-dependent operations are contained within the workflow. Metrics must correspond to the costs and objectives of the application, particularly in imbalanced classification. Explainability supports interpretation but does not establish causality or fairness. Reliable professional practice requires leakage prevention, temporal and group awareness, documentation, governance, and continuous monitoring.

Key Terms

Pipeline: An ordered scikit-learn object that applies transformations and a final estimator as one workflow.

Transformer: An estimator that learns a transformation through `fit` and applies it through `transform`.

ColumnTransformer: A scikit-learn utility that applies different transformations to selected columns.

Feature engineering: Creation or transformation of variables to represent useful predictive information.

Data leakage: Unintended use of information unavailable at prediction time or reserved for evaluation.

Training set: Data used to estimate model and preprocessing parameters.

Validation: A procedure for comparing models or hyperparameters using held-out portions of training data.

Test set: Data isolated for final estimation of generalisation performance.

Cross-validation: Repeated training and validation across complementary data folds.

Hyperparameter: A model or workflow setting selected outside the ordinary parameter-fitting process.

Stratification: Preservation of class proportions when forming data splits.

Group split: A split that keeps related observations in the same partition.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Precision: The proportion of predicted positives that are actually positive.

Recall: The proportion of actual positives that are correctly identified.

RMSE: The square root of mean squared error, expressed in the target’s units.

Permutation importance: Change in model performance after a feature’s values are randomly permuted.

Concept drift: Change over time in the relationship between inputs and the target.

Training-serving skew: A discrepancy between data processing during training and production inference.

Review Questions

Short Answer Questions

1. Why should a scaler be fitted inside a pipeline rather than on the complete dataset before splitting?

2. What is the role of `ColumnTransformer` in a tabular workflow?

3. Distinguish a hyperparameter from a parameter estimated during model fitting.

4. Why can accuracy be misleading for an imbalanced classification problem?

5. What is meant by training-serving skew?

Long Answer Questions

1. Explain the stages of a reproducible machine-learning workflow and the decisions made at each stage.

2. Describe how to construct a mixed-type classification workflow using imputation, scaling, one-hot encoding, and logistic regression.

3. Compare random, stratified, group-based, and time-aware validation designs.

4. Explain how cross-validation and hyperparameter tuning should be performed without contaminating the test set.

5. Discuss the roles and limitations of calibration, permutation importance, and local explanations.

Analytical / Application Questions

1. A hospital dataset contains multiple visits per patient. Design a validation strategy and explain why ordinary random k-fold cross-validation may be inappropriate.

2. A fraud model has 99 percent accuracy but detects only 5 percent of fraudulent transactions. Recommend evaluation metrics and a threshold-selection procedure.

3. A feature representing a customer’s annual expenditure substantially improves a churn model. Explain how you would determine whether the feature causes temporal leakage and redesign the feature if necessary.

22. Multiple Choice Questions with Answers

Question 1

Which operation must be fitted only on the training portion within each cross-validation fold?

A. Printing the column names

B. Standardisation using a learned mean and standard deviation

C. Counting the number of rows in the dataset

D. Selecting the name of the target column

Correct Answer: B. Standardisation using a learned mean and standard deviation

Question 2

What is the principal advantage of placing preprocessing and a predictor in a scikit-learn Pipeline?

A. It guarantees that the model will be accurate

B. It removes the need to define a target variable

C. It applies learned transformations consistently during validation and inference

D. It automatically eliminates all biased features

Correct Answer: C. It applies learned transformations consistently during validation and inference

Question 3

A dataset contains several records for each patient. Which validation approach best prevents patient-specific information from appearing in both training and validation sets?

A. Group-based cross-validation by patient

B. Random sampling of individual rows

C. Sorting rows alphabetically before splitting

D. Using the test set for hyperparameter selection

Correct Answer: A. Group-based cross-validation by patient

Question 4

Which metric is generally more informative than accuracy when the positive class is rare and identifying positives is important?

A. Recall

B. Number of features

C. Training-set size

D. R-squared for the categorical target

Correct Answer: A. Recall

Question 5

Why should a field entered after a customer cancels a subscription be excluded from a model predicting future cancellation?

A. It is necessarily numerical

B. It creates temporal leakage because it is unavailable at prediction time

C. It prevents one-hot encoding

D. It always decreases model complexity

Correct Answer: B. It creates temporal leakage because it is unavailable at prediction time

Question 6

What does nested cross-validation primarily provide?

A. A way to avoid having any validation data

B. A stronger estimate when tuning and performance estimation use the same dataset

C. Automatic conversion of classification into regression

D. A guarantee that the test set contains no missing values

Correct Answer: B. A stronger estimate when tuning and performance estimation use the same dataset

Question 7

Which statement about probability calibration is correct?

A. A calibrated probability of 0.8 should correspond approximately to an 80 percent event frequency under suitable conditions

B. Calibration means that every prediction is correct

C. Calibration is identical to maximising accuracy

D. Calibration can be assessed only for regression models

Correct Answer: A. A calibrated probability of 0.8 should correspond approximately to an 80 percent event frequency under suitable conditions

Question 8

Why can removing a protected attribute fail to remove unfairness?

A. Models cannot use numerical variables without protected attributes

B. Other variables may act as proxies for the protected attribute

C. Protected attributes are required for every pipeline

D. Removing a column always causes underfitting

Correct Answer: B. Other variables may act as proxies for the protected attribute

Question 9

In a time-dependent prediction problem, why might a random train-test split be invalid?

A. Random splitting always produces equal class sizes

B. It can allow future information or changing regimes to influence evaluation of past-like predictions

C. It prevents the use of numerical features

D. It automatically removes temporal ordering from the target

Correct Answer: B. It can allow future information or changing regimes to influence evaluation of past-like predictions

Question 10

Which practice best supports deployment consistency?

A. Saving only the final model coefficients and rewriting preprocessing manually

B. Persisting the complete fitted workflow, including preprocessing and the estimator

C. Selecting the threshold after examining final test outcomes

D. Discarding feature definitions after training

Correct Answer: B. Persisting the complete fitted workflow, including preprocessing and the estimator