Predictive Modeling Workflow and Evaluation

Introduction and Context

Predictive modeling converts historical observations into estimates of unknown or future outcomes. In data science with R, the quality of a predictive solution depends not only on the algorithm selected, but also on how data are partitioned, transformed, tuned, evaluated, and connected to a real decision. A highly accurate model can be invalid if information from the future has entered the training data, if the test set has been repeatedly inspected, or if the evaluation metric does not represent the business objective.

The tidymodels ecosystem provides a coherent framework for constructing reproducible modeling workflows. Its principal components include `rsample` for resampling, `recipes` for preprocessing, `parsnip` for model specification, `workflows` for combining models and preprocessing, `tune` for hyperparameter optimization, `yardstick` for metrics, and `dials` for parameter definitions. Together, these packages support a disciplined separation between model development and final performance assessment.

This chapter develops a leakage-free workflow for supervised learning. It covers preprocessing, feature engineering, train-validation-test splitting, cross-validation, hyperparameter tuning, model comparison, and the interpretation of statistical and business metrics.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Distinguish training, validation, cross-validation, and test data.

2. Explain data leakage and design leakage-free preprocessing pipelines.

3. Construct `recipes` and `workflows` in tidymodels.

4. Select and engineer predictors appropriately for regression and classification.

5. Tune model hyperparameters using resampling.

6. Compare models with suitable statistical and business metrics.

7. Interpret uncertainty, threshold decisions, and model limitations.

8. Design a reproducible predictive modeling process for a practical case.

1. Predictive Modeling as a Decision Process

A predictive project has at least four distinct objects:

• Observations: rows representing customers, transactions, patients, machines, or other units.
• Predictors: variables available when a prediction must be made.
• Outcome: the response to be predicted, such as a continuous loss or a binary default indicator.
• Decision: an action based on the prediction, such as approving a loan, contacting a customer, or scheduling maintenance.

The distinction between prediction and explanation is important. A variable may improve predictive accuracy without representing a causal factor. Conversely, a causally important variable may have limited predictive value if it is measured noisily. Predictive modeling therefore requires explicit attention to the point in time at which predictions will be generated and to the costs of different errors.

A useful modeling statement is: given predictors available at time t, estimate the conditional distribution or a useful summary of an outcome at time t+h. The phrase “available at time t” is a governance requirement, not merely a technical detail.

Common terminology

A parameter is learned from data, such as a regression coefficient. A hyperparameter is selected during model development, such as tree depth or a penalty strength. Resampling repeatedly creates analysis and assessment subsets to estimate generalization performance. Generalization is performance on new data from the intended population. Calibration describes the agreement between predicted probabilities and observed frequencies. Discrimination describes how well a model ranks cases by outcome risk.

2. Partitioning Data Without Contamination

Training, validation, and test roles

The training set is used to estimate model parameters and learn preprocessing operations. A validation set is used to compare configurations or tune hyperparameters. A test set is held back until the modeling decisions are substantially complete and provides an approximately unbiased estimate of final performance.

For moderate data sets, a common design is a training set containing 60–80% of observations and a test set containing the remainder. Within the training set, cross-validation can replace a single validation split. A separate validation set is especially useful when computationally intensive tuning or repeated model selection is expected.

The test set must not be used to choose features, transformations, thresholds, algorithms, or hyperparameters. If it is consulted repeatedly, it becomes part of the development process and ceases to be a genuine final assessment. In that situation, a new untouched test set or an appropriately nested resampling design is required.

Stratification and grouped data

For classification, stratified sampling helps preserve outcome proportions in each split. In tidymodels, `initial_split(data, prop = 0.8, strata = outcome)` performs a stratified initial split. Stratification is particularly important for rare outcomes, although it does not solve the statistical difficulties of extremely small minority classes.

Observations from the same individual, household, machine, or organization should generally remain in one partition. A random row-wise split can otherwise place nearly identical records in both training and test sets, producing overoptimistic results. Grouped resampling, such as `group_initial_split()` and grouped cross-validation, should be considered when dependence exists.

For temporal data, random splitting is often inappropriate. Training observations should precede assessment observations, using rolling or time-based resampling. A model intended to predict next month’s demand must not be assessed using data from earlier and later periods in a way that permits future patterns to influence the past.

Reproducibility

Random operations should be controlled with `set.seed()`. A seed does not make a poor design valid, but it makes results reproducible. Analysts should record package versions, data extraction dates, split identifiers, preprocessing code, model specifications, tuning grids, and metric definitions.

3. Leakage-Free Preprocessing and Feature Engineering

Data leakage occurs when information unavailable at prediction time, or information derived from assessment outcomes, influences model development. Leakage can arise through the outcome itself, through future variables, through duplicated entities, or through preprocessing performed before resampling.

Examples include:

• imputing missing values using the mean calculated from the entire data set;
• selecting variables using correlations calculated with the full data set;
• scaling predictors before the train/test split;
• using a claim settlement amount to predict whether a claim will later be fraudulent;
• allowing records from one patient to appear in both training and testing;
• tuning a model repeatedly against the final test set.

The principle is simple: every operation that learns from data must be estimated using only the relevant analysis portion and then applied to the corresponding assessment portion.

Recipes

A tidymodels recipe expresses preprocessing declaratively. For example:

```r

library(tidymodels)

set.seed(2025)

split <- initial_split(customer_data, prop = 0.80, strata = churn)

train_data <- training(split)

test_data <- testing(split)

rec <- recipe(churn ~ ., data = train_data) |>

step_rm(customer_id) |>

step_unknown(all_nominal_predictors()) |>

step_impute_median(all_numeric_predictors()) |>

step_other(all_nominal_predictors(), threshold = 0.02) |>

step_dummy(all_nominal_predictors()) |>

step_zv(all_predictors())

```

The recipe is prepared within a workflow. During resampling, `prep()` is effectively estimated separately for each analysis fold, and `bake()` applies the learned transformations to the assessment fold. This behavior prevents fold information from contaminating the analysis data.

The order of steps matters. For example, removing identifiers should occur before modeling; imputation must be defined consistently; dummy encoding should be fitted using the analysis data; and zero-variance removal should occur after transformations that may create constant columns. Domain knowledge may justify a different order.

Useful recipe operations include:

• `step_log()` or `step_log10()` for strongly right-skewed positive variables;
• `step_normalize()` for methods sensitive to predictor scale, such as penalized regression and nearest neighbors;
• `step_interact()` for selected interactions;
• `step_ns()` or `step_spline_natural()` for nonlinear effects;
• `step_corr()` for removing highly correlated predictors, when justified;
• `step_smote()` for minority-class oversampling, applied within resampling rather than before splitting.

Preprocessing is model-dependent. Tree-based methods generally do not require normalization, while regularized linear models often benefit from it. One-hot encoding is required by many engines, although some implementations handle categorical variables directly.

Feature engineering and temporal validity

Feature engineering should represent information that would genuinely be available at prediction time. A customer’s number of support contacts during the previous 90 days may be valid; the number of contacts during the following 30 days is not. Aggregations must define a reference date and an observation window. In production, feature definitions should be versioned and tested for point-in-time correctness.

4. Workflows and Model Specifications

A workflow combines a recipe with a model specification, reducing the risk that preprocessing and modeling are accidentally separated.

```r

logistic_spec <- logistic_reg(penalty = tune(), mixture = tune()) |>

set_engine("glmnet") |>

set_mode("classification")

wf <- workflow() |>

add_recipe(rec) |>

add_model(logistic_spec)

```

A model specification describes the statistical model at an abstract level, while `set_engine()` selects an implementation. This separation enables the same conceptual model to be used with different computational engines, subject to engine-specific capabilities.

For a regression task, alternative specifications might include:

```r

lm_spec <- linear_reg() |>

set_engine("lm")

rf_spec <- rand_forest(mtry = tune(), trees = 500, min_n = tune()) |>

set_engine("ranger") |>

set_mode("regression")

```

For classification, a random forest or boosted tree may capture nonlinearities and interactions, while penalized logistic regression offers a relatively transparent baseline. Baselines are essential: a complex model should demonstrate improvement over a simple, defensible alternative.

5. Cross-Validation and Hyperparameter Tuning

K-fold cross-validation

In K-fold cross-validation, the training data are divided into K approximately equal folds. For each iteration, K−1 folds form the analysis set and the remaining fold forms the assessment set. The reported resampling estimate is commonly the mean of the K assessment scores:

\[

\bar{M}=\frac{1}{K}\sum_{k=1}^{K}M_k,

\]

where \(M_k\) is the metric calculated on assessment fold k. This average estimates expected performance under the same data-generating conditions, subject to sampling variability and the validity of the resampling design. The folds should be independent enough for the intended interpretation; grouped or temporal dependence requires specialized resampling.

Five- or ten-fold cross-validation is common. Repeated cross-validation provides more estimates but increases computation. The standard deviation of fold scores is informative: two models with similar means may differ substantially in stability.

```r

set.seed(2025)

folds <- vfold_cv(train_data, v = 10, strata = churn)

```

Tuning

Tuning searches over hyperparameter configurations. For a penalized logistic model, `penalty` controls the strength of shrinkage and `mixture` controls the balance between L1 and L2 penalties. For a random forest, `mtry` controls the number of candidate predictors considered at a split and `min_n` controls the minimum observations required in a node.

```r

grid <- grid_regular(

penalty(range = c(-4, 0)),

mixture(range = c(0, 1)),

levels = 5

)

results <- tune_grid(

wf,

resamples = folds,

grid = grid,

metrics = metric_set(roc_auc, pr_auc, accuracy, sens, spec),

control = control_grid(save_pred = TRUE)

)

```

For `penalty`, the search is commonly conducted on a logarithmic scale because useful values can differ by orders of magnitude. Grid search is transparent but may be inefficient. Random search, Bayesian optimization, and racing methods can allocate computation more effectively, particularly when there are many hyperparameters.

The best configuration must be selected using a predefined primary metric. `select_best(results, metric = "pr_auc")` identifies the configuration with the highest estimated value. In some settings, `select_by_one_std_err()` selects a simpler configuration whose performance is within one standard error of the best result. This can improve parsimony and operational robustness.

After selection, `finalize_workflow()` inserts the chosen values, and `last_fit()` fits the finalized workflow on the complete training data and evaluates it once on the untouched test set.

6. Evaluation Metrics and Their Meaning

No metric is universally best. The correct metric depends on the outcome type, class balance, decision threshold, and costs of errors.

Regression metrics

For observed values \(y_i\) and predictions \(\hat y_i\), the mean absolute error is

\[

MAE=\frac{1}{n}\sum_{i=1}^{n}|y_i-\hat y_i|.

\]

MAE is expressed in the outcome’s units and gives equal linear weight to errors. The root mean squared error is

\[

RMSE=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat y_i)^2}.

\]

RMSE also uses the outcome’s units but penalizes large errors more heavily because errors are squared before averaging. Neither metric is automatically superior. RMSE is appropriate when large mistakes are disproportionately costly; MAE is often more robust to outliers.

The coefficient of determination is

\[

R^2=1-\frac{\sum_i(y_i-\hat y_i)^2}{\sum_i(y_i-\bar y)^2},

\]

where \(\bar y\) is the mean of the observed outcomes in the evaluation sample. This compares the model’s squared error with a mean-only benchmark under the same sample. It can be negative for a model worse than that benchmark and should not be interpreted as a universal percentage of explained causality.

Classification metrics

For binary classification, a confusion matrix contains true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Accuracy is

\[

Accuracy=\frac{TP+TN}{TP+TN+FP+FN}.

\]

Sensitivity, or recall, is \(TP/(TP+FN)\), while specificity is \(TN/(TN+FP)\). Precision is \(TP/(TP+FP)\). These ratios require the relevant denominator to be nonzero.

Accuracy can be misleading with class imbalance. If only 2% of transactions are fraudulent, predicting “not fraud” for every case gives 98% accuracy but detects no fraud. Precision-recall analysis is often more informative for rare positive classes. ROC AUC measures ranking performance across classification thresholds, whereas PR AUC emphasizes positive-class retrieval and is sensitive to prevalence.

A probability threshold converts predicted probabilities into actions. The default threshold of 0.5 has no universal justification. If the cost of missing a positive case is high, a lower threshold may be appropriate. Threshold selection should be performed using validation predictions, not the untouched test set.

Probabilities, calibration, and business metrics

A model predicting a probability of 0.80 should correspond, approximately, to an 80% event rate among comparable cases if it is well calibrated. A model can have strong discrimination but poor calibration. Calibration plots, calibration intercepts and slopes, and the Brier score can provide additional evidence. The Brier score for binary outcomes is the mean squared difference between predicted probability and the observed 0/1 outcome; lower values are better.

Business metrics translate model outputs into decisions. Examples include expected profit, cost per intervention, prevented loss, capacity utilization, or net revenue. A simple expected-cost calculation is

\[

Expected\ Cost = C_{FP}FP+C_{FN}FN+C_{I}N_I,

\]

where costs are defined by the organization, and \(N_I\) is the number of interventions. This equation is useful only when costs are measured consistently and include relevant operational consequences. If benefits and costs vary by case, case-specific expected utility is preferable.

7. Model Comparison and Statistical Discipline

Models should be compared on identical assessment folds, identical outcome definitions, and an agreed primary metric. Confidence intervals or bootstrap intervals can communicate uncertainty, although resampling intervals require careful interpretation because folds are not always independent.

A small observed difference in mean cross-validation score may not justify a more complex model, especially if its variance, maintenance cost, latency, or interpretability is worse. Comparison should therefore include:

• predictive performance and uncertainty;
• calibration and threshold behavior;
• subgroup performance;
• computational cost;
• explainability and governance requirements;
• data freshness and production reliability.

Repeated test-set evaluation is a form of adaptive overfitting. Nested cross-validation is appropriate when an unbiased estimate is needed after extensive model selection. In nested cross-validation, an outer loop estimates performance while an inner loop performs tuning; the outer assessment data are never used for selecting hyperparameters.

8. Illustrative Case: Predicting Customer Churn

Background and problem

Consider a fictional telecommunications provider that wants to identify customers likely to cancel service within the next 30 days. The objective is to prioritize retention calls, but the call center can contact only 8% of customers per week. The data contain contract type, tenure, monthly charges, service usage, complaint counts, and historical churn labels.

The target is binary. The observation date is available for each customer, and all predictors must be measured no later than that date. A post-cancellation billing adjustment is excluded because it is unavailable when the retention decision is made.

Analysis and approach

The analyst first removes duplicated customer-period records and assigns all records for a customer to the same partition. Because the organization wants future performance, a time-based split is used: earlier months form development data and the latest month forms the final test period. Within development data, time-based folds are used.

The recipe removes the customer identifier, imputes numeric missing values using training-fold medians, assigns an explicit level to unknown categorical values, groups rare categories, creates dummy variables, and removes zero-variance predictors. A penalized logistic regression is selected as a transparent baseline; a boosted-tree model is evaluated as a nonlinear alternative.

The primary development metric is PR AUC because churn is uncommon and the call center needs effective positive-case ranking. Secondary metrics include recall among the top 8% of ranked customers, precision at that capacity, calibration, and expected retention value. The threshold is not chosen at 0.5; instead, the model ranks customers and the operational cutoff is the top 8%.

Outcome

Suppose the boosted model has higher ROC AUC but only a marginally higher PR AUC than the penalized model. The logistic model is better calibrated and more stable across time folds. Its top-8% precision is slightly lower, but it is easier to explain to customer-service managers and has lower scoring latency. The organization selects the logistic model because the incremental benefit of the complex model does not justify its governance and maintenance costs.

Before deployment, the team evaluates performance by contract type and region, checks whether contact recommendations disproportionately affect protected groups, and establishes monitoring for drift, calibration, missingness, and response rates. The test period is reported once as the final estimate; subsequent monitoring uses new production data.

Learning points

The case demonstrates that a model is not chosen by one score alone. Time validity, capacity constraints, class imbalance, calibration, interpretability, and operational cost jointly determine usefulness. It also shows why feature availability and resampling design are part of the model, not administrative details.

9. Challenges, Governance, and Contemporary Practice

Predictive workflows face several limitations. Random samples may not represent future populations. Missingness may change after deployment. Labels may be delayed, selectively observed, or influenced by earlier interventions. A model can perform well historically while failing after a policy change or market shock.

Fairness assessment should examine error rates, calibration, selection rates, and benefits across relevant groups, while recognizing that many fairness criteria cannot generally be satisfied simultaneously when base rates differ. Group comparisons require adequate sample sizes and careful attention to legitimate business context. Protected attributes may be excluded from production prediction but still be necessary for auditing; this decision should be governed rather than assumed.

Privacy principles include data minimization, purpose limitation, access control, retention limits, and protection against re-identification. Regulatory obligations may require documentation of automated decisions, human review, explanation, or impact assessment. Feature stores and automated machine-learning platforms can improve consistency, but they can also conceal leakage and make assumptions difficult to audit.

Contemporary practice increasingly emphasizes reproducible pipelines, model cards, data and feature lineage, drift monitoring, champion–challenger testing, and continuous evaluation. Explainability tools such as permutation importance and SHAP-style methods can aid investigation, but explanations are not proof of causality and can be unstable under correlated predictors. Human oversight should be meaningful: a reviewer must have sufficient information, authority, and time to challenge a model output.

Best-Practice Workflow in tidymodels

A robust workflow generally follows these steps:

1. Define the prediction time, outcome horizon, decision, and cost structure.

2. Inspect data quality without using the test set to make modeling choices.

3. Create grouped, stratified, or temporal partitions appropriate to the problem.

4. Establish a simple baseline and document the primary metric.

5. Encode preprocessing and feature engineering in a recipe.

6. Combine the recipe and model in a workflow.

7. Tune using training-only resampling and a declared search space.

8. Examine mean performance, variability, calibration, and operational metrics.

9. Select and finalize the workflow using development data.

10. Evaluate once on the untouched test set.

11. Document limitations, fairness checks, reproducibility information, and monitoring plans.

Chapter Summary

Leakage-free predictive modeling requires more than fitting an algorithm. The analyst must define when predictors are available, partition data according to the dependence and time structure, learn preprocessing only from analysis data, and preserve an untouched test set. Tidymodels supports these requirements through resampling objects, recipes, model specifications, workflows, tuning functions, and metric tools.

Cross-validation estimates development performance, while final test evaluation estimates performance on new data only when the test set has remained untouched. Hyperparameters should be selected using a declared primary metric, and model comparisons should include variability, calibration, business consequences, interpretability, and operational constraints. Regression metrics such as MAE and RMSE emphasize different error properties; classification metrics such as precision, recall, ROC AUC, PR AUC, and calibration answer different questions. The most useful model is therefore the one that produces reliable, fair, and actionable decisions under realistic deployment conditions.

Key Terms

Data leakage: Use of information unavailable at prediction time or improperly obtained from assessment data.

Training set: Data used to estimate model parameters and learn preprocessing.

Validation set: Data used during development to compare models or tune choices.

Test set: Held-back data used for final performance assessment.

Cross-validation: Repeated partitioning into analysis and assessment samples.

Resampling: General process of generating alternative analysis and assessment samples.

Recipe: A tidymodels specification of preprocessing and feature engineering.

Workflow: A tidymodels object combining preprocessing and a model.

Hyperparameter: A model setting selected during development rather than learned directly as a parameter.

Feature engineering: Creation or transformation of predictors using domain knowledge.

Stratification: Sampling that approximately preserves outcome proportions.

Calibration: Agreement between predicted probabilities and observed frequencies.

Discrimination: Ability to rank observations according to outcome risk.

Sensitivity: Proportion of actual positives correctly identified.

Specificity: Proportion of actual negatives correctly identified.

Precision: Proportion of predicted positives that are actually positive.

PR AUC: Area under the precision-recall curve.

RMSE: Square root of mean squared prediction error.

Nested cross-validation: Outer resampling for evaluation with inner resampling for tuning.

Concept drift: Change over time in relationships between predictors and outcomes.

Review Questions

Short Answer Questions

1. Why must a final test set remain untouched during model development?

2. Define data leakage and give one example.

3. What is the role of a recipe in tidymodels?

4. Distinguish sensitivity from precision.

5. Why can accuracy be misleading for rare-event classification?

Long Answer Questions

1. Explain how training, validation, cross-validation, and test data contribute to a reliable predictive workflow.

2. Describe how a tidymodels recipe prevents preprocessing leakage during cross-validation.

3. Compare MAE, RMSE, and R-squared, including situations in which each may be useful.

4. Explain why temporal and grouped data require specialized partitioning strategies.

5. Discuss how statistical performance, calibration, interpretability, fairness, and business cost should influence model selection.

Analytical / Application Questions

1. A fraud model has 99% accuracy but detects only 10% of fraudulent transactions. Diagnose the problem and recommend metrics and a threshold strategy.

2. Design a leakage-free workflow for predicting next month’s hospital readmissions when several records belong to each patient.

3. Two models have similar PR AUC, but one is better calibrated and substantially easier to explain. Recommend a selection process for a capacity-constrained business and justify it.

Multiple Choice Questions with Answers

Question 1

Which data should be used to estimate an imputation median during a cross-validation iteration?

A. The complete data set, including all assessment folds

B. Only the analysis portion of that iteration

C. Only the final test set

D. A separate external data set in every case

Correct Answer: B. Only the analysis portion of that iteration

Question 2

What is the principal purpose of an untouched test set?

A. To tune hyperparameters repeatedly

B. To create dummy variables

C. To provide a final estimate after development decisions are complete

D. To replace all cross-validation

Correct Answer: C. To provide a final estimate after development decisions are complete

Question 3

Which metric is generally most appropriate when the positive class is rare and positive-case retrieval is important?

A. Precision-recall AUC

B. Accuracy alone

C. Mean absolute error

D. R-squared

Correct Answer: A. Precision-recall AUC

Question 4

In the formula for RMSE, why can large errors have a strong influence?

A. Errors are converted to categories

B. Errors are squared before averaging

C. Errors are divided by the prevalence

D. Errors are removed before calculation

Correct Answer: B. Errors are squared before averaging

Question 5

Which situation is a form of data leakage?

A. Using a training-fold median to impute its assessment fold

B. Removing a customer identifier before modeling

C. Using a post-outcome settlement amount as a predictor

D. Evaluating a finalized model once on a held-out test set

Correct Answer: C. Using a post-outcome settlement amount as a predictor

Question 6

Why are grouped splits useful when multiple rows belong to the same customer?

A. They guarantee perfect accuracy

B. They prevent related records from appearing in both development and assessment data

C. They eliminate the need for a target variable

D. They force all classes to have equal size

Correct Answer: B. They prevent related records from appearing in both development and assessment data

Question 7

What does calibration measure for a probabilistic classifier?

A. Whether predicted probabilities agree with observed event frequencies

B. The number of predictors in the model

C. The computational time required for prediction

D. Whether all classes have identical prevalence

Correct Answer: A. Whether predicted probabilities agree with observed event frequencies

Question 8

What is the role of a hyperparameter?

A. It is always a causal effect estimate

B. It is selected during model development to control model behavior

C. It is the outcome variable after prediction

D. It is an observation excluded from the data

Correct Answer: B. It is selected during model development to control model behavior

Question 9

Why might a temporal split be preferable to a random split?

A. It guarantees no missing values

B. It reflects the requirement that future predictions use past information

C. It automatically improves every metric

D. It removes the need for monitoring

Correct Answer: B. It reflects the requirement that future predictions use past information

Question 10

If false negatives are much more costly than false positives, what is a defensible modeling action?

A. Always use a threshold of 0.5

B. Ignore the confusion matrix

C. Evaluate cost-sensitive thresholds using validation predictions

D. Select the model using accuracy only

Correct Answer: C. Evaluate cost-sensitive thresholds using validation predictions