Predictive Modeling and Machine Learning

Introduction

Predictive modeling is the process of using existing data to estimate an unknown or future outcome. It is central to data science because organizations rarely need data merely to describe what has happened; they also want to anticipate demand, identify risk, recommend actions, and allocate resources. Machine learning provides a collection of statistical and computational methods for constructing predictive models from data.

This chapter focuses on supervised learning, in which a model learns from examples that include both input variables and a known target outcome. Two introductory supervised-learning tasks are especially important:

• Regression, which predicts a numerical quantity such as sales revenue or delivery time.
• Classification, which predicts a category such as fraud/not fraud or customer churn/no churn.

The practical quality of a model depends not only on the algorithm selected, but also on data preparation, separation of training and evaluation data, appropriate metrics, comparison with simple baselines, and responsible interpretation.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain supervised learning and distinguish regression from classification.
2. Describe the roles of features, targets, training data, validation data, and test data.
3. Prepare numerical and categorical data for modeling.
4. Build introductory linear-regression and classification models using suitable software.
5. Select and interpret common evaluation metrics.
6. Compare a machine-learning model with an appropriate baseline.
7. Identify common sources of overfitting, leakage, bias, and inappropriate use.

1. The Supervised-Learning Framework

A supervised-learning dataset consists of observations for which the desired outcome is known. Each observation has input variables, called features, and an outcome, called the target, label, or response.

For observation i, the data may be represented as:

(x_i, y_i)

where x_i is the vector of features and y_i is the observed target. A learning algorithm uses a training dataset to estimate a function:

Here, f̂ is the fitted model and ŷ is its prediction. The hat symbol indicates an estimate rather than a true or known value.

In regression, y is numerical. In classification, y is a class label. For binary classification, the model may estimate a probability such as:

A classification decision is then made by comparing p̂ with a threshold, often 0.5, although the threshold should be chosen according to the costs of different errors.

Features and targets

A feature might be age, temperature, transaction value, or number of previous purchases. A target might be annual income, energy consumption, or whether a customer cancels a subscription. Features can be:

• Numerical: measured or counted quantities.
• Categorical: labels such as region or product type.
• Binary: two-valued variables such as yes/no.
• Ordinal: categories with an order, such as low, medium, and high.
• Text, image, or time-series variables: more complex inputs requiring specialized representation.

The target must not be used as an input feature. Variables that become available only after the prediction time must also be excluded. Including such information creates target leakage and produces misleadingly strong results.

2. Regression and Classification

Regression

Regression predicts a continuous or approximately continuous quantity. Examples include:

• predicting house prices;
• estimating hospital length of stay;
• forecasting monthly electricity demand;
• estimating delivery time.

A simple linear-regression model with one feature is:

where β₀ is the intercept, β₁ is the slope, and ε_i is the error for observation i. The fitted prediction is:

The slope describes the estimated change in the target for a one-unit increase in the feature, assuming the model form is suitable. In multiple linear regression:

The coefficients are estimated from training data, commonly by minimizing the sum of squared errors:

This objective is appropriate when large errors should be penalized strongly and the usual linear-model assumptions are reasonably acceptable. It does not prove that the relationship is causal.

Classification

Classification predicts a discrete class. Binary classification has two classes; multiclass classification has three or more. Examples include predicting whether an email is spam, whether a loan will default, or which product category a customer will choose.

Logistic regression is a useful introductory classifier. It models the log-odds of class 1:

where p is the probability that the observation belongs to class 1. Equivalently:

The logistic function constrains predicted probabilities to the interval 0 to 1. A threshold converts probabilities into class predictions. A threshold of 0.5 is not automatically optimal: in medical screening, for example, missing a true case may be more serious than investigating a false positive.

Other introductory classifiers include decision trees, k-nearest neighbours, and naïve Bayes. Each has different assumptions and practical strengths.

3. Generalization, Overfitting, and Underfitting

A model is useful when it performs well on new data, not merely on the observations used to fit it. This property is called generalization.

• Underfitting occurs when a model is too simple to capture important structure. It performs poorly on both training and new data.
• Overfitting occurs when a model learns random variation or peculiarities of the training sample. Training performance is strong, but performance on new data is weak.
• The bias–variance trade-off describes the tension between overly rigid models and overly sensitive models.

Model complexity can be controlled using feature selection, regularization, tree depth limits, minimum sample requirements, or choice of algorithm. A model should be no more complex than justified by the data and the decision problem.

4. Training, Validation, and Test Data

A sound evaluation design separates model development from final assessment.

1. Training set: used to fit model parameters.
2. Validation set: used to compare models, tune hyperparameters, and select thresholds.
3. Test set: held back until the final model and decisions have been chosen.

A common arrangement is 60% training, 20% validation, and 20% test, although the choice depends on dataset size. With limited data, k-fold cross-validation can provide a more stable estimate. In k-fold cross-validation, the training data are divided into k folds; each fold is used once for validation while the other k − 1 folds are used for fitting.

The test set should be treated as an independent examination. Repeatedly inspecting test performance turns the test set into an informal validation set and makes the reported estimate optimistic.

Stratification and time order

For classification, stratified splitting preserves approximately the same class proportions in each partition. This is particularly important when the positive class is uncommon.

For time-dependent prediction, random splitting can be invalid because it may allow future information to influence a model intended for the past. A chronological split is usually more appropriate: earlier observations are used for training and later observations for validation or testing.

5. Preprocessing and Data Pipelines

Preprocessing must be learned from the training data and then applied unchanged to validation and test data. This principle prevents leakage.

Missing values

Possible approaches include:

• removing observations when missingness is limited and plausibly random;
• imputing numerical values with a training-set median or mean;
• imputing categorical values with the training-set mode or an explicit “unknown” category;
• adding a missingness indicator when absence itself may be informative.

The imputation statistic must be calculated using the training set only.

Scaling numerical features

Scaling is particularly important for distance-based methods, gradient-based methods, and regularized models. Standardization is commonly defined as:

where μ is the training-set mean and σ is the training-set standard deviation. The transformed variable is dimensionless. If σ = 0, the feature has no variation in the training set and should normally be removed or handled separately.

Tree-based models generally do not require scaling because their decisions are based on thresholds rather than distances.

Categorical encoding

A nominal categorical variable can be represented using one-hot encoding. If a variable has k categories, k indicator columns can be created, or k − 1 columns can be used with an intercept to avoid exact linear dependence in ordinary linear regression. The encoding scheme must be fitted on training data and applied consistently to new data. Unseen categories require a defined policy, such as an “unknown” category.

Pipelines

A software pipeline combines preprocessing and modeling into a single reproducible object. A typical pipeline is:

raw data → split → fit preprocessing on training data → transform data → fit model → evaluate

Pipelines reduce accidental leakage and ensure that the same transformations are used during deployment.

6. Measuring Predictive Performance

No single metric is best for every problem. Metrics should reflect the decision context.

Regression metrics

For n observations, the mean absolute error is:

MAE has the same units as the target and represents the average absolute prediction error.

The mean squared error is:

MSE is expressed in squared target units and gives greater weight to large errors. Its square root, root mean squared error, is:

RMSE returns to the target’s units. It is especially sensitive to large mistakes.

The coefficient of determination is:

where SSE = ∑(y_i − ŷ_i)² and SST = ∑(y_i − ȳ)², with ȳ the mean target in the evaluation set. R² measures improvement relative to predicting the evaluation-set mean under the stated calculation. It can be negative when a model performs worse than that mean baseline; it is not a universal measure of accuracy and does not establish causation.

Classification metrics

A binary classifier produces a confusion matrix:

Accuracy is:

Precision and recall are:

Precision answers: among predicted positives, how many were positive? Recall answers: among actual positives, how many were found? The F1 score is the harmonic mean:

These formulas require nonzero denominators; software must define how undefined cases are handled. Accuracy can be misleading with imbalanced classes. For example, a classifier that always predicts the majority class may achieve high accuracy while detecting no minority cases.

A probability model can also be evaluated using log loss or a calibration assessment. A well-calibrated model is one for which predictions of 0.7 correspond approximately to events occurring 70% of the time in comparable groups.

7. Baselines and Model Comparison

A baseline is a simple reference method against which a predictive model is compared. Appropriate baselines include:

• predicting the training-set mean or median for regression;
• predicting the majority class for classification;
• using the previous period’s value in a time-series problem;
• applying an existing operational rule.

A complex model is not automatically useful. It should improve on the baseline by a meaningful amount, remain stable across validation samples, and be acceptable in terms of cost, interpretability, fairness, and operational risk.

For regression, compare MAE or RMSE with the baseline on the same test observations. For classification, compare metrics such as recall, precision, and balanced accuracy, not just overall accuracy. Confidence intervals or repeated cross-validation can help communicate uncertainty, especially when the test sample is small.

8. Practical Implementation with Python

A typical implementation can use pandas and scikit-learn. The following outline illustrates the workflow; column names and parameters must be adapted to the dataset.

from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score

X = data.drop(columns=["target"])
y = data["target"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)

numeric_features = ["age", "account_value"]
categorical_features = ["region", "plan"]

numeric_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])

categorical_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OneHotEncoder(handle_unknown="ignore"))
])

preprocess = ColumnTransformer([
    ("numeric", numeric_pipeline, numeric_features),
    ("categorical", categorical_pipeline, categorical_features)
])

model = Pipeline([
    ("preprocess", preprocess),
    ("classifier", LogisticRegression(max_iter=1000))
])

model.fit(X_train, y_train)
predictions = model.predict(X_test)

print(accuracy_score(y_test, predictions))
print(precision_score(y_test, predictions, zero_division=0))
print(recall_score(y_test, predictions, zero_division=0))

The example uses a single train-test split for clarity. In a professional project, a validation strategy or cross-validation would normally be added before the final test evaluation. The preprocessing steps are inside the pipeline, so their parameters are learned only from the training data.

For regression, replace the classifier and metrics with a regressor such as LinearRegression or RandomForestRegressor, and report MAE or RMSE.

9. Worked Example: Customer-Churn Classification

Consider a fictional telecommunications provider seeking to identify customers likely to cancel within the next month. The dataset contains customer tenure, monthly charge, contract type, support contacts, and a churn label.

Problem and preparation

The target is binary: churn or no churn. The prediction must use information available before the next month begins. A field recording a cancellation request already submitted would be leakage and must be excluded.

The data are divided into training and test sets using an 80:20 stratified split. Numerical missing values are imputed with training-set medians and standardized. Contract type is one-hot encoded. A logistic-regression pipeline is fitted on the training data.

Baseline and results

Suppose the test set contains 200 customers, of whom 40 churn. A majority-class baseline predicts “no churn” for every customer. It obtains:

However, its recall for churn is 0 because it identifies none of the 40 churners.

Suppose the logistic model produces the following confusion matrix:

Then:

The model improves accuracy only modestly over the baseline, but its recall is substantially better. Whether it is valuable depends on the cost of retention offers and the cost of missing a likely churner. If retention resources are limited, the probability threshold could be increased to improve precision; if retaining customers is highly important, a lower threshold may be appropriate.

Interpretation and action

The model should support, not replace, business judgment. A retention team might contact high-probability customers with a suitable offer. Before deployment, the organization should test whether the intervention actually reduces churn, rather than assuming that prediction alone creates value. Model performance should be monitored over time because customer behavior, products, and market conditions change.

10. Professional, Ethical, and Governance Considerations

Predictive models can reproduce historical inequities. If past decisions reflect discrimination or unequal access, a model trained on those decisions may encode the same patterns. Important governance practices include:

• documenting data sources, target definitions, exclusions, and preprocessing;
• checking performance across relevant demographic or operational groups;
• examining both error rates and practical consequences;
• limiting access to sensitive personal data;
• obtaining an appropriate legal basis and consent where required;
• explaining decisions at a level suitable for affected people;
• providing review or appeal procedures for high-impact decisions;
• monitoring drift, calibration, and performance after deployment.

Privacy regulations may restrict the collection, use, retention, and sharing of personal data. Data minimization and purpose limitation are therefore important. A technically accurate model may still be inappropriate if its use is unfair, intrusive, insecure, or inconsistent with organizational policy.

Common failure modes

1. Leakage: using information unavailable at prediction time.
2. Inconsistent preprocessing: transforming training and production data differently.
3. Test-set overuse: repeatedly tuning against the final test set.
4. Unrepresentative samples: applying a model outside the population on which it was developed.
5. Metric blindness: optimizing accuracy while ignoring error costs.
6. Data drift: relationships or feature distributions changing over time.
7. False causal interpretation: treating predictive association as proof that changing a feature will change the outcome.
8. Uncalibrated probabilities: treating a score as a reliable probability without testing calibration.

11. Contemporary Developments

Modern predictive modeling increasingly combines traditional statistical models with ensemble methods, neural networks, automated machine-learning tools, and model-monitoring platforms. These technologies can improve predictive performance, but they do not remove the need for sound problem definition, representative data, leakage prevention, evaluation design, and governance.

Explainable-AI methods can help users understand which features influence predictions, although explanations are not always causal and may themselves require validation. Responsible practice emphasizes reproducibility, version-controlled code, documented datasets, clear model cards or technical reports, and continuous monitoring after deployment.

Best-Practice Workflow

A practical workflow is:

1. Define the decision, prediction time, target, and success metric.
2. Inspect data quality, missingness, class balance, and sampling process.
3. Separate features from the target and remove leakage.
4. Split data using a method suitable for the sampling and time structure.
5. Construct preprocessing inside a pipeline.
6. Establish a meaningful baseline.
7. Fit one or more appropriately simple models.
8. Tune using validation data or cross-validation.
9. Evaluate once on the held-out test set.
10. Examine errors, subgroup performance, calibration, and operational consequences.
11. Document assumptions, limitations, and intended use.
12. Monitor the model after deployment and retrain only through a controlled process.

Chapter Summary

Supervised learning uses labeled examples to estimate a relationship between features and a target. Regression predicts numerical values, while classification predicts categories or class probabilities. Linear regression and logistic regression provide interpretable introductory models, but their usefulness depends on data quality and suitable assumptions.

Reliable predictive modeling requires separation of training, validation, and test data; prevention of target leakage; consistent preprocessing; and evaluation against an appropriate baseline. Regression may be evaluated using MAE, MSE, RMSE, and R². Classification requires careful use of confusion-matrix metrics such as precision, recall, F1 score, and accuracy, particularly when classes are imbalanced.

A model should be assessed not only for predictive performance but also for fairness, privacy, interpretability, cost, stability, and operational value. Responsible deployment includes documentation, subgroup analysis, monitoring, and recognition that prediction does not establish causation.

Key Terms

Supervised learning: Learning from examples that contain features and known target outcomes.

Feature: An input variable used by a predictive model.

Target: The outcome a model is trained to predict.

Regression: Prediction of a numerical outcome.

Classification: Prediction of a discrete class or category.

Training set: Data used to estimate model parameters.

Validation set: Data used for model selection and tuning.

Test set: Held-out data used for final evaluation.

Overfitting: Learning training-specific noise rather than general patterns.

Underfitting: Using a model too simple to capture important structure.

Data leakage: Use of information that would not be available at prediction time.

Preprocessing: Transformations applied before modeling, such as imputation and encoding.

Standardization: Rescaling a feature using its training mean and standard deviation.

One-hot encoding: Representing categories as binary indicator variables.

Baseline: A simple reference method used for comparison.

Precision: The proportion of predicted positives that are actually positive.

Recall: The proportion of actual positives that are correctly identified.

Calibration: Agreement between predicted probabilities and observed frequencies.

Data drift: Change in data distributions or relationships after model development.

Generalization: Performance of a model on previously unseen data.

Review Questions

Short Answer Questions

1. What distinguishes supervised learning from unsupervised learning?
2. Give two examples of regression problems and two examples of classification problems.
3. Why must preprocessing parameters be learned from training data only?
4. What is the purpose of a baseline model?
5. Define precision and recall.

Long Answer Questions

1. Explain the roles of training, validation, and test sets, and describe the danger of repeatedly using the test set during model development.
2. Compare linear regression, logistic regression, decision trees, and k-nearest neighbours in terms of assumptions, interpretability, and preprocessing requirements.
3. Explain overfitting and underfitting, including at least three methods for controlling overfitting.
4. Discuss why accuracy may be inappropriate for an imbalanced classification problem and recommend alternative evaluation measures.
5. Describe a complete, leakage-resistant workflow for building and deploying a supervised-learning model.

Analytical / Application Questions

1. A fraud classifier has high accuracy but identifies very few fraudulent transactions. Analyze why this may occur and recommend metrics, threshold changes, and validation checks.
2. A model predicts next month’s sales using randomly shuffled data from five years. Explain when random splitting may be invalid and design a more suitable evaluation strategy.
3. Using the churn confusion matrix in this chapter, assess whether the logistic model is preferable to the majority baseline. State what additional business information is needed before deployment.

Multiple Choice Questions with Answers

Question 1

Which task is a regression problem?

A. Predicting whether an email is spam

B. Predicting a customer’s monthly spending

C. Assigning a product to a category

D. Predicting whether a machine will fail: yes or no

Correct Answer: B. Predicting a customer’s monthly spending

Question 2

What is the primary purpose of a test set?

A. To fit model parameters

B. To calculate training-set imputation values

C. To provide a final assessment of the selected model

D. To create additional features after deployment

Correct Answer: C. To provide a final assessment of the selected model

Question 3

Which action creates target leakage?

A. Standardizing a feature using training-set statistics

B. One-hot encoding a categorical feature

C. Using a cancellation request already submitted to predict future cancellation

D. Removing a constant feature

Correct Answer: C. Using a cancellation request already submitted to predict future cancellation

Question 4

If TP = 30 and FP = 10, what is precision?

A. 0.25

B. 0.50

C. 0.75

D. 0.90

Correct Answer: C. 0.75

Question 5

Which metric is expressed in the same units as the regression target?

A. MAE

B. MSE

C. R²

D. Log loss

Correct Answer: A. MAE

Question 6

Why is stratification often used when splitting classification data?

A. To guarantee perfect accuracy

B. To preserve approximate class proportions in each split

C. To remove all missing values

D. To force features to have equal scales

Correct Answer: B. To preserve approximate class proportions in each split

Question 7

Which model generally does not require feature scaling because it uses threshold splits?

A. Decision tree

B. k-nearest neighbours

C. Logistic regression with regularization

D. Distance-based clustering

Correct Answer: A. Decision tree

Question 8

A model has excellent training performance but poor test performance. What is the most likely issue?

A. Underfitting

B. Overfitting

C. Class balancing

D. Standardization

Correct Answer: B. Overfitting

Question 9

If a classifier must minimize missed positive cases, which measure is especially important?

A. Recall

B. Specificity only

C. Mean squared error

D. R²

Correct Answer: A. Recall

Question 10

Which statement about a baseline is correct?

A. It must always be a neural network

B. It is a simple reference used to judge whether a model adds value

C. It is calculated only after deployment

D. It replaces the need for a test set

Correct Answer: B. It is a simple reference used to judge whether a model adds value