Machine Learning Foundations

Introduction and Context

Machine learning (ML) is a subfield of artificial intelligence concerned with methods that improve their performance on a task through experience. Instead of specifying every rule explicitly, a developer supplies data, defines an objective, and uses an algorithm to estimate a model. The model can then make predictions or decisions for new cases.

A typical machine-learning system contains four elements:

1. Data: observations describing cases, events, or entities.

2. Features: measurable input variables used by the model.

3. Target or label: the outcome to be predicted, when available.

4. Learning algorithm: a procedure that estimates model parameters from data.

Machine learning is useful because many real-world relationships are too complex, variable, or numerous to encode manually. Applications include credit-risk estimation, medical-image analysis, demand forecasting, recommendation, fraud detection, predictive maintenance, and natural-language processing. However, learning from data does not automatically produce a reliable system. Data may be incomplete or biased, the model may overfit, and an apparently accurate system may fail when deployed in a new environment.

This chapter develops the foundations needed to design and evaluate supervised learning systems responsibly.

Learning Objectives

By the end of this chapter, you should be able to:

• describe a supervised learning workflow;
• distinguish features, targets, training data, validation data, and test data;
• prepare data while avoiding leakage;
• explain regression and classification;
• select suitable models and evaluation metrics;
• analyze the bias–variance trade-off;
• explain regularization and generalization;
• interpret a machine-learning case from a technical and professional perspective; and
• identify practical, ethical, and governance risks in deployed models.

Supervised Learning and the End-to-End Workflow

In supervised learning, each training example consists of an input vector and an observed target. Let the dataset be

\[

D=\{(x_i,y_i)\}_{i=1}^{n},

\]

where \(x_i\) is the feature vector for observation \(i\), \(y_i\) is its target, and \(n\) is the number of observations. The learning algorithm uses the dataset to estimate a function \(f\) such that \(f(x)\) predicts the target for a previously unseen input.

For a regression problem, \(y\) is usually numerical, such as income or temperature. For a classification problem, \(y\) belongs to a finite set of classes, such as fraudulent or legitimate.

A robust workflow is iterative rather than purely linear:

1. Define the problem. Specify the decision to support, the prediction unit, the prediction time, the target, and the cost of errors.

2. Collect and understand data. Identify sources, sampling procedures, missingness, measurement definitions, and legal constraints.

3. Create an appropriate data split. Separate training, validation, and test information before fitting transformations or models.

4. Prepare features. Clean, encode, transform, and scale variables using procedures learned only from the training data.

5. Establish a baseline. A simple rule or statistical model provides a reference point.

6. Train candidate models. Fit several plausible algorithms and tune their hyperparameters.

7. Evaluate. Use metrics aligned with the real objective and examine performance across relevant subgroups.

8. Diagnose errors. Inspect false positives, false negatives, residuals, and unusual cases.

9. Deploy and monitor. Track data drift, performance, fairness, latency, and operational outcomes.

10. Update or retire the system. A model is not finished when it is first deployed.

The most important early question is not “Which algorithm is most advanced?” but “What exactly is being predicted, for whom, and at what point in time?” A target that is unavailable at prediction time, or that is influenced by the decision being automated, can make a model unusable despite impressive test accuracy.

Data Representation and Preparation

Features and Targets

A feature is an input variable used to make a prediction. Features may be numerical, categorical, ordinal, textual, temporal, spatial, or derived from other measurements. A target is the outcome the model is intended to predict. Feature engineering is the deliberate construction or transformation of features using domain knowledge.

A useful feature should be available at prediction time, measured consistently, and related to the target without creating unacceptable ethical or privacy risks. A feature can be statistically predictive yet operationally inappropriate. For example, a hospital model should not use a diagnosis recorded after treatment when the aim is to decide treatment at admission.

Data Quality

Common data problems include:

• missing values;
• duplicate records;
• inconsistent units or categories;
• outliers and sensor errors;
• incorrect labels;
• sampling bias;
• class imbalance; and
• changes in data definitions over time.

Missingness must be analyzed rather than automatically replaced. A missing value may be random, related to observed variables, or informative in itself. Simple approaches include median imputation for numerical variables and a separate “unknown” category for categorical variables. More sophisticated methods may model missingness, but they also introduce assumptions and must be validated.

Outliers should not be removed solely because they are unusual. They may be errors, rare but important events, or members of a minority population. Any exclusion rule should be documented and justified by domain knowledge.

Encoding and Scaling

Most algorithms require numerical inputs. Nominal categories can be represented using one-hot encoding, in which a category is represented by indicator variables. Ordinal categories may sometimes be assigned ordered numerical values, but this is appropriate only when the ordering and spacing are meaningful.

Scaling is important for distance-based and gradient-based methods. Standardization transforms a feature using

\[

z=\frac{x-\mu}{\sigma},

\]

where \(\mu\) and \(\sigma\) are the mean and standard deviation estimated from the training data, with \(\sigma>0\). The same training-set values must be used to transform validation, test, and production data. Tree-based methods are generally less sensitive to feature scale, whereas k-nearest neighbors, support vector machines, and many neural networks can be strongly affected by it.

Data Leakage

Data leakage occurs when information unavailable at prediction time enters the training process. Leakage produces overly optimistic evaluation and is one of the most serious practical errors in ML. Examples include:

• calculating an imputation mean using the entire dataset before splitting;
• using a post-outcome variable as a feature;
• allowing duplicate individuals in both training and test sets;
• selecting features using the complete dataset; and
• randomly splitting time-dependent data when future information can influence the past.

A preprocessing pipeline should fit every learned transformation on the training portion only. In temporal applications, training should generally use earlier periods and testing should use later periods. In grouped data, such as multiple records per patient or customer, splitting should often occur by group.

Training, Validation, and Test Data

The training set is used to estimate model parameters. The validation set supports model comparison and hyperparameter selection. The test set is reserved for a final, relatively unbiased estimate of performance after decisions have been made.

If the dataset is small, k-fold cross-validation divides the training data into \(k\) folds. The model is trained on \(k-1\) folds and evaluated on the remaining fold, repeating until every fold has served as validation data. The average score estimates expected performance within the sampled population. Stratified cross-validation preserves approximate class proportions for classification. Grouped or time-series cross-validation is needed when ordinary random folds would cause dependence or leakage.

The test set should not be repeatedly consulted during development. Repeated test-set decisions gradually turn the test set into another training resource, weakening its role as an independent check.

Regression

Regression predicts a numerical quantity. In linear regression, a common model is

\[

\hat{y}=\beta_0+\beta_1x_1+\cdots+\beta_px_p,

\]

where \(\hat{y}\) is the prediction, \(\beta_0\) is the intercept, \(\beta_j\) are coefficients, and \(x_j\) are features. The model is linear in its parameters; it can still use transformed features such as \(x^2\) if those transformations are defined in advance.

The ordinary least-squares estimate minimizes the residual sum of squares:

\[

\operatorname{RSS}=\sum_{i=1}^{n}(y_i-\hat{y}_i)^2.

\]

The squared-error objective penalizes large errors more heavily than small errors. Linear regression is interpretable and often a strong baseline, but it can be inadequate when relationships are nonlinear, interactions are important, or errors have unequal variance.

Important regression metrics include:

• Mean absolute error (MAE):

\[

\operatorname{MAE}=\frac{1}{n}\sum_{i=1}^{n}|y_i-\hat{y}_i|.

\]

It is expressed in the target’s units and is less sensitive to extreme errors than squared-error metrics.

• Mean squared error (MSE):

\[

\operatorname{MSE}=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2.

\]

• Root mean squared error (RMSE):

\[

\operatorname{RMSE}=\sqrt{\operatorname{MSE}}.

\]

RMSE has the target’s units but emphasizes large errors.

• Coefficient of determination \(R^2\):

\[

R^2=1-\frac{\sum_i(y_i-\hat y_i)^2}{\sum_i(y_i-\bar y)^2},

\]

where \(\bar y\) is the mean target in the evaluation sample. Relative to a constant-mean baseline, positive \(R^2\) indicates improvement, but a high \(R^2\) does not prove causal validity or guarantee good performance in deployment.

Worked Regression Example

Suppose actual demand values are \([10, 14, 20]\), and predictions are \([12, 13, 17]\). Absolute errors are \([2,1,3]\), so

\[

\operatorname{MAE}=\frac{2+1+3}{3}=2.

\]

Squared errors are \([4,1,9]\), giving

\[

\operatorname{MSE}=\frac{14}{3}\approx4.67,

\qquad

\operatorname{RMSE}=\sqrt{4.67}\approx2.16.

\]

The MAE says that the average absolute error is 2 demand units, while RMSE reflects the larger third error more strongly.

Classification

Classification predicts a discrete class. Binary classification has two classes; multiclass classification has more than two. A classifier may output a class label or a probability. A threshold converts a probability into a decision. For example, classifying an observation as positive when \(P(y=1\mid x)\geq0.5\) is only one possible policy; a different threshold may be appropriate when false negatives and false positives have different consequences.

A confusion matrix records:

• true positives (TP): positive cases correctly identified;
• true negatives (TN): negative cases correctly identified;
• false positives (FP): negative cases incorrectly flagged positive; and
• false negatives (FN): positive cases missed by the model.

Common metrics are:

\[

\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN},

\]

\[

\text{Precision}=\frac{TP}{TP+FP},

\qquad

\text{Recall} = \frac{TP}{TP+FN}.

\]

Precision measures the reliability of positive predictions. Recall, also called sensitivity or true-positive rate, measures the proportion of actual positives detected. The F1 score is the harmonic mean of precision and recall:

\[

F_1=\frac{2\,\text{Precision}\,\text{Recall}}{\text{Precision}+\text{Recall}},

\]

provided both denominators are defined. Accuracy can be misleading under class imbalance. In disease screening, for example, a model that predicts “not ill” for everyone may have high accuracy when illness is rare but zero recall for ill patients.

The receiver operating characteristic (ROC) curve displays true-positive rate against false-positive rate across thresholds. The area under this curve (ROC AUC) summarizes ranking performance across thresholds, but it may appear optimistic in highly imbalanced settings. Precision–recall curves can be more informative when the positive class is rare. Metrics should be selected according to operational costs, not convention alone.

Model Families and Selection

A baseline model may predict the training-set mean in regression or the majority class in classification. More useful baselines may reflect current business rules. Candidate models include:

• Linear and logistic models, which are efficient and relatively interpretable. Logistic regression models class probability using a logistic function and is suitable for linearly separable effects in the feature space.
• Decision trees, which recursively partition the feature space and can represent nonlinear interactions. Deep trees may overfit.
• Random forests, which average many randomized trees and often provide robust performance with limited preprocessing.
• Gradient-boosted trees, which build trees sequentially to correct previous errors and can be highly effective on structured data, although tuning and interpretation require care.
• k-nearest neighbors, which predicts from nearby training examples and is sensitive to scaling, irrelevant features, and the choice of neighborhood size.
• Support vector machines, which seek a separating boundary and can use kernels for nonlinear relationships; they may be computationally expensive for very large datasets.
• Neural networks, which can represent complex functions and are especially important for images, speech, and language, but generally require careful tuning, substantial data, and appropriate computational resources.

Model selection should consider predictive performance, calibration, interpretability, latency, memory, maintenance, security, fairness, and the consequences of errors. The most complex model is not automatically the best model.

Bias, Variance, and Generalization

Generalization is the ability to perform well on new data drawn from the relevant deployment distribution. A model that memorizes training examples may have low training error but poor test performance; this is overfitting. A model too simple to capture the underlying pattern may perform poorly on both training and test data; this is underfitting.

The bias–variance framework explains one source of this behavior. For squared-error prediction at a fixed input, expected prediction error can be decomposed conceptually into squared bias, variance, and irreducible noise:

\[

\mathbb{E}\left[(Y-\hat f(x))^2\mid x\right]

=\operatorname{Bias}(\hat f(x))^2+\operatorname{Var}(\hat f(x))+\sigma^2,

\]

under the assumptions that the data-generating relationship is \(Y=f(x)+\epsilon\), that \(\mathbb{E}[\epsilon\mid x]=0\), and that \(\operatorname{Var}(\epsilon\mid x)=\sigma^2\). Bias reflects systematic error from restrictive assumptions; variance reflects sensitivity to the particular training sample; irreducible noise cannot be eliminated by a better-fitting model.

Increasing model flexibility often decreases bias but increases variance. The objective is not to minimize training error alone, but expected error on future cases. Learning curves, cross-validation, error analysis, and simpler baselines help diagnose this trade-off.

Regularization

Regularization discourages unnecessarily complex solutions. Ridge regression adds an L2 penalty:

\[

\min_{\beta_0,\beta}\left\{\sum_{i=1}^{n}(y_i-\beta_0-x_i^T\beta)^2+\lambda\sum_{j=1}^{p}\beta_j^2\right\},

\]

where \(\lambda\geq0\) controls penalty strength. The intercept is commonly not penalized. Larger \(\lambda\) shrinks coefficients toward zero and can reduce variance, especially when features are correlated.

Lasso regression uses an L1 penalty:

\[

\min_{\beta_0,\beta}\left\{\sum_{i=1}^{n}(y_i-\beta_0-x_i^T\beta)^2+\lambda\sum_{j=1}^{p}|\beta_j|\right\}.

\]

L1 regularization can set some coefficients exactly to zero, producing a sparse model. It should not automatically be interpreted as reliable causal feature selection, particularly when predictors are correlated. Elastic net combines L1 and L2 penalties.

Regularization also appears in tree depth limits, minimum leaf sizes, early stopping, dropout, data augmentation, and other techniques. The regularization strength is a hyperparameter selected using validation procedures, not by examining the test set.

Calibration, Interpretability, and Fairness

A classifier can rank cases accurately while producing poorly calibrated probabilities. A model is calibrated when, among cases assigned probability approximately 0.7, about 70% are positive in the relevant population. Calibration matters when probabilities guide resource allocation or risk thresholds.

Interpretability is not a single property. A linear model may provide coefficient-level explanations, while a tree provides rule-like structure. Post hoc methods such as feature permutation or local explanation techniques can be useful, but explanations may be unstable or mistaken for causal claims. Explanations should be checked against domain knowledge.

Fairness requires examining how data, labels, features, thresholds, and errors affect groups. Relevant analyses may compare selection rates, false-positive rates, false-negative rates, calibration, and performance across groups. Many fairness criteria cannot all be satisfied simultaneously when base rates differ. Therefore, fairness decisions require explicit organizational and ethical judgment rather than a single universal metric.

Privacy, security, and governance also matter. Personal data should be collected and retained lawfully, access should be controlled, and model documentation should identify purpose, provenance, limitations, and responsible owners. Regulations may require transparency, human oversight, impact assessment, or the right to contest significant automated decisions. A technically accurate model can still be unacceptable if it violates privacy, discriminates, or cannot be responsibly monitored.

Illustrative Case: Predicting Customer Loan Delinquency

Background and Problem

A fictional regional bank wants to identify accounts at risk of becoming 60 days delinquent within the next six months. The purpose is to offer voluntary financial counseling, not to make an automatic lending refusal. The bank has transaction summaries, repayment history, account age, income information, and customer-service records.

Analysis

The team first defines the prediction time as the end of each monthly reporting period. It excludes information recorded after that date. Because customers appear repeatedly over time, a random row-level split could place the same customer in training and testing and produce overly optimistic results. The team therefore uses a temporal split: earlier months for training and validation, later months for testing.

The target is imbalanced: only 6% of account-months become delinquent. Accuracy is therefore unsuitable as the primary metric. The team reports recall, precision, PR AUC, calibration, and expected intervention cost. It also evaluates performance by age range, geographic region, and language group.

Approach and Solution

The bank establishes a baseline using recent missed-payment indicators. It compares regularized logistic regression with gradient-boosted trees. Numerical imputations and categorical encoding are fitted within a training-only pipeline. Hyperparameters are selected by time-aware validation. The final threshold is chosen with operations staff: counselors can contact only a limited number of customers, and false alarms consume scarce resources.

The model is calibrated on validation data and produces a risk score rather than an automatic adverse decision. Counselors receive a short explanation based on recent repayment behavior and account changes, while prohibited or proxy-sensitive variables are excluded where appropriate. Every intervention and outcome is logged.

Outcome

On the untouched future test period, the selected system identifies a substantially larger share of future delinquencies than the baseline within the same outreach capacity. However, performance declines after a change in economic conditions. Monitoring detects the drift, and the bank pauses automated expansion, reviews the target definition, and retrains only after a new validation cycle.

Learning Points

The case demonstrates that model quality depends on problem definition, temporal evaluation, imbalance-aware metrics, leakage prevention, threshold decisions, fairness review, calibration, and monitoring. It also illustrates that the model’s purpose—supporting assistance rather than denying service—affects acceptable design and governance.

Practical Best Practices

• Define the target and prediction time before selecting features.
• Split data according to real deployment conditions.
• Put preprocessing and modeling in one reproducible pipeline.
• Use a meaningful baseline.
• Select metrics according to error costs and class prevalence.
• Tune hyperparameters through cross-validation or a validation set only.
• Preserve a final test set and report uncertainty where feasible.
• Inspect subgroup performance and calibration.
• Prefer the simplest model that meets the requirements.
• Document data sources, assumptions, exclusions, versions, and limitations.
• Monitor drift, missingness, latency, fairness, and real-world outcomes after deployment.
• Provide a process for human review, appeal, correction, and model retirement.

Challenges and Limitations

Machine learning cannot recover information absent from the data, and prediction is not the same as explanation or causation. Historical labels may reflect past institutional decisions rather than objective outcomes. Distribution shift can invalidate a model when user behavior, policy, technology, or the environment changes. Small datasets produce uncertain estimates, while large datasets can amplify systematic bias. Correlated observations can make standard validation misleading. Finally, a model may optimize a measurable proxy while harming the broader objective—for example, maximizing clicks rather than user well-being.

Contemporary Developments

Current practice increasingly emphasizes foundation models, automated machine-learning tools, probabilistic prediction, privacy-preserving learning, and responsible MLOps. These developments can improve capability and productivity, but they do not remove the need for careful data design, evaluation, documentation, and human accountability. Automated model search still requires a correctly specified objective and leakage-free validation. Generative and multimodal systems also require additional checks for hallucination, security, provenance, and misuse.

Chapter Summary

Supervised learning estimates a function from labeled examples to predict future targets. A reliable workflow begins with a precise problem definition, continues through leakage-free data preparation and appropriate train–validation–test design, and ends with deployment monitoring. Regression predicts numerical outcomes and can be evaluated using MAE, MSE, RMSE, and related measures. Classification predicts classes or probabilities and requires confusion-matrix metrics whose relevance depends on error costs and class imbalance.

Model choice should balance performance, complexity, interpretability, operational constraints, fairness, and governance. Underfitting and overfitting are connected to the bias–variance trade-off, while regularization controls complexity and can improve generalization. Responsible practice requires temporal or grouped validation when appropriate, subgroup analysis, calibration, documentation, privacy protection, human oversight, and ongoing monitoring. Machine learning is therefore both a technical modeling activity and an organizational decision-making process.

Key Terms

Feature: An input variable used by a model.

Target: The outcome a supervised model is trained to predict.

Supervised learning: Learning from examples containing inputs and observed targets.

Regression: Prediction of a numerical or continuous quantity.

Classification: Prediction of a discrete class or class probability.

Training set: Data used to estimate model parameters.

Validation set: Data used to select models or hyperparameters.

Test set: Held-back data used for final performance estimation.

Feature engineering: Construction or transformation of predictive variables.

Data leakage: Use of information that would not be available at prediction time.

Cross-validation: Repeated train–validation evaluation using partitions of data.

Overfitting: Fitting training data so closely that performance on new data deteriorates.

Underfitting: Using a model too limited to capture important patterns.

Bias: Systematic prediction error caused by restrictive assumptions or flawed data.

Variance: Sensitivity of a fitted model to the particular training sample.

Regularization: A method that penalizes complexity to improve generalization.

Precision: The proportion of predicted positives that are actually positive.

Recall: The proportion of actual positives that are detected.

Calibration: Agreement between predicted probabilities and observed frequencies.

Data drift: Change in the distribution or meaning of data over time.

Review Questions

Short Answer Questions

1. What distinguishes a feature from a target in supervised learning?

2. Why should preprocessing transformations be fitted using training data only?

3. Define precision and recall.

4. What is the purpose of a validation set?

5. Why can accuracy be misleading for an imbalanced classification problem?

Long Answer Questions

1. Explain the complete supervised learning workflow, from problem definition through post-deployment monitoring.

2. Compare linear regression, decision trees, and gradient-boosted trees in terms of assumptions, interpretability, and likely use cases.

3. Explain the bias–variance trade-off and its relationship to underfitting and overfitting.

4. Compare ridge and lasso regularization, including their penalties and practical implications.

5. Discuss why fairness, calibration, privacy, and human oversight are important in deployed machine-learning systems.

Analytical / Application Questions

1. A fraud dataset contains 1% fraudulent transactions. Recommend evaluation metrics and explain why accuracy alone is inadequate.

2. A hospital has several records per patient, including a procedure code entered after treatment. Identify two risks in a naive modeling workflow and propose corrective actions.

3. A model performs well on historical data but poorly after a major economic change. Analyze possible causes and design a monitoring and response plan.

Multiple Choice Questions with Answers

Question 1

Which dataset should be reserved for a final relatively unbiased estimate after model and hyperparameter decisions are complete?

A. Training set

B. Validation set

C. Test set

D. Feature-engineering set

Correct Answer: C. Test set

Question 2

A model predicts 90 positive cases, of which 72 are truly positive. What is its precision?

A. 0.20

B. 0.72

C. 0.80

D. 0.90

Correct Answer: C. 0.80

Question 3

Which situation is an example of data leakage?

A. Scaling features using training-set statistics

B. Using a post-treatment variable to predict treatment success at admission

C. Evaluating a model on a later time period

D. Using a baseline majority-class classifier

Correct Answer: B. Using a post-treatment variable to predict treatment success at admission

Question 4

Which metric is expressed in the same units as the regression target and places greater emphasis on large errors than MAE?

A. RMSE

B. Precision

C. Recall

D. ROC AUC

Correct Answer: A. RMSE

Question 5

What is the primary effect of increasing the regularization parameter in ridge regression?

A. It removes the test set

B. It generally shrinks coefficients toward zero

C. It guarantees zero training error

D. It converts regression into classification

Correct Answer: B. It generally shrinks coefficients toward zero

Question 6

Recall is defined as:

A. TP divided by TP plus FP

B. TN divided by TN plus FN

C. TP divided by TP plus FN

D. TP plus TN divided by all observations

Correct Answer: C. TP divided by TP plus FN

Question 7

Which validation strategy is generally most appropriate when future observations must be predicted from past observations?

A. Randomly mixing all observations into folds

B. Temporal or forward-chaining validation

C. Using only the majority class

D. Evaluating on the training data

Correct Answer: B. Temporal or forward-chaining validation

Question 8

In the bias–variance framework, a highly flexible model that changes substantially with the training sample typically has:

A. High variance

B. Zero irreducible noise

C. High calibration by definition

D. No possibility of overfitting

Correct Answer: A. High variance

Question 9

What does calibration of predicted probabilities mean?

A. Every feature has mean zero

B. Predicted probabilities correspond to observed frequencies

C. The model has 100% accuracy

D. All demographic groups have identical base rates

Correct Answer: B. Predicted probabilities correspond to observed frequencies

Question 10

Why might precision–recall analysis be especially useful for rare-event detection?

A. It ignores false positives

B. It focuses on positive predictive performance and detection of positives

C. It guarantees causal interpretation

D. It requires no labeled data

Correct Answer: B. It focuses on positive predictive performance and detection of positives