10. Foundations of Machine Learning

Introduction and Context

Machine learning (ML) is the study of computational methods that enable systems to learn patterns from data and use those patterns to make predictions, classifications, recommendations, or decisions. It is a central area of data science because many practical problems cannot be solved efficiently through manually written rules. Instead, the system is provided with examples, and an algorithm estimates a relationship between inputs and outputs.

Machine learning does not replace statistical reasoning; it extends it through algorithms capable of handling large numbers of observations, variables, and complex relationships. A data scientist must therefore understand not only how to train a model, but also how to define the problem, prepare data, select appropriate evaluation criteria, identify sources of error, and interpret the result responsibly.

The central distinction in this chapter is between supervised learning, in which the desired output is known for historical examples, and unsupervised learning, in which the algorithm searches for structure without a specified target. Regression, classification, clustering, and dimensionality reduction provide four foundational families of techniques. Their usefulness depends on the quality of the data, the assumptions of the method, and the decisions that follow from the model.

Learning Objectives

After completing this chapter, the reader should be able to:

1. Explain the purpose and terminology of machine learning.

2. Distinguish supervised, unsupervised, semi-supervised, and reinforcement learning.

3. Build and interpret baseline regression and classification models.

4. Describe clustering and dimensionality-reduction methods.

5. Explain training, validation, testing, overfitting, underfitting, regularization, and the bias–variance trade-off.

6. Select suitable evaluation metrics for different analytical problems.

7. Recognize practical, ethical, organizational, and technical risks in ML projects.

8. Design a defensible workflow for developing and deploying a machine-learning solution.

From Data to a Learning Problem

A machine-learning problem begins with a representation of observations. The input variables are commonly called features, predictors, or independent variables and are denoted by X. The output to be predicted is called the target, response, label, or dependent variable and is denoted by y. A model estimates a function f such that:

ŷ = f(X)

where ŷ is the model’s prediction. The model is fitted using a training dataset, but its value is determined by how well it performs on new data. This distinction is essential: memorizing historical records is not the same as learning a relationship that generalizes.

A typical workflow includes problem definition, data collection, data quality assessment, exploratory analysis, preprocessing, splitting the data, model training, validation, evaluation, interpretation, deployment, and monitoring. These stages are iterative rather than strictly linear. For example, an unexpected validation result may reveal leakage, poor feature construction, a wrong target definition, or a mismatch between the model and the business objective.

Several terms are used throughout ML. A parameter is learned from data, such as a regression coefficient. A hyperparameter is selected before training, such as the depth of a decision tree or the strength of regularization. An observation is one record, while a sample is a collection of observations used for learning or evaluation. The loss function measures the discrepancy between predictions and actual outcomes during training. The evaluation metric measures performance for a particular purpose.

The Main Learning Paradigms

Supervised learning

In supervised learning, every training example contains both inputs and a known outcome. The algorithm learns from pairs (X, y). Regression is used when the target is numerical, such as electricity demand or house price. Classification is used when the target represents categories, such as fraudulent versus legitimate or diseased versus non-diseased.

Supervised learning is appropriate when historical outcomes are available and the future task resembles the process that generated the training data. It is not sufficient merely to have a label; the label must be reliable, relevant, and available at the time a prediction is required.

Unsupervised learning

Unsupervised learning uses input data without a predefined target. The objective may be to discover groups, identify unusual observations, compress variables, or reveal latent structure. Clustering and dimensionality reduction are common examples. Because there is no universally correct answer, unsupervised results require domain interpretation and stability analysis.

Other paradigms

Semi-supervised learning combines a small labelled dataset with a larger unlabelled dataset. It can be useful when labelling requires expert time or expense. Reinforcement learning differs from both supervised and unsupervised learning: an agent takes actions in an environment and learns from rewards or penalties over time. Although reinforcement learning is important in robotics and sequential decision-making, this chapter concentrates on the foundational methods most commonly encountered in general data science.

A Practical Modelling Workflow

The first decision is not algorithmic. It is to define the unit of analysis, the prediction target, the prediction time, and the consequences of an error. Predicting customer churn six months in advance is different from explaining why customers left last month. A clearly defined target prevents inappropriate features and misleading evaluation.

Data should then be inspected for missing values, inconsistent categories, duplicate records, outliers, impossible measurements, and sampling bias. Numerical features may require scaling, while categorical features may require encoding. A common approach is one-hot encoding, in which a categorical variable is represented by binary indicator columns. Imputation should be performed using information from the training data and then applied to validation or test data; otherwise, information can leak from the evaluation set into the model.

The dataset is usually divided into training and test portions. The training set is used to estimate parameters, and the test set is reserved for final assessment. A validation set, or cross-validation within the training data, supports model selection and hyperparameter tuning. The test set should be used sparingly, because repeated inspection effectively turns it into another training resource.

A baseline is a simple reference model. For regression, predicting the training-set mean or a domain-based constant can be a baseline. For classification, the majority class or a simple logistic regression may be appropriate. A sophisticated model is useful only if it improves upon a meaningful baseline under an evaluation procedure that reflects the real application.

Baseline Regression

Regression predicts a continuous numerical value. In ordinary least squares linear regression, the predicted value is expressed as:

ŷ = β₀ + β₁x₁ + β₂x₂ + ... + βₚxₚ

The coefficients are generally estimated by minimizing the sum of squared residuals, where a residual is the difference between an observed and predicted value. A coefficient describes the expected change in the prediction associated with a one-unit change in a feature, holding the other included features constant. This interpretation can be misleading when predictors are strongly correlated, when important variables are omitted, or when the relationship is nonlinear.

Important regression metrics include mean absolute error (MAE), mean squared error (MSE), root mean squared error (RMSE), and the coefficient of determination, R². MAE is the average absolute prediction error and is relatively easy to interpret in the target’s units. RMSE gives greater influence to large errors. R² describes the proportion of variation explained relative to a mean-prediction baseline, but it should not be interpreted as proof of causation or as a complete measure of practical usefulness.

A regression model should be evaluated using residual analysis. Patterns in residuals may indicate nonlinearity, heteroscedasticity, omitted variables, or data problems. For example, errors that increase with the size of the prediction suggest that a percentage-based metric or a transformation may be more informative than an unscaled error measure.

Baseline Classification

Classification predicts a discrete class. Logistic regression is an important baseline for binary classification. It models the log-odds of a class as a linear combination of predictors and converts the result into a probability. A threshold is then applied to produce a class decision. The commonly used threshold of 0.5 is not automatically optimal; it should reflect the costs of false positives and false negatives.

A confusion matrix records true positives, true negatives, false positives, and false negatives. From it, several metrics can be calculated. Accuracy is the proportion of all predictions that are correct. Precision is the proportion of predicted positives that are actually positive, while recall or sensitivity is the proportion of actual positives that are detected. Specificity is the proportion of actual negatives correctly identified. The F1 score is the harmonic mean of precision and recall.

Accuracy can be deceptive in imbalanced data. If only one percent of transactions are fraudulent, a model that labels every transaction as legitimate achieves 99 percent accuracy but detects no fraud. Precision–recall analysis is often useful when the positive class is rare. Receiver operating characteristic (ROC) analysis examines the trade-off between true-positive and false-positive rates across thresholds, whereas a precision–recall curve focuses more directly on positive-class retrieval.

Classification performance must be evaluated in relation to the decision. In medical screening, missing a positive case may be more harmful than sending a healthy person for further testing. In a marketing campaign, excessive false positives may waste resources. A model’s threshold, not only its algorithm, is therefore part of the decision system.

Clustering: Finding Structure Without Labels

Clustering groups observations according to a measure of similarity. In k-means clustering, the analyst specifies k clusters. The algorithm alternates between assigning each observation to the nearest centroid and updating each centroid to the mean of the assigned observations. It attempts to minimize within-cluster squared distances.

K-means is computationally useful but has important assumptions. It tends to work best for compact, reasonably separated groups and is sensitive to feature scale, outliers, initialization, and the chosen value of k. Standardizing numerical variables is often necessary when variables use different units. The resulting clusters are not automatically meaningful customer segments; they must be profiled, visualized, and assessed for stability and actionability.

Hierarchical clustering creates a nested sequence of groups and can be displayed as a dendrogram. It is useful for examining structure at different levels, although computational cost may become substantial for large datasets. Density-based methods such as DBSCAN identify dense regions and can mark isolated observations as noise. They may be valuable when clusters have irregular shapes, but their performance depends on distance and density parameters.

Cluster evaluation is more difficult than supervised evaluation. The silhouette coefficient compares how close an observation is to its own cluster with how close it is to neighbouring clusters. Internal metrics can indicate compactness and separation, but external validation—such as whether the segments support different interventions—is often more important.

Dimensionality Reduction

High-dimensional data can contain redundant variables, noise, and difficult-to-visualize structure. Dimensionality reduction transforms a large set of features into a smaller representation while attempting to preserve relevant information.

Principal component analysis (PCA) identifies orthogonal directions, called principal components, that successively capture maximum variance. Before PCA, features measured on different scales are commonly standardized. The component loadings indicate how original variables contribute to each component, while the explained-variance ratio indicates the amount of total variance represented. PCA can reduce computational burden, assist visualization, and address correlated predictors, but components may be difficult to interpret and maximum variance is not necessarily maximum predictive value.

Nonlinear methods such as t-distributed stochastic neighbour embedding (t-SNE) and uniform manifold approximation and projection (UMAP) are often used for visualization. They can reveal local neighbourhood structure, but their plots depend on parameter settings and should not be treated as definitive evidence of clusters. A visually separated two-dimensional projection does not prove that the original data contains operationally distinct groups.

Generalization, Overfitting, and Model Complexity

A model generalizes when it performs well on unseen cases. Overfitting occurs when a model learns random noise, peculiarities, or leakage in the training data rather than a stable relationship. It usually produces very low training error but substantially higher validation or test error. Underfitting occurs when the model is too simple or insufficiently trained to capture relevant structure, leading to poor performance on both training and unseen data.

The bias–variance framework explains one source of this tension. Bias is systematic error arising from restrictive assumptions; variance is sensitivity to the particular sample used for training. Simple models may have high bias and low variance. Highly flexible models may have low training bias but high variance. The aim is not to minimize training error at any cost, but to achieve the best expected performance on new data.

Cross-validation provides a more reliable estimate of generalization than a single arbitrary validation split, especially when data is limited. In k-fold cross-validation, the training data is divided into k parts; each part serves as validation data once while the remaining parts are used for fitting. Time-dependent data requires chronological splitting, because random splitting can allow future information to influence the past.

Regularization and Model Selection

Regularization controls complexity by adding a penalty to the training objective. Ridge regression uses an L2 penalty that shrinks coefficients toward zero. Lasso regression uses an L1 penalty and can shrink some coefficients exactly to zero, producing a form of feature selection. Elastic net combines L1 and L2 penalties and can be useful when predictors are numerous and correlated.

The regularization strength is a hyperparameter. It should be selected through validation or cross-validation, not by examining the test set. Regularization does not automatically eliminate bias, resolve poor data quality, or make a causal claim valid. It is one component of a broader strategy that may also include better features, more data, simpler representations, early stopping, pruning, or ensemble methods.

A fair comparison of models requires a consistent preprocessing and evaluation pipeline. For example, scaling, imputation, feature selection, and PCA must be fitted within each training fold during cross-validation. Performing these operations on the full dataset before cross-validation can produce optimistic estimates.

An Illustrative Applied Case: Predicting Customer Churn

Background

Consider a fictional subscription-based software company, Northstar Systems, serving small and medium-sized businesses. The company wants to identify accounts at risk of cancelling so that customer-success teams can provide appropriate support.

Situation and problem

The historical data contains account age, product usage, number of support requests, payment history, contract type, and a binary churn indicator. Churn is relatively uncommon, and the company has limited staff. It therefore needs a ranked list of accounts and a threshold that balances missed churners against unnecessary interventions.

Analysis and approach

The data science team first defines churn as cancellation within the next 60 days and ensures that every feature was available before the prediction date. Records are divided chronologically rather than randomly to imitate future use. Missing numerical values are imputed within the training pipeline, categorical variables are encoded, and usage variables are standardized for logistic regression.

A majority-class classifier is established as a baseline. Logistic regression is then fitted with regularization, and a tree-based model is evaluated as a more flexible alternative. Because the positive class is uncommon, accuracy is not used as the primary metric. The team examines recall, precision, the precision–recall curve, and the expected cost of interventions. Cross-validation on earlier time periods supports model selection, while a later holdout period is reserved for final assessment.

The analysis reveals that a model with the highest recall would generate too many false alarms for the available staff. The chosen operating threshold therefore produces a smaller, higher-precision list. The team also reviews feature influence, checks whether performance differs across customer sizes and regions, and prevents support notes written after the prediction date from entering the model.

Outcome

The model is deployed as a decision-support tool rather than an automatic cancellation-prevention system. Customer-success employees receive risk scores together with permitted explanatory indicators, such as declining usage or repeated unresolved support issues. Performance is monitored monthly for data drift, calibration, subgroup differences, and changes in the cost of intervention.

Learning points

The case demonstrates that model selection is inseparable from operational capacity and error costs. It also illustrates the importance of temporal validation, leakage prevention, appropriate metrics, human review, and monitoring after deployment. A technically accurate model can still fail if its predictions arrive too late, cannot be acted upon, or are interpreted as certain outcomes rather than risk estimates.

Challenges, Risks, and Professional Responsibilities

Data leakage is among the most damaging errors in ML. It occurs when information unavailable at prediction time is used during training. Leakage can arise from future transactions, target-derived variables, duplicated individuals across splits, or preprocessing performed before data partitioning.

Sampling bias occurs when the training data does not represent the population in which the model will operate. Label bias occurs when historical outcomes reflect inconsistent or discriminatory decisions. A model trained on such outcomes may reproduce them while appearing statistically successful. Fairness should therefore be examined across relevant groups using suitable measures, while recognizing that fairness definitions can conflict and must be connected to the application context.

Privacy and security also require attention. Personal data should be collected and retained for legitimate purposes, protected through access controls, and processed in accordance with applicable law and organizational policy. Models can leak sensitive information, be manipulated through adversarial inputs, or expose personal records through inappropriate explanations. Documentation should describe the data source, target definition, exclusions, known limitations, evaluation design, and intended use.

Interpretability is especially important in high-impact settings. A transparent baseline may be preferable to a marginally more accurate opaque model if decisions require explanation or contestability. Human oversight should be meaningful rather than merely symbolic: reviewers need authority, information, training, and a mechanism for correcting errors.

Emerging Directions and Best Practice

Current practice increasingly emphasizes reproducible ML pipelines, automated experiment tracking, model registries, containerized deployment, and continuous monitoring. MLOps integrates software engineering and data engineering principles with model development. Monitoring should cover not only prediction accuracy, but also input drift, missingness, calibration, latency, fairness, and business outcomes.

Foundation models and generative AI have expanded the range of machine-learning applications, particularly in language, vision, and multimodal analysis. Nevertheless, the foundational principles remain: define the task carefully, evaluate on representative data, control leakage, understand uncertainty, and govern use responsibly. Automated tools can accelerate modelling but cannot decide whether a target is ethically appropriate or whether a prediction should determine a person’s treatment.

Good practice includes establishing a strong baseline, using a validation design that mirrors deployment, selecting metrics linked to consequences, tuning only on training data, documenting assumptions, testing subgroup performance, and monitoring after release. Model performance is a property of a model-data-system combination, not of an algorithm in isolation.

Chapter Summary

Machine learning estimates useful patterns from data and applies them to unseen cases. Supervised learning uses known targets and includes regression for numerical outcomes and classification for categorical outcomes. Unsupervised learning searches for structure through methods such as clustering and dimensionality reduction. Linear and logistic regression provide interpretable baselines, while clustering and PCA support exploration and representation.

Reliable ML depends on sound problem definition, careful preprocessing, appropriate data splitting, meaningful baselines, and metrics aligned with decisions. Overfitting and underfitting reflect the challenge of generalization, while the bias–variance trade-off explains the relationship between model simplicity and flexibility. Regularization, cross-validation, and disciplined pipelines help control complexity. Ethical practice requires attention to leakage, privacy, fairness, interpretability, security, documentation, and post-deployment monitoring.

Key Terms

Feature: An input variable used by a machine-learning model.

Target: The outcome that a supervised-learning model attempts to predict.

Parameter: A quantity estimated by the learning algorithm from data.

Hyperparameter: A modelling choice specified outside the fitting procedure.

Supervised learning: Learning from examples that include known outcomes.

Unsupervised learning: Learning patterns from data without a specified target.

Regression: Prediction of a numerical or continuous outcome.

Classification: Prediction of a discrete category or class.

Clustering: Grouping observations according to similarity.

Dimensionality reduction: Transformation of data into fewer variables or dimensions.

Loss function: A quantity minimized during model training to represent prediction error.

Baseline model: A simple reference model used for comparison.

Overfitting: Learning noise or sample-specific detail that harms performance on new data.

Underfitting: Excessive simplicity that fails to capture important structure.

Bias: Systematic error caused by restrictive assumptions or an inadequate representation.

Variance: Sensitivity of a model’s fitted result to the training sample.

Regularization: A complexity-control method that penalizes certain model parameters.

Data leakage: Use of information during training that would not be available at prediction time.

Precision: The proportion of predicted positives that are truly positive.

Recall: The proportion of actual positives that are correctly identified.

Review Questions

Short Answer Questions

1. What distinguishes supervised learning from unsupervised learning?

2. Why is a baseline model important in a machine-learning project?

3. Define overfitting and identify one sign of its occurrence.

4. What is the purpose of regularization?

5. Why can accuracy be misleading for an imbalanced classification problem?

Long Answer Questions

1. Explain a complete machine-learning workflow from problem definition through deployment and monitoring.

2. Compare linear regression and logistic regression as baseline models, including their outputs and suitable evaluation measures.

3. Explain k-means clustering, its assumptions, and methods for evaluating whether its results are useful.

4. Discuss the bias–variance trade-off and the role of cross-validation in model selection.

5. Explain how data leakage, sampling bias, and privacy concerns can affect the validity and responsible use of an ML system.

Analytical / Application Questions

1. A fraud model achieves 99.5 percent accuracy but detects only 10 percent of fraudulent transactions. Diagnose the problem and recommend more suitable evaluation and threshold-selection methods.

2. A company wants to segment customers using annual spending, number of visits, and average session duration. Design a clustering workflow and explain how you would assess whether the segments are actionable.

3. A churn model performs extremely well on randomly split data but poorly when tested on the next quarter. Explain likely causes and propose a revised validation strategy.

22. Multiple Choice Questions with Answers

Question 1

Which situation is most appropriate for supervised learning?

A. Grouping customers when no outcome variable is available

B. Predicting loan default using historical records labelled default or non-default

C. Compressing correlated measurements into two dimensions

D. Detecting unusual observations without any target variable

Correct Answer: B. Predicting loan default using historical records labelled default or non-default

Question 2

Why is a majority-class classifier a useful classification baseline?

A. It always provides the best predictive performance

B. It establishes the performance obtained by ignoring minority-class structure

C. It eliminates class imbalance

D. It automatically calibrates predicted probabilities

Correct Answer: B. It establishes the performance obtained by ignoring minority-class structure

Question 3

Which metric is especially informative when the positive class is rare and the cost of false positives matters?

A. Precision

B. Training accuracy only

C. R-squared

D. Mean squared error

Correct Answer: A. Precision

Question 4

A model has very low training error but substantially higher validation error. What is the most likely explanation?

A. Underfitting

B. Overfitting

C. Perfect calibration

D. Random sampling without replacement

Correct Answer: B. Overfitting

Question 5

What is the main purpose of L1 regularization in a regression model?

A. To force all coefficients to have equal values

B. To increase model complexity without constraint

C. To shrink coefficients and potentially set some exactly to zero

D. To convert a continuous target into a class label

Correct Answer: C. To shrink coefficients and potentially set some exactly to zero

Question 6

Which practice creates data leakage?

A. Fitting an imputer within each training fold

B. Using a chronological split for time-dependent data

C. Including a variable recorded after the prediction event

D. Evaluating the final model on an untouched test set

Correct Answer: C. Including a variable recorded after the prediction event

Question 7

A principal component in PCA is best described as:

A. A class label predicted by logistic regression

B. A direction in feature space capturing variance in the transformed data

C. A cluster centre selected by k-means

D. A penalty applied to a regression coefficient

Correct Answer: B. A direction in feature space capturing variance in the transformed data

Question 8

Why should features commonly be standardized before k-means clustering?

A. To make categorical labels continuous

B. To prevent variables with large numerical scales from dominating distance calculations

C. To guarantee the correct number of clusters

D. To remove the need for validation

Correct Answer: B. To prevent variables with large numerical scales from dominating distance calculations

Question 9

In the churn case, why was chronological splitting preferred to random splitting?

A. It guarantees that churn will not occur in the test set

B. It better represents the use of past data to predict future outcomes

C. It always produces higher accuracy

D. It removes all missing values

Correct Answer: B. It better represents the use of past data to predict future outcomes

Question 10

Which statement best reflects the bias–variance trade-off?

A. Increasing model flexibility can reduce bias but increase sensitivity to the training sample

B. Bias and variance are identical measures of classification accuracy

C. A model with zero training error must generalize best

D. Regularization always reduces both bias and variance simultaneously

Correct Answer: A. Increasing model flexibility can reduce bias but increase sensitivity to the training sample