Classical Machine Learning and Baseline Models

Introduction and Context

Classical machine learning (ML) refers to methods that learn patterns from data using statistical, geometric, or rule-based principles rather than relying primarily on deep neural networks. These methods include linear and logistic regression, support vector machines, decision trees, random forests, gradient-boosting models, clustering algorithms, dimensionality-reduction methods, and probabilistic models.

Although deep learning has achieved remarkable results in computer vision, speech, and natural-language processing, classical ML remains essential. Many business datasets are tabular, contain relatively few observations, or require transparent explanations. In such settings, a well-engineered classical model can be more accurate, faster, cheaper, and easier to govern than a deep neural network.

A central professional practice is to establish a baseline model before developing a complex system. A baseline provides a reference point against which feature engineering, model changes, and deep-learning approaches can be evaluated. Without a baseline, improvements may be claimed merely because of changes in data splitting, preprocessing, or evaluation.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish regression, classification, clustering, and dimensionality-reduction tasks.

2. Explain the assumptions and practical uses of major classical ML models.

3. Establish meaningful predictive and non-predictive baselines.

4. Compare tree, ensemble, linear, kernel, and probabilistic models.

5. Interpret common evaluation metrics and recognize data leakage.

6. Select classical ML methods appropriately for a given application.

7. Implement a reproducible classical-ML workflow using tools such as scikit-learn, TensorFlow, or PyTorch-supported preprocessing pipelines.

8. Evaluate the ethical, operational, and governance implications of model choice.

1. The Classical Machine-Learning Workflow

A machine-learning project begins with a precise definition of the prediction or discovery problem. In supervised learning, a dataset contains input variables, represented by a feature matrix X, and a target variable y. The objective is to learn a function that performs well on previously unseen examples. In unsupervised learning, target labels are absent and the objective is to identify structure, such as groups or lower-dimensional representations.

A sound workflow normally includes:

• Problem formulation and definition of the unit of analysis.
• Data collection, quality assessment, and labeling.
• Training, validation, and test-set design.
• Preprocessing and feature construction.
• Baseline establishment.
• Model training and hyperparameter selection.
• Evaluation using metrics appropriate to the decision.
• Error analysis, documentation, deployment, and monitoring.

The test set must not influence model selection or preprocessing decisions. When data are limited, cross-validation is commonly used. In k-fold cross-validation, the training data are divided into k approximately equal folds. Each fold is used once for validation while the remaining folds are used for training. The reported score is usually the mean of the k validation scores, often accompanied by their variability.

For time-dependent data, random splitting can produce an unrealistic evaluation. A temporal split trains on earlier observations and evaluates on later observations. For grouped data, such as multiple records from the same patient, all records from a group should generally remain in the same split to prevent information leakage.

2. Baselines and Generalization

A baseline is a simple reference system. For regression, a common baseline predicts the mean of the training targets. For classification, a baseline may always predict the majority class or use class frequencies. A baseline can also be operational: for example, a current human process, a previous production model, or a simple business rule.

Suppose the training targets are 10, 12, 14, and 16. The mean baseline is

ŷ = (10 + 12 + 14 + 16) / 4 = 13.

For squared-error evaluation, the mean is the constant prediction that minimizes the sum of squared errors. This does not make it a good production model; it makes it a useful reference.

For a regression model with true values y_i and predictions ŷ_i, the mean squared error is

MSE = (1/n) Σᵢ(yᵢ − ŷᵢ)²,

and the root mean squared error is RMSE = √MSE. MSE has squared target units, whereas RMSE has the same units as the target. Mean absolute error is

MAE = (1/n) Σᵢ |yᵢ − ŷᵢ|.

MAE is often easier to interpret and is less sensitive to large errors than MSE. R², the coefficient of determination, is

R² = 1 − [Σᵢ(yᵢ − ŷᵢ)² / Σᵢ(yᵢ − ȳ)²],

where ȳ is the mean target in the evaluation set. R² measures improvement over a constant mean predictor on that evaluation set. It can be negative when a model performs worse than that baseline, and it should not be interpreted automatically as the percentage of variance explained in every practical context.

For binary classification, accuracy is the fraction of correct predictions. Precision is TP/(TP+FP), and recall is TP/(TP+FN), where TP, FP, and FN denote true positives, false positives, and false negatives. The F1 score is the harmonic mean of precision and recall:

F1 = 2(precision × recall)/(precision + recall),

provided the denominator is nonzero. Accuracy is unsuitable as the sole metric when classes are highly imbalanced. Receiver operating characteristic (ROC) AUC measures ranking performance across classification thresholds, while precision-recall curves can be more informative when the positive class is rare.

A model may produce a probability rather than a hard label. The decision threshold should reflect the costs of false positives and false negatives. Threshold selection belongs to validation and should not be tuned on the test set.

3. Regression Models

3.1 Linear Regression

Linear regression predicts a continuous target using a weighted combination of features:

ŷ = β₀ + β₁x₁ + ... + β_px_p.

The ordinary least-squares estimate chooses coefficients that minimize the sum of squared residuals. Important assumptions for classical statistical inference include linearity of the conditional mean, independent errors, and, for conventional standard-error calculations, suitable assumptions about error variance. Accurate prediction does not require every assumption to hold perfectly, but violations can affect interpretation and uncertainty estimates.

Coefficients are conditional associations within the specified model. A coefficient should not automatically be interpreted as a causal effect. Correlated features can make coefficients unstable or difficult to interpret. Standardization is not necessary for ordinary least squares, but it is useful when comparing coefficient magnitudes or combining linear regression with regularization.

3.2 Regularization

Ridge regression minimizes squared error plus an L2 penalty on coefficients:

Σᵢ(yᵢ − ŷᵢ)² + λΣⱼβⱼ²,

where λ ≥ 0 controls the penalty. The intercept is generally excluded from the penalty. Ridge shrinks correlated coefficients and can reduce variance.

Lasso regression uses an L1 penalty:

Σᵢ(yᵢ − ŷᵢ)² + λΣⱼ|βⱼ|.

The L1 penalty can produce exactly zero coefficients, which may support sparse feature selection. However, when predictors are strongly correlated, lasso may select one predictor and discard another somewhat arbitrarily. Elastic net combines L1 and L2 penalties and is often useful with correlated, high-dimensional features.

The regularization parameter must be selected using training data and validation or cross-validation. Features should generally be scaled before regularized linear modeling because the penalty otherwise depends on measurement units.

3.3 Generalized Linear Classification: Logistic Regression

Logistic regression models the probability of a binary outcome. It uses the logistic function

p(y=1|x) = 1/[1 + exp(−(β₀ + βᵀx))].

The log-odds, or logit, are linear in the features:

log[p/(1−p)] = β₀ + βᵀx.

A coefficient β_j represents the change in log-odds associated with a one-unit increase in x_j while other included features are held constant. exp(β_j) is the corresponding odds ratio under the model. Logistic regression is often effective for sparse text features, credit-risk screening, and regulated applications because it is computationally efficient and relatively interpretable.

Its linear decision boundary can be limiting when relationships are strongly nonlinear. Polynomial features, interactions, splines, or nonlinear models can address this limitation, but they increase complexity and may require careful validation.

4. Classification with Geometric and Rule-Based Models

4.1 k-Nearest Neighbors

The k-nearest-neighbors (k-NN) classifier assigns a new observation the majority class among its k closest training observations. Distance-based methods require meaningful feature scaling; otherwise, a variable with a large numerical range can dominate the distance. Small k values produce flexible, high-variance decision boundaries, while larger k values smooth the boundary and may increase bias.

k-NN has low training cost but can have substantial prediction cost because distances to many training observations must be computed. It is less attractive in high-dimensional spaces, where distances become less discriminative, a phenomenon associated with the curse of dimensionality.

4.2 Support Vector Machines

A linear support vector classifier seeks a separating hyperplane with a large margin. Only observations near the boundary, called support vectors, directly determine the fitted boundary. A soft-margin formulation permits violations and balances margin width against classification errors through a regularization parameter C.

Kernel methods implicitly represent data in a transformed feature space. The radial basis function kernel is commonly written as

K(x, z) = exp(−γ||x − z||²),

where γ controls how rapidly similarity decreases with distance. Kernel SVMs can model nonlinear boundaries but may become computationally expensive for very large datasets. Feature scaling is important because distances influence the kernel.

4.3 Decision Trees

A decision tree recursively partitions the feature space using rules such as x_j < t. For classification, a node impurity measure may be Gini impurity:

Gini = 1 − Σ_c p_c²,

where p_c is the proportion of observations in the node belonging to class c. A pure node has Gini impurity zero. Entropy, defined as −Σ_c p_c log₂(p_c), is another common criterion.

Trees capture nonlinear relationships and interactions without requiring feature scaling. They are easy to visualize and can accommodate mixed feature types after suitable encoding. However, unrestricted trees can memorize training data. Maximum depth, minimum samples per leaf, pruning, and other constraints control overfitting. Individual tree feature importance measures can be misleading when features are correlated; permutation importance or partial-dependence analysis should be interpreted carefully.

5. Ensemble Models

Ensembles combine multiple models to improve predictive performance or stability.

5.1 Bagging and Random Forests

Bagging trains models on bootstrap samples and aggregates their predictions. A random forest adds random feature selection at each split. These sources of randomness reduce correlation among trees, and averaging can reduce variance. Random forests are strong general-purpose baselines for tabular data, require limited preprocessing, and can estimate class probabilities, although those probabilities may require calibration.

5.2 Boosting

Boosting builds models sequentially, with later models emphasizing errors made by earlier models. Gradient boosting fits new learners to an approximation of the negative gradient of a chosen loss function. In practice, learning rate, number of estimators, tree depth, subsampling, and regularization are important controls.

Modern gradient-boosting implementations, including XGBoost, LightGBM, and CatBoost, are frequently excellent for structured tabular data. Their advantages include nonlinear modeling, interaction detection, and strong accuracy with moderate data volumes. Their disadvantages include sensitivity to hyperparameters, greater interpretive complexity than a single tree, and potential overfitting or leakage if validation is careless.

Stacking combines predictions from several model types using a meta-model. The meta-model must be trained on out-of-fold predictions rather than predictions generated from the same data used to fit the base models; otherwise, optimistic leakage can occur.

6. Clustering and Dimensionality Reduction

6.1 k-Means Clustering

k-means partitions observations into k clusters by minimizing within-cluster squared Euclidean distance:

Σᵢ ||xᵢ − μ_{c(i)}||²,

where μ_{c(i)} is the centroid assigned to observation i. The standard algorithm alternates between assigning each observation to its nearest centroid and recomputing centroids. It converges to a local optimum, so multiple initializations are advisable.

k-means works best when clusters are reasonably compact, separated, and comparable in scale. The number k must be selected or justified; a silhouette score, domain knowledge, stability analysis, or downstream usefulness may help. Clustering is exploratory: a cluster is not automatically a natural social category or a causal group.

Hierarchical clustering creates a nested sequence of partitions and can be visualized with a dendrogram. Density-based methods such as DBSCAN can discover irregularly shaped clusters and identify noise, but their results depend on density parameters and may struggle when cluster densities vary substantially.

6.2 Principal Component Analysis

Principal component analysis (PCA) transforms correlated variables into orthogonal components. After centering features, PCA identifies directions of maximum variance. If X is a centered data matrix, the first principal direction is a unit vector w that maximizes the variance of Xw. Subsequent directions are constrained to be orthogonal to earlier directions.

PCA can reduce dimensionality, compress data, remove some noise, and support visualization. It is sensitive to feature scale, so standardization is often appropriate when variables have different units. PCA components are linear combinations and may not have straightforward meanings. PCA must be fitted only on training data when used in a predictive pipeline; fitting it before the train-test split leaks information from the test distribution.

7. Probabilistic Models

Probabilistic models represent uncertainty explicitly. Naive Bayes applies Bayes’ rule while assuming conditional independence of features given the class:

P(y|x) ∝ P(y)P(x|y).

For a multinomial text model, word counts and class-specific frequencies can be used to estimate likelihoods, commonly with smoothing to avoid zero probabilities. The independence assumption is often unrealistic, yet the classifier can perform well, particularly for high-dimensional text classification.

Gaussian mixture models (GMMs) represent data as a weighted mixture of Gaussian distributions. Unlike k-means, a GMM provides soft membership probabilities and can model elliptical clusters. The expectation-maximization algorithm alternates between estimating membership probabilities and updating mixture parameters. Model selection may use likelihood-based criteria such as AIC or BIC, but practical usefulness and stability should also be examined.

Calibration is important when predicted probabilities guide decisions. A model can rank cases well while producing probabilities that are systematically too high or too low. Reliability diagrams, calibration curves, and proper scoring rules such as log loss can assess probabilistic quality.

8. Worked Comparison: A Credit-Default Screening Task

Consider an illustrative dataset of 20,000 loan applications. Features include income, debt-to-income ratio, employment length, loan amount, past repayment indicators, and a categorical purpose variable. The target is whether a borrower defaults within twelve months. Defaults represent 8% of applications.

A majority-class baseline predicts no default for every applicant. Its accuracy is 92%, but its recall for defaults is 0%, making it operationally weak. A stratified split preserves the approximate default rate across training, validation, and test sets. Categorical variables are encoded, numerical variables are imputed and scaled where required, and all transformations are fitted only on the training data.

Three candidate models are trained:

1. Logistic regression with L2 regularization.

2. Random forest with constrained tree depth.

3. Gradient-boosted trees.

Suppose validation results are as follows:

• Logistic regression: ROC AUC 0.78, recall 0.64 at the selected threshold, and relatively good calibration.
• Random forest: ROC AUC 0.81, recall 0.68, but less reliable probabilities.
• Gradient boosting: ROC AUC 0.84, recall 0.72, with improved performance after calibration.

These values are illustrative rather than claims about a universal ranking. The gradient-boosting model appears strongest by discrimination, but deployment should not be decided by AUC alone. A lender must specify the relative costs of missed defaults, unnecessary reviews, and potential adverse impact on applicants. Thresholds may be selected to meet a minimum recall or a review-capacity constraint.

The logistic model may remain attractive if explanations, stability, and regulatory review are priorities. The boosted model can be examined using feature-attribution tools, monotonic constraints where appropriate, subgroup evaluation, and calibration analysis. All models should be evaluated across relevant demographic and socioeconomic groups, subject to applicable law and privacy requirements. A model that improves average accuracy but produces systematically higher error rates for a protected group may be unacceptable.

The case illustrates a general principle: model selection is a multi-objective decision involving predictive performance, calibration, fairness, interpretability, latency, maintenance, and governance.

9. Classical ML Compared with Deep Learning

Classical ML is often preferable when:

• Data are tabular and sample sizes are moderate.
• Features have meaningful domain interpretations.
• Training and inference resources are constrained.
• Rapid experimentation and interpretability are important.
• The organization requires stable, auditable behavior.
• The task does not benefit substantially from learned representations.

Deep learning is often advantageous for unstructured data such as images, audio, and natural language, especially when large labeled datasets and computational resources are available. It can also model complex representations directly, reducing dependence on manual feature engineering. However, deep networks can be costly to train, difficult to calibrate, harder to explain, and sensitive to distribution shifts.

The comparison is not absolute. Classical models can use embeddings generated by neural networks, and neural networks can be effective on tabular data in some settings. A fair comparison requires the same data partitions, leakage controls, tuning budget, and evaluation metrics.

10. Implementation Practices with Modern ML Tools

In Python, scikit-learn provides estimators, preprocessing transformers, pipelines, cross-validation, and evaluation tools. A professional pipeline combines preprocessing and prediction so that training and inference apply identical operations. A conceptual workflow is:

1. Split data according to the deployment setting.

2. Define numerical imputation, scaling, and categorical encoding steps.

3. Combine transformations in a pipeline.

4. Fit candidate estimators on training data.

5. Tune hyperparameters using cross-validation within the training set.

6. Evaluate once on the untouched test set.

7. Save the complete pipeline, metadata, feature definitions, and software versions.

TensorFlow and PyTorch are primarily deep-learning frameworks, but they can coexist with classical ML. For example, neural embeddings can be exported as features for a logistic-regression or gradient-boosting model. Conversely, classical preprocessing and baseline metrics are useful when assessing a neural model.

Reproducibility requires fixed or documented random seeds where possible, versioned datasets, clear feature-generation timestamps, deterministic split logic, and experiment tracking. Random seeds do not guarantee complete reproducibility across hardware and software versions, but they improve traceability.

11. Limitations, Ethics, and Governance

Classical models are not automatically fair, interpretable, or safe. Bias can enter through historical labels, incomplete sampling, proxy variables, measurement error, or the use of a target that reflects unequal institutional treatment. Removing a protected attribute does not necessarily remove discrimination because other variables may act as proxies.

Important governance practices include:

• Documenting intended use, prohibited use, training data, and known limitations.
• Measuring performance and calibration by relevant subgroups.
• Assessing privacy, security, and data-retention requirements.
• Maintaining human oversight for high-impact decisions.
• Monitoring drift in inputs, labels, error rates, and subgroup outcomes.
• Providing processes for contesting or correcting consequential decisions.
• Preserving audit trails for model versions and threshold changes.

Unsupervised models introduce additional risks. Clusters can be treated as if they were objective categories even when they reflect arbitrary preprocessing choices. Dimensionality-reduction plots can create visual impressions of separation that are not stable or statistically meaningful. Analysts should test sensitivity to scaling, initialization, features, and sample selection.

12. Best Practices for Model Selection

A reliable selection process follows these principles:

1. Begin with a simple baseline and a transparent benchmark.

2. Match the metric to the operational cost of errors.

3. Use splits that reflect deployment, including time and groups.

4. Put preprocessing inside the validation pipeline.

5. Scale features for distance-based and regularized models when appropriate.

6. Control tree complexity and use validation for ensemble tuning.

7. Inspect errors, not only aggregate scores.

8. Evaluate calibration when probabilities inform decisions.

9. Compare subgroup performance and investigate disparities.

10. Prefer the simplest model that satisfies performance and governance requirements.

11. Test robustness under plausible distribution shifts.

12. Record assumptions, data lineage, hyperparameters, and decisions.

Chapter Summary

Classical machine learning provides a broad toolkit for prediction, classification, clustering, representation, and uncertainty estimation. Linear and logistic regression offer efficient, interpretable baselines; regularization controls variance and supports high-dimensional modeling. k-NN and support vector machines use geometric structure, while decision trees represent nonlinear rules. Random forests reduce variance through bagging, and gradient boosting often delivers excellent performance on structured data. k-means, hierarchical clustering, DBSCAN, and Gaussian mixtures reveal different forms of unsupervised structure, while PCA creates lower-dimensional representations.

Baselines, leakage prevention, deployment-aware data splitting, appropriate metrics, and error analysis are foundational professional practices. Classical models are frequently preferable to deep learning for moderate-sized tabular datasets, constrained environments, and applications requiring transparency and governance. Model selection must consider not only accuracy but also calibration, fairness, interpretability, cost, latency, robustness, and maintainability.

Key Terms

Baseline model: A simple reference system used for comparison.

Regression: Supervised learning for predicting a continuous quantity.

Classification: Supervised learning for assigning observations to categories.

Regularization: A method that penalizes model complexity to reduce overfitting.

Ridge regression: Regression using an L2 coefficient penalty.

Lasso regression: Regression using an L1 coefficient penalty.

Logistic regression: A model that estimates class probabilities through a logistic link.

Decision tree: A model that recursively partitions data using feature-based rules.

Random forest: An ensemble of randomized decision trees aggregated by voting or averaging.

Gradient boosting: Sequential construction of models that improve a loss-based objective.

Support vector machine: A margin-based model that can use kernels for nonlinear boundaries.

k-nearest neighbors: A distance-based method using nearby labeled observations.

Clustering: Unsupervised grouping of observations according to similarity.

k-means: A centroid-based clustering algorithm minimizing within-cluster squared distance.

Principal component analysis: A method that creates orthogonal directions of maximum variance.

Naive Bayes: A probabilistic classifier using conditional-independence assumptions.

Calibration: Agreement between predicted probabilities and observed frequencies.

Data leakage: Use of information unavailable at prediction time or from evaluation data.

Concept drift: Change over time in the relationship between inputs and targets.

Review Questions

Short Answer Questions

1. Why is a baseline necessary before comparing complex machine-learning models?

2. Distinguish precision from recall.

3. Why should features commonly be scaled before using k-NN or an RBF-kernel SVM?

4. What is the main purpose of regularization?

5. Why must PCA be fitted only on training data in a predictive workflow?

Long Answer Questions

1. Explain the assumptions, advantages, and limitations of linear regression, logistic regression, and decision trees.

2. Compare bagging, random forests, and gradient boosting in terms of variance, bias, training strategy, and practical use.

3. Explain how train-validation-test splitting and cross-validation should be designed for ordinary, grouped, and time-dependent data.

4. Compare k-means, hierarchical clustering, DBSCAN, and Gaussian mixture models.

5. Discuss when a classical model should be preferred to a deep neural network, considering data, performance, interpretability, cost, and governance.

Analytical / Application Questions

1. A fraud dataset contains 1% fraudulent transactions. Propose a baseline, evaluation metrics, threshold-selection strategy, and validation design. Justify each choice.

2. A company reports 98% accuracy for a loan-default classifier, but default recall is 4%. Analyze why the result may be misleading and recommend a better evaluation framework.

3. Design a model-comparison experiment for a tabular healthcare dataset using logistic regression, random forest, and gradient boosting. Identify measures needed to prevent leakage and assess fairness, calibration, and deployment readiness.

Multiple Choice Questions with Answers

Question 1

Which model is commonly used as a constant baseline for regression with squared-error loss?

A. The training-target mean

B. The training-target median

C. The largest training target

D. A random target value

Correct Answer: A. The training-target mean

Question 2

What does L1 regularization tend to encourage?

A. Larger coefficients

B. Exactly sparse coefficients

C. Removal of the intercept only

D. Unscaled distance calculations

Correct Answer: B. Exactly sparse coefficients

Question 3

Which metric is generally most directly affected by false negatives?

A. Recall

B. Specificity only

C. Mean squared error

D. R-squared

Correct Answer: A. Recall

Question 4

Why is feature scaling important for k-nearest neighbors?

A. It changes class labels

B. It prevents all overfitting

C. Distance calculations depend on feature magnitudes

D. It converts regression into classification

Correct Answer: C. Distance calculations depend on feature magnitudes

Question 5

What is a principal characteristic of random forests?

A. They use only one unpruned tree

B. They aggregate many randomized decision trees

C. They require linearly separable data

D. They estimate only continuous targets

Correct Answer: B. They aggregate many randomized decision trees

Question 6

What does k-means minimize?

A. Between-class entropy

B. Within-cluster squared distance to centroids

C. Classification cross-entropy

D. The number of input features

Correct Answer: B. Within-cluster squared distance to centroids

Question 7

Which situation is an example of data leakage?

A. Using a validation set for model selection

B. Scaling training data before fitting a model

C. Using information generated after the prediction time

D. Evaluating on an untouched test set

Correct Answer: C. Using information generated after the prediction time

Question 8

What does calibration evaluate?

A. Whether probabilities correspond to observed frequencies

B. Whether features have equal units

C. Whether clusters have identical sizes

D. Whether a tree has maximum depth

Correct Answer: A. Whether probabilities correspond to observed frequencies

Question 9

Which model explicitly uses a conditional-independence assumption?

A. Naive Bayes

B. Random forest

C. k-means

D. Ridge regression

Correct Answer: A. Naive Bayes

Question 10

Why can accuracy be misleading on a dataset with 1% positives?

A. Accuracy cannot be computed for binary labels

B. Predicting the majority class may yield high accuracy while missing positives

C. Accuracy always equals recall

D. Accuracy requires calibrated probabilities

Correct Answer: B. Predicting the majority class may yield high accuracy while missing positives