Additional Machine Learning Models

Introduction

Predictive analytics rarely has a single best algorithm. The appropriate model depends on the structure of the data, the decision problem, the cost of errors, computational resources, interpretability requirements, and the speed at which predictions must be produced. Linear and tree-based models are valuable foundations, but many business problems benefit from additional model families with different assumptions and inductive biases.

This chapter examines four important supervised-learning approaches: k-nearest neighbours (k-NN), naïve Bayes, support vector machines (SVMs), and introductory neural networks. The models are useful for classification and, in modified form, regression. They differ substantially in how they represent similarity, probability, boundaries, and nonlinear relationships. Understanding these differences enables an analyst to select a model rather than merely apply one.

The chapter emphasizes model assumptions, data preparation, evaluation, operational deployment, and responsible use. Examples focus on business analytics applications such as customer churn, spam detection, credit risk, and customer response prediction.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the principles and assumptions of k-NN, naïve Bayes, SVMs, and neural networks.
2. Apply the principal equations used by these models.
3. Identify how scaling, sparsity, dimensionality, and class imbalance affect model performance.
4. Compare the models using predictive, operational, and governance criteria.
5. Select and justify an algorithm for a business problem.
6. Interpret the limitations and ethical implications of model deployment.

1. Model Selection as a Data-and-Decision Problem

Algorithm selection should begin with the prediction task, not with a preference for a particular algorithm. Important questions include:

• Is the target categorical or numerical?
• Are observations naturally represented by distances, conditional probabilities, margins, or learned feature combinations?
• Are the predictors continuous, categorical, text-based, sparse, or mixed?
• Is prediction speed more important than training speed?
• Must an analyst explain individual decisions to customers, regulators, or managers?
• What are the relative costs of false positives and false negatives?

A useful validation workflow is to establish a simple baseline, create a leakage-resistant training and test design, preprocess data within each training fold, tune hyperparameters using cross-validation, and evaluate on an untouched test set. A model should not be selected solely because it has the highest accuracy. In a fraud application, for example, recall, precision, expected financial loss, latency, and investigation capacity may be more important.

2. k-Nearest Neighbours

2.1 Core idea

The k-nearest neighbours method predicts an observation from the labels or values of nearby training observations. It is an instance-based, or lazy, learning method: rather than estimating a compact global equation during training, it stores the training data and performs most computation at prediction time.

For an input vector x, the algorithm calculates a distance to every training observation, identifies the k closest observations, and aggregates their outcomes. For classification, the usual prediction is the majority class. For regression, the prediction is often the mean response of the neighbours.

For numeric features, the Euclidean distance between observations x and z with p features is:

Here, xᵢ and zᵢ are values of feature i, and p is the number of features. The formula is appropriate when features are numeric and a Euclidean geometry is meaningful. Other distances include Manhattan distance, d(x,z) = ∑ᵢ₌₁ᵖ |xᵢ − zᵢ|, and cosine distance, which is often useful for text vectors.

For classification, the unweighted prediction is:

Nₖ(x) is the set of k nearest observations, c is a possible class, yᵢ is the observed class, and 𝟙(·) equals 1 when its condition is true and 0 otherwise. A distance-weighted version gives greater influence to nearby observations, commonly using weights proportional to 1/(dᵢ + ε), where dᵢ is distance and ε is a small positive constant preventing division by zero.

For regression, the basic prediction is:

2.2 Worked example

Suppose a retailer classifies a new customer as likely or unlikely to respond to an offer. After standardization, the three nearest customers have distances 0.20, 0.35, and 0.50, with response labels Yes, No, and Yes. For k = 3, the votes are two Yes and one No; therefore, the predicted class is Yes. A larger k might reduce sensitivity to unusual observations but could blur meaningful local patterns.

2.3 Choosing k and preparing data

A small k creates a flexible model with low bias but potentially high variance. It can be strongly affected by noise. A large k produces smoother predictions, generally reducing variance but increasing bias. The value of k should be selected through validation rather than chosen from a universal rule.

Feature scaling is usually essential. If annual income ranges from 20,000 to 200,000 while the number of website visits ranges from 0 to 100, income will dominate Euclidean distance unless the features are transformed. A common standardization is:

where μᵢ and σᵢ are the training-set mean and standard deviation for feature i. The same training means and standard deviations must be applied to validation, test, and production data. Scaling parameters must never be calculated using the test set.

Categorical variables require careful treatment. One-hot encoding can create many dimensions and may make ordinary Euclidean distance less meaningful. Mixed-type data may require a specialized distance or a carefully designed feature representation.

2.4 Strengths and limitations

k-NN is intuitive, easy to implement, and capable of representing irregular decision boundaries. It can be effective when similar cases genuinely have similar outcomes and the data set is moderate in size.

Its disadvantages include expensive prediction, sensitivity to scaling and irrelevant variables, weak performance in high dimensions, and limited interpretability when many neighbours contribute to a prediction. The curse of dimensionality means that as p increases, observations tend to become relatively far apart, and the distinction between near and far neighbours becomes less informative. Approximate-neighbour search can reduce latency, but it introduces an additional accuracy–speed trade-off.

3. Naïve Bayes

3.1 Probabilistic foundation

Naïve Bayes applies Bayes’ theorem to estimate the probability of a class given observed features:

P(c|x) is the posterior probability of class c given feature vector x; P(x|c) is the likelihood of observing x in class c; P(c) is the prior probability of class c; and P(x) is the evidence. For classification, P(x) is common to all classes, so the selected class can be written as:

The naïve assumption is conditional independence of features given the class:

This assumption is often false—for example, word frequencies in a document are related—but the classifier can still perform well because estimating many simple conditional distributions may be more reliable than estimating one complex joint distribution.

3.2 Variants and smoothing

Gaussian naïve Bayes is commonly used for continuous features and assumes each feature has a class-specific normal distribution. Multinomial naïve Bayes is widely used for non-negative count features such as word counts. Bernoulli naïve Bayes is appropriate when features represent binary events such as whether a word appears.

Zero probabilities can cause the product of likelihoods to become zero. Laplace smoothing addresses this problem for discrete counts. With additive smoothing parameter α, an estimated category probability can be written:

N_{c,j,v} is the count of value v for feature j in class c, N_{c,j} is the relevant total count, V is the number of possible values, and α > 0. The denominator must include the corresponding number of categories so that probabilities sum to one.

3.3 Worked example: a spam classifier

Assume a message classifier compares Spam and Legitimate classes. The prior probabilities are P(Spam) = 0.40 and P(Legitimate) = 0.60. For a message containing the words “free” and “offer,” suppose the estimated likelihoods are:

Under the naïve conditional-independence assumption, the unnormalized scores are:

• Spam: 0.40 × 0.50 × 0.40 = 0.080
• Legitimate: 0.60 × 0.05 × 0.10 = 0.003

Because 0.080 is greater than 0.003, the prediction is Spam. Normalizing the scores gives an estimated Spam probability of 0.080 ÷ (0.080 + 0.003) ≈ 0.964. This probability is model-dependent and should not automatically be interpreted as a perfectly calibrated 96.4% real-world frequency.

In practical text systems, calculations are performed in log space to avoid numerical underflow:

3.4 Strengths and limitations

Naïve Bayes is fast, computationally economical, and effective for high-dimensional sparse data, especially text classification. It can work with relatively small training sets and naturally provides class-probability estimates.

Its independence assumption can produce poorly calibrated probabilities when predictors are strongly correlated. Redundant features can effectively be counted multiple times. The model also depends on suitable distributional assumptions for continuous variables. Nonetheless, its speed and transparency make it a strong baseline for email filtering, document routing, sentiment classification, and rapid triage systems.

4. Support Vector Machines

4.1 Maximum-margin classification

An SVM seeks a decision boundary that separates classes while maximizing the margin between the boundary and the closest training observations. The closest observations are called support vectors because they determine the position of the boundary.

For a linearly separable binary problem, a separating hyperplane is:

w is the vector of feature weights, b is the intercept, and x is the feature vector. The signed prediction is based on the sign of wᵀx + b. For labels yᵢ ∈ {−1, +1}, the hard-margin constraints are:

The margin width is 2/||w||, so maximizing the margin is equivalent to minimizing (1/2)||w||², subject to the constraints. This formulation assumes perfect linear separability and is not suitable when noise or overlapping classes exist.

4.2 Soft margins and the parameter C

Real business data are rarely perfectly separable. Soft-margin SVMs introduce non-negative slack variables ξᵢ and solve:

C > 0 controls the trade-off between a wide margin and training violations. A large C penalizes violations heavily and may produce a more complex boundary. A small C permits more violations in exchange for stronger regularization. The correct value depends on validation performance and operational costs.

4.3 Kernels and nonlinear boundaries

The kernel approach permits an SVM to model nonlinear relationships without explicitly constructing all transformed features. A kernel function computes an inner product in an implicit feature space. Common choices include the polynomial kernel and radial basis function (RBF) kernel:

γ > 0 controls how quickly similarity declines with distance. A large γ makes each observation influential only in a small region, creating a highly flexible boundary; a small γ produces broader influence and a smoother boundary. C and γ must generally be tuned together.

4.4 Strengths, limitations, and implementation

SVMs can perform strongly in medium-sized, high-dimensional data sets and are often effective when a clear margin exists. Linear SVMs are particularly useful for sparse text features. Kernel SVMs can represent complex boundaries.

Limitations include potentially high training cost for very large data sets, sensitivity to feature scaling, difficulty explaining nonlinear decisions, and the need to tune several hyperparameters. Standard SVM outputs are not automatically calibrated probabilities. If probability estimates are required, a separate calibration procedure should be fitted using appropriate validation data.

5. Introductory Neural Networks

5.1 Architecture

A feed-forward neural network consists of input features, one or more hidden layers, and an output layer. Each neuron calculates a weighted sum, adds a bias, and applies an activation function. For neuron j:

xᵢ are inputs, wᵢⱼ are weights, bⱼ is a bias, and g is an activation function. Common hidden-layer activations include the rectified linear unit:

For binary classification, a sigmoid output is common:

It maps a real-valued score to the interval (0, 1). For K mutually exclusive classes, the softmax function produces class probabilities:

The probabilities sum to one. For binary classification, cross-entropy loss is:

where y ∈ {0,1} is the observed label and p is the predicted probability. Training seeks weights that minimize average loss, often with regularization.

5.2 Training by gradient-based optimization

The network’s predictions depend on all weights, so training uses backpropagation to calculate derivatives of the loss with respect to each parameter. An optimization method then updates parameters. A basic gradient-descent update is:

θ represents all weights and biases, η > 0 is the learning rate, and ∇L(θₜ) is the gradient at iteration t. If η is too large, training may become unstable; if too small, learning may be unnecessarily slow.

A network can overfit when it has excessive capacity relative to the information in the training data. Regularization, early stopping, dropout, suitable architecture, and sufficient data can reduce this risk. Neural networks usually require numerical inputs, careful scaling, and systematic tuning of architecture, learning rate, batch size, and training duration.

5.3 Strengths and limitations

Neural networks can learn nonlinear interactions and are highly extensible. They are especially powerful for images, speech, sequences, and large-scale unstructured data. For ordinary tabular business data, however, their advantage over carefully tuned tree ensembles or other models is not guaranteed.

Networks may require substantial data and computing resources, can be sensitive to initialization and hyperparameters, and are difficult to explain in simple causal terms. Their output is predictive rather than proof that a feature causes an outcome. Explainability tools can describe patterns in model behaviour, but they do not remove the need for domain validation.

6. Comparative Model Selection

A practical selection framework is:

1. Define the business decision and error costs.
2. Inspect feature geometry, sparsity, missingness, and sample size.
3. Establish interpretable and computationally simple baselines.
4. Preprocess within cross-validation pipelines.
5. Tune only meaningful hyperparameters.
6. Compare discrimination, calibration, fairness, latency, memory, and maintenance.
7. Test robustness under temporal or population shift.
8. Select the simplest model that meets the operational requirement.

7. Applied Case: Predicting Customer Churn

Background

The following is an illustrative case. A subscription business wants to identify customers likely to cancel within the next 30 days so that retention offers can be targeted. The data include tenure, monthly charge, support contacts, recent usage, contract type, and customer-service text.

Problem

The company must balance two costs. Contacting a customer who would have stayed creates unnecessary incentive cost, while failing to contact a likely churner loses future revenue. The model must therefore support ranking and targeted intervention, not merely produce a class label.

Analysis and approach

The analytics team first defines the outcome using only information available before the prediction date. Records after that date are excluded from features to prevent leakage. Numerical variables are standardized for k-NN, SVM, and the neural network. Missing-value treatment and scaling are fitted within training folds. Text is represented using a sparse term-frequency representation.

Four candidate models are evaluated:

• k-NN, to test whether customers with similar behaviour have similar churn outcomes;
• naïve Bayes, especially for the sparse service-text features;
• a linear SVM, to provide a strong high-dimensional baseline;
• a small feed-forward neural network, to model nonlinear interactions among usage, tenure, and charges.

The evaluation uses a time-based validation design because future customers may differ from historical customers. Metrics include area under the precision–recall curve, recall among the top 10% of customers, calibration, expected campaign profit, and prediction latency. Accuracy is not emphasized because churn is relatively infrequent.

Solution and outcome

Suppose the linear SVM has strong ranking performance and low latency, while the neural network has marginally better validation ranking but materially poorer calibration and greater maintenance requirements. Naïve Bayes performs well on text-only features but less well when numeric behaviour is added. k-NN performs adequately in a small pilot but becomes slow and unstable after many features are introduced.

The team selects the linear SVM for production ranking and calibrates its scores on a separate validation set. A simple policy contacts customers above a threshold determined by expected incremental profit, not by an arbitrary probability of 0.5. A monitored champion–challenger process periodically compares the SVM with the neural network as new data arrive.

Learning points

The case illustrates that model choice is multidimensional. The most complex model is not necessarily the best business solution. Data representation, leakage control, temporal validation, calibration, intervention economics, and operational simplicity all affect the final decision.

8. Governance, Ethics, and Responsible Deployment

These models can reproduce historical discrimination if training labels reflect unequal treatment or unequal access to services. Protected attributes may be excluded from the model while proxy variables—such as location, device type, or purchasing history—still encode them. Analysts should assess performance across relevant demographic and operational groups, document data provenance, and investigate meaningful disparities.

A prediction should not be confused with a justification. If a model predicts that a customer is likely to default or churn, the organization should not assume that the prediction identifies an underlying cause. High-impact decisions may require human review, an explanation appropriate to the audience, an appeal process, and compliance with applicable privacy and sector regulations.

Data minimization, access controls, retention limits, audit logs, versioned models, and documented monitoring are important controls. Monitoring should cover drift in input distributions, changes in class prevalence, calibration, subgroup performance, latency, missingness, and business outcomes. Model governance should also specify who can approve changes, when retraining occurs, and how incidents are handled.

9. Contemporary Developments and Best Practices

Modern machine-learning practice increasingly combines these classical models with automated pipelines, representation learning, calibrated probability methods, and model monitoring. Sparse linear SVMs and naïve Bayes remain competitive in many text tasks, while neural networks dominate several unstructured-data applications. Approximate-neighbour indexes make k-NN more feasible at scale. Hybrid systems may use a neural network to create representations and a simpler classifier to produce decisions.

Best practices include:

• use a reproducible preprocessing and modeling pipeline;
• prevent leakage from imputation, scaling, feature selection, and target encoding;
• tune hyperparameters using validation data only;
• use stratified or time-aware splitting as appropriate;
• assess calibration when decisions depend on probabilities;
• report uncertainty and subgroup performance;
• compare against simple baselines;
• record computational cost, latency, and retraining requirements;
• retain a human escalation route for consequential decisions;
• monitor performance after deployment rather than treating evaluation as a one-time event.

Chapter Summary

k-nearest neighbours predicts from local similarity and is intuitive but sensitive to scaling, irrelevant variables, dimensionality, and prediction cost. Naïve Bayes applies Bayes’ theorem with a conditional-independence assumption; despite that simplification, it is fast and powerful for many sparse text problems. Support vector machines construct maximum-margin boundaries, with soft-margin parameter C controlling the penalty for violations and kernels enabling nonlinear decision surfaces. Neural networks learn layered nonlinear transformations through gradient-based optimization and are especially valuable for large or unstructured data, although they require careful tuning and governance.

The appropriate model depends on data geometry, sample size, sparsity, error costs, interpretability, calibration, latency, and maintenance. Reliable predictive analytics therefore requires more than fitting an algorithm: it requires leakage-resistant validation, suitable preprocessing, operational evaluation, responsible governance, and continuing monitoring.

Key Terms

k-nearest neighbours: An instance-based method that predicts from the outcomes of nearby training observations.

Distance metric: A rule for quantifying separation between observations.

Curse of dimensionality: The degradation of distance-based reasoning as the number of features increases.

Standardization: Transformation of a feature using its training mean and standard deviation.

Naïve Bayes: A probabilistic classifier using Bayes’ theorem and conditional independence of features given the class.

Prior probability: The probability assigned to a class before observing the current features.

Likelihood: The probability of observing features conditional on a class.

Laplace smoothing: Additive adjustment that prevents zero estimated probabilities.

Support vector: A training observation that influences the position of an SVM boundary.

Margin: The separation between an SVM decision boundary and its nearest observations.

Kernel: A function that computes similarity in an implicit feature space.

Slack variable: A quantity allowing an SVM observation to violate a strict separation constraint.

Hyperparameter: A configuration value selected outside the model’s learned parameters.

Neuron: A computational unit that applies weights, a bias, and an activation function.

Activation function: A nonlinear function applied to a neuron’s weighted input.

Backpropagation: Calculation of parameter gradients through a layered network.

Cross-entropy: A loss function commonly used for probabilistic classification.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Data leakage: Use of information during training that would not be available at prediction time.

Concept drift: Change over time in the relationship between predictors and the target.

Review Questions

Short Answer Questions

1. Why is feature scaling usually important for k-NN and SVMs?
2. State the conditional-independence assumption made by naïve Bayes.
3. What is the role of the parameter C in a soft-margin SVM?
4. Why are support vectors important?
5. Distinguish a model parameter from a hyperparameter.

Long Answer Questions

1. Explain how k-NN makes predictions and evaluate its strengths and limitations for a large customer database.
2. Describe naïve Bayes classification, including priors, likelihoods, smoothing, and the effect of correlated features.
3. Explain maximum-margin classification and the role of kernels in SVMs.
4. Describe the architecture and training process of a feed-forward neural network.
5. Develop a systematic framework for selecting among the four model families in a business application.

Analytical / Application Questions

1. A text-classification data set contains 100,000 sparse word features and 20,000 observations. Which models would you test first, and why? Specify how you would evaluate them.
2. A k-NN system performs well in development but becomes slow after deployment. Diagnose likely causes and propose at least three remedies.
3. A neural network has the best ranking metric but poor calibration, high latency, and large subgroup performance differences. Should it be deployed? Construct a justified recommendation that considers technical and governance factors.

Multiple Choice Questions with Answers

Question 1

Which characteristic is most important for ordinary Euclidean k-NN?

A. All features must be binary

B. Distances between observations should be meaningful

C. The target must be continuous

D. The data must be perfectly linearly separable

Correct Answer: B. Distances between observations should be meaningful

Question 2

In naïve Bayes, what does P(c|x) represent?

A. The probability of the features given the class

B. The prior probability of the class

C. The probability of the class given the observed features

D. The probability of the features independent of the class

Correct Answer: C. The probability of the class given the observed features

Question 3

What is the primary purpose of Laplace smoothing?

A. To increase the number of features

B. To prevent zero estimated probabilities

C. To standardize continuous variables

D. To maximize the SVM margin

Correct Answer: B. To prevent zero estimated probabilities

Question 4

In an SVM, support vectors are observations that:

A. Are always misclassified

B. Determine or strongly influence the decision boundary

C. Have the largest feature values

D. Are removed before training

Correct Answer: B. Determine or strongly influence the decision boundary

Question 5

What does a large value of C generally do in a soft-margin SVM?

A. Penalizes margin violations more heavily

B. Removes the intercept

C. Forces all features to be binary

D. Guarantees a linear boundary

Correct Answer: A. Penalizes margin violations more heavily

Question 6

Which activation function is defined as max(0,a)?

A. Sigmoid

B. Softmax

C. ReLU

D. Cosine

Correct Answer: C. ReLU

Question 7

A binary classifier predicts p = 0.8 for an observation whose true label is y = 1. Using cross-entropy loss, which expression is correct?

A. −log(0.8)

B. −log(0.2)

C. −[0.8 log(1) + 0.2 log(0)]

D. 0.8 − 1

Correct Answer: A. −log(0.8)

Question 8

Why can k-NN become unreliable in very high-dimensional data?

A. The target disappears from the data

B. Nearby and distant observations become less distinguishable

C. The algorithm always becomes linear

D. Standardization becomes mathematically impossible

Correct Answer: B. Nearby and distant observations become less distinguishable

Question 9

Which model is especially associated with sparse text classification?

A. Naïve Bayes

B. Unscaled k-NN only

C. Hard-margin SVM only

D. A single linear regression equation

Correct Answer: A. Naïve Bayes

Question 10

Why should preprocessing such as scaling be fitted within training folds during cross-validation?

A. To increase the test-set size

B. To prevent validation information from leaking into training

C. To guarantee perfect accuracy

D. To eliminate the need for a test set

Correct Answer: B. To prevent validation information from leaking into training