11. Statistical Learning and Classical Computer Vision

Introduction and Context

Image processing transforms, enhances, and represents visual data; computer vision uses such representations to infer properties of objects, scenes, and events. The transition from image processing to vision is therefore a transition from measurement to interpretation. Classical computer vision addresses this transition through engineered features, statistical models, geometric reasoning, and carefully designed decision pipelines.

Although deep neural networks dominate many contemporary benchmarks, classical methods remain important in postgraduate study and professional systems. They are useful when training data are limited, interpretability is required, computational resources are constrained, or domain knowledge can be expressed through features and geometric rules. They also provide the conceptual foundations of modern machine learning: feature spaces, probabilistic inference, margins, model selection, clustering, ensembles, and structured prediction.

A practical vision system rarely consists of a single algorithm. It normally includes image acquisition, preprocessing, feature extraction, model training, validation, decision-making, and post-processing. Errors may arise at any stage. A classifier with excellent laboratory accuracy may fail in deployment because illumination changes, objects are partially occluded, or the training data do not represent the operational environment. Statistical learning provides the principles needed to analyze and manage these issues.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Formulate image classification and detection as statistical decision problems.

2. Explain generative, discriminative, parametric, and non-parametric learning models.

3. Apply clustering, support vector machines, ensemble methods, and graphical models to vision tasks.

4. Design training, validation, testing, and threshold-selection pipelines.

5. Select suitable performance measures for imbalanced classification and detection.

6. Analyze the effects of feature design, distribution shift, overfitting, and uncertainty.

7. Evaluate practical, ethical, and organizational considerations in deploying classical vision systems.

From Images to Statistical Decisions

Let an image or image region be represented by a feature vector x in a feature space X. The target may be a class label y, a continuous quantity, an object location, or a structured configuration. Classification estimates a decision rule g(x) that assigns x to one of K classes. Detection adds the question of where an object occurs, often by applying a classifier to candidate windows or region proposals.

A probabilistic formulation models the posterior probability P(y|x). The Bayes decision rule selects the class with the largest posterior probability when all errors have equal cost:

ŷ = arg max_y P(y|x).

Using Bayes' rule,

P(y|x) = P(x|y)P(y) / P(x).

A generative classifier estimates class-conditional distributions P(x|y) and priors P(y). A discriminative classifier estimates P(y|x) directly or learns a boundary between classes. The distinction is important: generative models can synthesize or complete observations and may work well with limited data under correct assumptions, whereas discriminative models often achieve stronger predictive performance when sufficient labeled data are available.

The equal-cost assumption is frequently unrealistic. In medical screening, missing a disease may be more costly than producing a false alarm. A cost-sensitive decision rule minimizes expected loss:

ŷ = arg min_a Σ_y L(a,y)P(y|x),

where L(a,y) is the cost of taking action a when the true class is y. Thus, classification is not merely the production of a label; it is a decision process involving uncertainty, costs, and operating conditions.

Features and Representation

Classical methods depend strongly on the representation supplied to them. A raw pixel vector preserves detailed information but is sensitive to translation, illumination, scale, and deformation. Useful engineered features encode properties expected to be stable for the task. Examples include intensity histograms, color moments, local binary patterns, texture descriptors, image gradients, edges, corners, shape contexts, and histogram-of-oriented-gradient (HOG) descriptors.

A feature should ideally have high discriminative power, limited redundancy, and reasonable invariance to nuisance variation. Feature normalization is often necessary, particularly for distance-based methods and support vector machines. Principal component analysis (PCA) may reduce dimensionality, remove correlated directions, and accelerate computation, although variance preservation does not necessarily imply class separability.

Probabilistic Models for Visual Recognition

Naïve Bayes and Gaussian Classifiers

The naïve Bayes model assumes conditional independence of features given the class:

P(x|y) = Π_j P(x_j|y).

This assumption is usually false for image features, yet the model can be effective when estimation simplicity offsets model mismatch. Gaussian naïve Bayes models each feature using a class-specific normal distribution. A multivariate Gaussian classifier instead represents the feature vector by a mean vector and covariance matrix. With a shared covariance matrix across classes, the resulting decision boundary is linear; with class-specific covariances, it becomes quadratic.

Covariance estimation is a major issue in high-dimensional vision. If the number of features is comparable to or greater than the number of training samples, covariance matrices may be unstable or singular. Regularization, diagonal covariance approximations, shrinkage, or dimensionality reduction can help.

Logistic Regression

Logistic regression is a discriminative probabilistic model. For binary classification,

P(y=1|x) = 1/(1 + exp(-(wᵀx+b))).

The parameters are usually estimated by minimizing cross-entropy loss, often with an L1 or L2 penalty. L1 regularization encourages sparse feature selection, whereas L2 regularization discourages large weights and generally yields smoother solutions. Logistic regression is attractive because its scores have a direct probabilistic interpretation, its optimization is well understood, and the effect of features can often be inspected.

Mixture Models and the Expectation-Maximization Algorithm

A mixture model assumes that observations arise from several latent subpopulations. A Gaussian mixture model (GMM) represents the density as

p(x) = Σ_{k=1}^{M} π_k N(x; μ_k, Σ_k),

where π_k are mixture weights. The hidden variable indicates the component responsible for an observation. Parameters can be estimated using expectation-maximization (EM). In the expectation step, responsibilities are computed for each observation and component. In the maximization step, parameters are updated using those responsibilities. The procedure repeats until convergence or a stopping criterion is met.

GMMs support soft assignments and can model clusters with different orientations and sizes. However, the likelihood may have local optima, covariance estimates may degenerate, and the number of components must be selected using domain knowledge, information criteria, or validation.

Clustering and Visual Data Organization

Clustering is unsupervised learning: the training data do not contain class labels. It is used for image segmentation, visual vocabulary construction, anomaly discovery, data exploration, and semi-automatic labeling.

K-means partitions observations into K groups by minimizing within-cluster squared Euclidean distance. It alternates between assigning each observation to its nearest centroid and recomputing centroids. Its simplicity and scalability are valuable, but it assumes compact, approximately spherical clusters and is sensitive to initialization, scale, outliers, and the selected value of K.

Hierarchical clustering builds a tree of nested groups. Agglomerative methods begin with individual observations and repeatedly merge clusters according to a linkage criterion. The resulting dendrogram allows the analyst to select a resolution after inspecting the hierarchy. Density-based methods, such as DBSCAN, identify dense regions separated by sparse areas and can label isolated points as noise. They are useful when clusters have irregular shapes, but their results depend on neighborhood parameters and may be difficult when cluster densities vary greatly.

Clustering quality should not be judged solely by a numerical index. The appropriate evaluation depends on purpose. If clusters are intended to represent semantic categories, external validation against labels may be appropriate. If the objective is compression or segmentation, reconstruction quality, boundary accuracy, or downstream task performance may be more meaningful.

Support Vector Machines and Maximum-Margin Learning

A support vector machine (SVM) seeks a separating hyperplane wᵀx+b=0. For linearly separable data, it chooses the hyperplane with the largest margin, where the margin is the distance to the closest training points. These closest points are the support vectors.

Real data are rarely perfectly separable. The soft-margin formulation introduces slack variables and minimizes

(1/2)||w||² + CΣ_i ξ_i,

subject to y_i(wᵀx_i+b) ≥ 1−ξ_i and ξ_i≥0. The parameter C controls the trade-off between a wide margin and training errors. A large C penalizes violations strongly and may produce a complex boundary; a small C permits more violations and stronger regularization.

The kernel trick replaces inner products with a kernel function K(x_i,x_j), allowing an implicit mapping into a higher-dimensional feature space. Common choices include linear, polynomial, and radial basis function kernels. Kernel parameters, feature scaling, and C must be selected using validation rather than training accuracy.

SVMs are effective in medium-sized, high-dimensional problems, especially when feature vectors such as HOG or texture descriptors are informative. Their limitations include expensive training for very large datasets, sensitivity to parameter choices, and less direct probabilistic interpretation. Multiclass classification can be implemented through one-versus-rest or one-versus-one strategies.

Ensembles: Combining Imperfect Learners

An ensemble combines multiple models to obtain a more reliable predictor. The central idea is that errors can be reduced when individual learners are both reasonably accurate and not perfectly correlated.

Bagging trains models on bootstrap samples and aggregates their predictions. Random forests extend this idea by using randomized feature subsets at tree nodes. They handle nonlinear relationships, mixed feature scales, and feature interactions with little preprocessing. They also provide useful variable-importance measures, although such measures can be biased and should not be interpreted as causal evidence.

Boosting builds learners sequentially, emphasizing observations that previous learners handled poorly. AdaBoost combines weak classifiers through weighted voting. Gradient boosting fits new learners to the residual structure of a loss function. Boosting can produce highly accurate models but may overfit noisy data and requires careful control of depth, learning rate, number of iterations, and regularization.

Ensemble validation must preserve the data structure. For video, randomly distributing frames across training and test sets can cause near-duplicate scenes to appear in both sets, producing inflated performance. Splitting by subject, session, location, or time is often more defensible.

Graphical Models and Structured Vision

Many visual problems contain dependencies that independent classification cannot represent. Neighboring pixels tend to have related labels; object parts have spatial relationships; and successive video frames are temporally correlated. Graphical models express these dependencies using nodes for variables and edges for relationships.

In a Bayesian network, directed edges encode conditional dependencies and the joint distribution factorizes according to the graph. In a Markov random field (MRF), an undirected graph represents compatibility among neighboring variables. A conditional random field (CRF) models P(y|x) directly and can incorporate image features while encouraging coherent label configurations.

For image segmentation, an energy function may be written as

E(y|x) = Σ_i D_i(y_i|x) + λΣ_(i,j) V_ij(y_i,y_j|x),

where D_i is a data term measuring how well label y_i explains pixel i, and V_ij is a pairwise smoothness or compatibility term. The optimal labeling minimizes this energy. Graph cuts, belief propagation, mean-field inference, and dynamic programming are among the methods used, depending on the structure and form of the energy.

Graphical models are valuable because they separate local evidence from relational constraints. However, inference may be computationally demanding, pairwise smoothness can oversimplify boundaries, and the model may impose unwanted regularity on thin or disconnected structures.

Designing a Training and Decision Pipeline

A defensible vision pipeline begins with a precise operational definition of the target. “Detect defects” is incomplete unless defect types, minimum size, acceptable false-alarm rate, processing latency, and consequences of error are specified.

The dataset should then be partitioned into training, validation, and test sets. The test set must remain untouched until final evaluation. Preprocessing steps such as normalization, feature selection, PCA, and imputation must be fitted using training data only; otherwise, information leakage occurs. When observations are correlated, the unit of splitting should be the independent entity rather than the individual image.

Model selection uses validation data or nested cross-validation. Hyperparameters include the number of clusters, SVM kernel and penalty, tree depth, ensemble size, and graphical-model smoothness. Repeated cross-validation can estimate variability, but it does not replace an independent test set.

For binary decisions, a score threshold converts a continuous output into an action. Lowering the threshold generally increases recall and decreases precision. Receiver operating characteristic (ROC) curves show true-positive rate against false-positive rate at varying thresholds. Precision-recall curves are often more informative when positive cases are rare. A confusion matrix reveals class-specific errors, while calibration measures whether predicted probabilities correspond to observed frequencies.

Detection systems require localization metrics in addition to classification metrics. A predicted bounding box is commonly considered correct when its intersection over union (IoU) with the reference box exceeds a selected threshold. Precision, recall, average precision, and false detections per image should be interpreted together with computational latency and missed-object costs.

Applied Scenario: Visual Inspection of Surface Defects

Background

Consider an illustrative manufacturing system for inspecting coated metal panels. Each panel is photographed under controlled illumination, and the system must identify scratches, pinholes, and coating irregularities. Human inspectors are available for escalation, but continuous manual inspection is costly and inconsistent.

Situation and Problem

Only a modest number of labeled defect images are available. Defects vary in size and orientation, and some normal textures resemble defects. A simple accuracy score is inadequate because defects are uncommon and missed defects have greater financial consequences than additional inspections.

Analysis

The engineering team first standardizes camera exposure and removes irrelevant borders. Candidate regions are generated using local contrast and edge information. Each region is represented by intensity statistics, local binary pattern texture features, gradient-orientation histograms, and shape measurements. Features are scaled using parameters calculated from the training set.

Exploratory clustering is used to identify recurring normal-texture patterns and unusual observations. A linear SVM provides a strong baseline because the engineered descriptors are moderately separable. A random forest is added to capture nonlinear interactions between texture and shape. A small graphical post-processing model links neighboring candidate regions so that isolated single-pixel detections are suppressed while connected defects are retained.

The data are split by production batch rather than by random image, preventing nearly identical panels from appearing in both training and testing. Model parameters are selected through cross-validation within the training batches. The decision threshold is chosen on the validation set to achieve a recall target established with quality engineers. The untouched test batches are then used once for final reporting.

Outcome

The combined pipeline is more useful than any individual classifier because it integrates local appearance, ensemble decisions, and spatial coherence. Its acceptance is based not only on detection metrics but also on false alarms per panel, inspection time, interpretability of flagged regions, and the percentage of cases safely escalated to a human.

Learning Points

The case illustrates five principles: representation quality often determines classical-model performance; data splitting must reflect operational dependence; threshold selection is a business and safety decision; spatial context can improve noisy local predictions; and deployment evaluation must include workflow costs rather than a single accuracy figure.

Challenges, Risks, and Common Mistakes

Overfitting is common when feature dimensions are large relative to labeled samples. Increasing model complexity cannot compensate for unrepresentative data. Leakage is another serious error: using test-set statistics for normalization, allowing images from the same subject into multiple partitions, or selecting a threshold after viewing test results all produce optimistic estimates.

Class imbalance can cause a classifier to appear accurate while ignoring the minority class. Resampling, class-weighted loss, anomaly detection, and threshold adjustment are possible responses, but each changes the decision problem. Synthetic examples may help, yet unrealistic augmentation can introduce artifacts or alter class boundaries.

Domain shift occurs when camera properties, illumination, object design, or operating conditions change. Monitoring feature distributions and error patterns is essential. A model should have an abstention or human-review option when uncertainty is high. Calibration and confidence intervals are particularly important when decisions affect safety, finance, or access to services.

Classical features may fail under severe deformation, clutter, occlusion, or large appearance variation. Conversely, an apparently sophisticated model may be inappropriate when it cannot meet latency, memory, explainability, or maintenance requirements. Model selection should therefore consider the entire system rather than benchmark accuracy alone.

Professional, Ethical, and Governance Considerations

Vision systems can influence employment, medical triage, surveillance, and access control. Training data should be collected lawfully, with appropriate consent and retention policies. Sensitive attributes should not be inferred or used without a legitimate purpose. Performance must be examined across relevant demographic, environmental, and operational subgroups.

Documentation should describe the intended use, data sources, known failure modes, validation protocol, threshold rationale, and escalation process. Audit logs are valuable when decisions must be explained or investigated. Human oversight should be meaningful rather than nominal: reviewers need adequate information, authority to override the system, and procedures for correcting recurring errors.

Contemporary Directions

Classical methods increasingly operate as components of hybrid systems. Engineered descriptors may provide interpretable baselines, quality-control checks, or inputs to learned models. Self-supervised representation learning can reduce dependence on labels, while active learning prioritizes samples whose labels are most informative. Domain adaptation addresses changes between training and deployment environments.

Uncertainty estimation, conformal prediction, and selective classification are gaining attention because reliable systems should know when not to decide. Edge deployment also encourages compact models, efficient feature extraction, and hardware-aware inference. These developments do not eliminate the need for statistical reasoning; they make data governance, validation design, and decision analysis even more important.

Best Practices for Deployment

A professional implementation should:

• define the decision, costs, and acceptable operating range before selecting a model;
• establish a simple, interpretable baseline;
• separate entities, sessions, batches, or time periods when creating evaluation sets;
• fit all learned preprocessing only on training data;
• tune thresholds using validation data and report performance at the selected operating point;
• inspect errors visually and quantitatively, not only through aggregate scores;
• measure calibration, latency, memory use, and robustness to environmental changes;
• include monitoring, human escalation, version control, and retraining criteria.

Chapter Summary

Statistical learning converts image representations into predictions and decisions under uncertainty. Generative models estimate how features arise from classes, while discriminative models learn posterior probabilities or separating boundaries. Clustering supports unsupervised organization and discovery; SVMs provide maximum-margin classification; ensembles reduce variance or improve accuracy through aggregation; and graphical models represent spatial or temporal dependencies.

Successful computer vision systems depend on more than algorithm choice. Feature design, data partitioning, leakage prevention, hyperparameter validation, threshold selection, class imbalance management, and deployment monitoring are equally important. Evaluation should reflect the operational costs of false positives and false negatives, as well as localization, calibration, latency, and fairness. Classical methods remain valuable as interpretable baselines, efficient production tools, and conceptual foundations for contemporary hybrid vision systems.

Key Terms

Bayes decision rule: A rule that selects the action with minimum expected loss given posterior probabilities.

Class-conditional distribution: The probability distribution of features given a particular class.

Discriminative model: A model that estimates class boundaries or P(y|x) directly.

Generative model: A model that represents how observations and labels are jointly produced.

Feature descriptor: A numerical representation designed to capture useful visual properties.

PCA: A linear dimensionality-reduction method that projects data onto directions of greatest variance.

Gaussian mixture model: A probabilistic density represented as a weighted mixture of Gaussian components.

Expectation-maximization: An iterative method for estimating parameters with hidden variables.

K-means: A clustering algorithm that minimizes within-cluster squared distance to centroids.

Support vector: A training observation that determines an SVM decision boundary.

Kernel trick: A method for computing inner products in an implicit high-dimensional feature space.

Soft margin: An SVM formulation that permits classification violations through slack variables.

Bagging: Ensemble learning using bootstrap samples and aggregation of model predictions.

Boosting: Sequential ensemble learning that emphasizes errors made by earlier learners.

Graphical model: A probabilistic model whose graph expresses dependencies among variables.

Conditional random field: A discriminative graphical model for structured outputs conditioned on observations.

IoU: Intersection over union between predicted and reference regions.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Data leakage: Use of information during training that would be unavailable at deployment or evaluation.

Distribution shift: A change between the data-generating conditions of training and deployment.

Review Questions

Short Answer Questions

1. Distinguish a generative classifier from a discriminative classifier.

2. Why is feature scaling important for an SVM?

3. What is the purpose of the expectation step in EM?

4. Define intersection over union.

5. What is data leakage in a computer vision pipeline?

Long Answer Questions

1. Explain the Bayes decision framework and show how unequal error costs affect classification thresholds.

2. Compare K-means, Gaussian mixture models, and density-based clustering for visual data organization.

3. Describe the soft-margin SVM and explain the roles of C and kernel parameters.

4. Explain how graphical models incorporate spatial context into image segmentation.

5. Design a complete training and validation pipeline for an imbalanced image-detection problem.

Analytical / Application Questions

1. A defect detector has high accuracy but misses most rare defects. Which metrics, data strategies, and decision changes should be examined?

2. A video classifier performs extremely well under random frame splitting but poorly on a new camera. Diagnose the likely causes and propose a more valid evaluation design.

3. For a safety inspection system, compare a calibrated logistic model, an RBF-kernel SVM, and a random forest. Recommend an evaluation and deployment strategy rather than selecting solely by accuracy.

22. Multiple Choice Questions with Answers

Question 1

Which decision rule is appropriate when false negatives and false positives have unequal costs?

A. Choose the class with the largest raw feature value

B. Minimize expected loss using posterior probabilities and a loss function

C. Always select the majority class

D. Use the training accuracy as the threshold

Correct Answer: B. Minimize expected loss using posterior probabilities and a loss function

Question 2

What is the principal role of the expectation step in a Gaussian mixture model trained with EM?

A. It assigns hard class labels permanently

B. It computes responsibilities of mixture components for observations

C. It removes all covariance terms

D. It selects the number of image classes automatically

Correct Answer: B. It computes responsibilities of mixture components for observations

Question 3

A small value of C in a soft-margin SVM generally gives greater emphasis to:

A. Allowing margin violations in favor of regularization

B. Perfectly classifying every training sample

C. Increasing the number of support vectors to infinity

D. Removing the feature-space representation

Correct Answer: A. Allowing margin violations in favor of regularization

Question 4

Why can random frame-level splitting produce misleading results for video classification?

A. Frames cannot be represented numerically

B. Frames from the same scene may appear in both training and test sets

C. Validation sets always contain too few classes

D. SVMs cannot process sequential data

Correct Answer: B. Frames from the same scene may appear in both training and test sets

Question 5

Which clustering method is particularly designed to identify dense regions and label isolated observations as noise?

A. PCA

B. DBSCAN

C. Logistic regression

D. Bagging

Correct Answer: B. DBSCAN

Question 6

In an image-segmentation energy function, the pairwise term primarily encourages:

A. Independent pixel decisions

B. Spatial compatibility between neighboring labels

C. Random initialization of class means

D. Removal of all image boundaries

Correct Answer: B. Spatial compatibility between neighboring labels

Question 7

Which practice constitutes data leakage?

A. Selecting a threshold using a validation set

B. Fitting normalization parameters only on training data

C. Using test-set statistics to perform feature normalization

D. Reporting a confusion matrix on held-out data

Correct Answer: C. Using test-set statistics to perform feature normalization

Question 8

When positive cases are very rare, which evaluation tool is often more informative than ROC analysis alone?

A. Precision-recall curve

B. Image histogram only

C. Training loss without labels

D. Number of feature dimensions

Correct Answer: A. Precision-recall curve

Question 9

What is a central advantage of an ensemble?

A. It guarantees that every individual model is unbiased

B. It can reduce errors by combining learners with partially uncorrelated errors

C. It eliminates the need for validation

D. It always produces calibrated probabilities

Correct Answer: B. It can reduce errors by combining learners with partially uncorrelated errors

Question 10

In the surface-defect scenario, why was data split by production batch rather than by individual image?

A. To make the feature vectors shorter

B. To prevent highly related samples from inflating test performance

C. To ensure every batch had identical defects

D. To avoid using a validation set

Correct Answer: B. To prevent highly related samples from inflating test performance