6. Classical Machine Learning for Visual Recognition

Introduction and Context

Visual recognition is the task of assigning meaning to visual data: identifying an object, recognizing a scene, detecting a defect, classifying a medical image, or distinguishing one individual from another. Although deep neural networks dominate many contemporary benchmarks, classical machine learning remains important. It provides interpretable, computationally efficient, and often highly competitive solutions when datasets are small, computational resources are limited, or domain knowledge can be expressed through carefully designed features.

A classical visual-recognition system separates the problem into stages. Images are acquired and cleaned, informative representations are constructed, irrelevant variation is reduced, and a statistical classifier maps representations to labels. The quality of the final system depends not only on the classifier but also on preprocessing, feature design, sampling, validation, and error analysis. In practice, a modest classifier using a strong representation can outperform a sophisticated classifier operating on poorly prepared data.

This chapter develops a complete pipeline from image preprocessing to model evaluation. It also explains dimensionality reduction, clustering, ensemble learning, probabilistic models, cross-validation, and professional decision-making. The central principle is that visual recognition is a system-design problem rather than a choice of algorithm in isolation.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Construct a classical machine-learning pipeline for image classification.

2. Explain how preprocessing choices influence visual features and classifiers.

3. Select and compare handcrafted feature representations.

4. Apply dimensionality reduction and clustering appropriately.

5. Distinguish between discriminative, probabilistic, and ensemble classifiers.

6. Design valid training, validation, and testing procedures.

7. Diagnose errors using confusion matrices, subgroup analysis, and visual inspection.

8. Evaluate technical, operational, ethical, and governance considerations in visual recognition.

From Pixels to Decisions: The Classical Recognition Pipeline

An image is commonly represented as a tensor of pixel intensities. For a grayscale image, a pixel value may be a scalar; for a colour image, each pixel has channels such as red, green, and blue. The raw pixel vector is high-dimensional and sensitive to illumination, translation, scale, rotation, occlusion, and background variation. Classical machine learning therefore usually transforms the image into a feature vector before classification.

A general pipeline is:

image acquisition → preprocessing → feature extraction → feature selection or dimensionality reduction → model training → validation → deployment and monitoring.

The pipeline must be treated as a single experimental object. If normalization, feature selection, or principal component analysis is performed using the entire dataset before splitting, information from the test set can leak into training. The reported accuracy will then be optimistically biased.

Preprocessing as Controlled Variation Reduction

Preprocessing should remove variation that is irrelevant to the task while preserving variation that distinguishes classes. Typical operations include resizing, cropping, colour-space conversion, denoising, contrast adjustment, geometric alignment, and intensity normalization.

Resizing creates a common spatial format but may remove small objects or distort aspect ratios. Cropping can focus attention on a region of interest, although an inaccurate crop may eliminate useful context. Conversion from RGB to grayscale reduces dimensionality when colour is not discriminative; conversely, colour should be retained for tasks such as fruit maturity or tissue staining, where chromatic information is meaningful.

Noise reduction may use Gaussian, median, or bilateral filtering. A Gaussian filter smooths high-frequency noise but can blur edges. A median filter is useful for impulse noise. Contrast normalization can reduce illumination differences, while histogram equalization may improve visibility but can also amplify noise. These operations should be chosen through validation rather than applied automatically.

Geometric normalization is valuable when object pose is predictable. Alignment based on landmarks can reduce within-class variation in faces or documents. However, aggressive alignment may remove genuine class cues. Data augmentation, such as small rotations or translations, can increase robustness, but transformations must be physically plausible. A horizontally flipped road sign may be invalid if its meaning changes under reflection.

Feature Engineering for Images

Feature engineering converts an image into numerical descriptors designed to capture shape, texture, colour, or local structure. A useful feature should be discriminative between classes, relatively stable under expected nuisance variation, computationally feasible, and not excessively redundant.

Raw Pixels and Template-Like Representations

Raw pixels are simple and preserve complete information, but they have poor invariance. A small translation can produce a large Euclidean distance even when two images depict the same object. Raw pixels can nevertheless work for tightly aligned images with controlled backgrounds, especially with linear classifiers or dimensionality reduction.

Histogram of Oriented Gradients (HOG) describes local edge orientation. The image is divided into cells, gradient directions are accumulated into orientation histograms, and neighbouring cells are normalized in blocks. HOG is effective for shape and silhouette-based recognition, such as pedestrian detection, because local edge patterns are more stable than absolute intensity.

Local Binary Patterns (LBP) encode the relationship between a central pixel and its neighbours. The resulting codes are aggregated into histograms and are useful for texture analysis, including materials, facial texture, and surface inspection. LBP is relatively robust to monotonic illumination changes but can be sensitive to scale and noise.

Scale-Invariant Feature Transform (SIFT)-type descriptors identify salient keypoints and describe local gradient structure around them. Their strength is tolerance to scale and rotation and partial robustness to viewpoint changes. Speeded-Up Robust Features and related descriptors pursue similar goals with different computational compromises. Local descriptors may be matched directly or aggregated into a fixed-length representation.

Colour histograms represent the distribution of colours, often in HSV or another perceptually useful colour space. They are computationally inexpensive but discard spatial arrangement. Spatial pyramids partially restore layout by computing features over increasingly fine regions.

Texture features may include Gabor-filter responses, grey-level co-occurrence statistics, wavelet coefficients, or filter-bank energies. Shape descriptors may include contours, moments, Fourier descriptors, and geometric measurements. The appropriate representation follows the visual mechanism of the task: texture for fabric defects, edges for silhouettes, colour for produce classification, and geometry for document analysis.

Feature Scaling and Selection

Many algorithms depend on distances or dot products. If one feature has a much larger numerical range than another, it can dominate learning. Standardization transforms a feature using its training-set mean and standard deviation; min–max scaling maps values to a specified interval. Scaling parameters must be estimated only from training data and then applied unchanged to validation and test data.

Feature selection can improve generalization, interpretability, and computational efficiency. Filter methods rank features using correlation, mutual information, or statistical tests. Wrapper methods evaluate subsets through model performance. Embedded methods perform selection during training, as in L1-regularized linear models or tree-based importance measures. Selection should be nested within cross-validation when it is used to estimate performance.

Dimensionality Reduction and Visual Structure

Image descriptors can contain hundreds or thousands of dimensions. High dimensionality increases storage and computation, creates redundancy, and can make distance-based learning unreliable. Dimensionality reduction maps the original feature vector to a lower-dimensional representation.

Principal Component Analysis (PCA) finds orthogonal directions of maximum variance. If the centered feature matrix is decomposed into principal components, the first components provide a low-dimensional approximation that minimizes squared reconstruction error among linear projections of the same dimension. PCA is useful for compression, visualization, denoising, and reducing input size before classification. Maximum variance, however, is not necessarily maximum class separation; a high-variance illumination effect may be irrelevant to the labels.

Linear Discriminant Analysis (LDA), when used as a projection method, seeks directions that increase between-class scatter relative to within-class scatter. It is supervised and can be effective when class distributions are reasonably represented by class means and covariance structure. The number of useful LDA dimensions is limited by the number of classes.

Nonlinear methods such as t-SNE and UMAP are valuable for exploratory visualization, but their two-dimensional maps should not automatically be treated as evidence of classifier performance. Distances, cluster sizes, and global structure may be distorted. A visualization can suggest hypotheses; quantitative evaluation must test them.

Discovering Structure with Clustering

Clustering is unsupervised grouping. It can support exploratory analysis, image organization, prototype discovery, anomaly detection, and pseudo-labelling. It is not equivalent to classification: clusters are formed without access to ground-truth labels, and a cluster need not correspond to a semantic class.

K-means partitions observations into a chosen number of clusters by minimizing within-cluster squared Euclidean distance. It is efficient and interpretable but assumes compact, roughly spherical clusters and is sensitive to scaling and initialization. K-means++ initialization and multiple restarts improve stability.

Hierarchical clustering builds a tree of nested groups using either agglomerative merging or divisive splitting. The dendrogram allows the analyst to select a level of granularity, although computation can be expensive for large datasets. Density-based methods such as DBSCAN identify dense regions and can mark isolated points as noise. They are useful when clusters have irregular shapes, but parameter selection and varying density are challenging.

Clustering can reveal duplicate images, mislabeled samples, acquisition batches, or underrepresented populations. It should not be used to assign meaning to groups without domain inspection. A visually coherent cluster may reflect camera type or lighting rather than object category.

Classifiers for Visual Recognition

Linear Models

Logistic regression estimates class probabilities through a linear decision function followed by a logistic or softmax transformation. It is fast, interpretable, and an effective baseline for many engineered features. Regularization controls coefficient magnitude and reduces overfitting. L2 regularization spreads influence across correlated features, whereas L1 regularization can produce sparse solutions.

The support vector machine (SVM) seeks a separating hyperplane with a large margin. A soft-margin formulation permits errors and balances margin width against training violations. Kernel SVMs implicitly map data into richer feature spaces and can model nonlinear boundaries, but they become costly for very large datasets. The radial basis function kernel is powerful when its scale and regularization parameters are tuned carefully.

Tree-Based Methods

Decision trees recursively divide the feature space using threshold rules. They are easy to inspect but can overfit. Random forests average many trees trained with sample and feature randomness, reducing variance and often performing well on mixed or nonlinear features. Gradient boosting builds trees sequentially, each correcting errors made by earlier trees. Boosting can be highly accurate but is sensitive to hyperparameters, noise, and leakage.

Tree models do not require feature scaling, unlike many distance- and margin-based methods. Their feature importance scores should be interpreted cautiously because correlated features can distribute importance unpredictably.

Probabilistic Models

Naive Bayes applies Bayes’ rule while assuming conditional independence of features given the class. The assumption is often unrealistic for image descriptors, but the model can still perform effectively, particularly with histogram-like or high-dimensional sparse inputs. Gaussian mixture models represent each class or the overall data as a weighted mixture of Gaussian components. They provide soft assignments and likelihoods, supporting anomaly detection and uncertainty estimation.

A generative model describes how data may arise from a class; a discriminative model focuses directly on separating classes or estimating the conditional distribution of labels given features. Probabilistic outputs are useful in risk-sensitive applications, but scores should be calibrated. A classifier that assigns 0.9 probability should be correct approximately 90 percent of the time among comparable predictions if it is well calibrated.

Ensembles and Decision Policies

An ensemble combines multiple models. Bagging reduces variance by averaging models trained on resampled data. Boosting builds a sequence of increasingly focused learners. Voting ensembles combine predictions directly, while stacking trains a meta-model on outputs from base learners. Diversity is central: identical models making identical errors provide little benefit.

In professional systems, the classifier is only one component of the decision policy. A probability threshold may be selected to prioritize recall, precision, or a cost-weighted objective. In medical screening, missing a positive case may be more costly than referring a false positive. In automated inspection, an uncertain sample may be routed to a human reviewer rather than forced into a class.

Evaluation, Cross-Validation, and Error Analysis

A sound evaluation begins with a meaningful split. Images from the same subject, video, patient, product batch, or physical object should not appear across training and test sets when the intended deployment concerns unseen subjects or objects. Grouped or subject-level splitting prevents near-duplicate leakage.

Cross-validation divides the training data into folds, trains on some folds, and validates on the remaining fold. Stratified cross-validation preserves class proportions. Grouped cross-validation preserves subject or acquisition-group separation. Nested cross-validation is appropriate when hyperparameter selection must be separated from unbiased performance estimation.

Accuracy is informative under balanced and equally costly classes. Precision measures the proportion of predicted positives that are correct; recall measures the proportion of actual positives detected. The F1 score is their harmonic mean. Specificity measures the proportion of actual negatives correctly rejected. Balanced accuracy averages class-wise recall and is useful under imbalance. Receiver operating characteristic and precision–recall curves describe threshold behaviour, while calibration measures the reliability of probabilities.

A confusion matrix is a starting point for error analysis. Errors should then be examined by class, object scale, lighting, viewpoint, device, geographic region, demographic group, and confidence. Visualizing false positives and false negatives often exposes background shortcuts, annotation mistakes, systematic blur, or an unrepresented operating condition. Error analysis should lead to a specific intervention: revised preprocessing, additional data, a better feature, threshold adjustment, or a change in the task definition.

Applied Case: Surface-Defect Recognition in Manufacturing

Background

Consider an illustrative case involving a manufacturer of coated metal panels. Each panel is photographed after production, and inspectors label images as acceptable, scratched, dented, or stained. The company has a modest labelled dataset and wishes to reduce inspection time without removing human oversight.

Situation and Analysis

Initial experiments use raw RGB pixels with a nearest-neighbour classifier. Performance is unstable because lighting varies between production shifts, and the background occupies a large part of each image. Examination of false positives shows that reflections are being mistaken for scratches. The classes are also imbalanced: acceptable panels are much more common than defective panels.

Approach

The team first standardizes camera placement, crops the panel region, converts a copy of the image to grayscale, and applies mild illumination normalization. It extracts HOG features for elongated scratches and LBP and Gabor features for stain texture. Colour histograms are retained because stains differ chromatically from the coating. Features are standardized using training-set statistics, and PCA is fitted within each cross-validation training fold to reduce redundancy.

A linear SVM, random forest, and gradient-boosted model are compared using grouped cross-validation by production batch. Performance is reported using per-class recall, precision, balanced accuracy, and a precision–recall curve for the rare defect classes. An ensemble is considered only if its errors are sufficiently different. High-confidence acceptable predictions are automated, while low-confidence or defect predictions are referred to an inspector.

Outcome

The redesigned pipeline is more stable across production batches than the raw-pixel baseline. More importantly, error analysis identifies a need for better lighting control rather than merely a more complex classifier. The organization documents the threshold policy, keeps images associated with batch metadata, and monitors drift after deployment.

Learning Points

The case illustrates that preprocessing and acquisition design can be as important as model selection. It also demonstrates why grouped validation, class-sensitive metrics, calibrated confidence, and human escalation are essential in an operational vision system.

Challenges, Risks, and Professional Considerations

Classical pipelines are vulnerable to dataset shift. A new camera, season, supplier, compression format, or illumination condition may change feature distributions. Class imbalance can cause a model to achieve high accuracy while failing the minority class. Label noise can make apparent model errors actually annotation errors. Correlated samples can produce misleadingly strong validation results.

Handcrafted features encode assumptions. HOG may fail when shape is not stable; colour descriptors may fail under illumination changes; local descriptors may be computationally expensive; and PCA may remove low-variance but discriminative information. No method is universally best.

Visual recognition also raises privacy and fairness concerns. Face or identity-related systems require lawful data collection, purpose limitation, access control, retention policies, and careful assessment of subgroup performance. Training images should have documented provenance and consent or another valid legal basis where required. Human review, appeal mechanisms, audit logs, and clear accountability are important when model outputs affect employment, access, safety, or healthcare.

Reproducibility requires versioned data, fixed preprocessing definitions, recorded random seeds where appropriate, documented hyperparameters, and separation of development from final testing. A model should be monitored after deployment for performance degradation, confidence changes, and shifts in error distribution.

Current Developments and Best Practices

Classical methods increasingly operate in hybrid systems. Handcrafted descriptors may be combined with compact learned embeddings, while classical classifiers can be trained on representations produced by a pretrained vision model. Such systems are useful when labelled data are limited or when an organization needs a lightweight, interpretable decision layer.

Best practice is to establish a simple, reproducible baseline before increasing complexity. Use domain-informed preprocessing, prevent leakage, preserve group separation, tune on validation data, report more than one metric, inspect errors visually, and assess performance under realistic operating conditions. Automation should include abstention or human review when uncertainty is high. The final choice should reflect accuracy, latency, maintenance cost, interpretability, privacy, and the consequences of errors—not benchmark performance alone.

Chapter Summary

Classical machine learning for visual recognition is a structured process that transforms images into robust representations and then learns a decision rule. Preprocessing controls nuisance variation, while features such as HOG, LBP, local descriptors, colour histograms, and texture measurements encode visual evidence. PCA and LDA can reduce or reorganize feature spaces, whereas clustering supports exploratory analysis and anomaly discovery. Linear models, SVMs, tree methods, probabilistic models, and ensembles offer different trade-offs in accuracy, interpretability, scalability, and uncertainty representation.

Reliable evaluation requires leakage-free splitting, appropriate cross-validation, class-sensitive metrics, calibration, and detailed error analysis. In professional applications, model thresholds, human escalation, monitoring, privacy, fairness, and governance are part of the recognition system. The most effective pipeline is therefore not necessarily the most complex one; it is the one whose representation, validation design, decision policy, and operational controls match the real problem.

Key Terms

Classical machine learning: Learning methods that rely on engineered representations and statistical models rather than end-to-end deep feature learning.

Feature engineering: Designing numerical measurements that capture task-relevant visual properties.

Preprocessing: Transformations applied to images before feature extraction or modelling.

HOG: A descriptor based on local histograms of image-gradient orientations.

LBP: A texture descriptor formed from local pixel-intensity comparisons.

SIFT-type descriptor: A local feature representation designed for tolerance to scale and rotation.

Feature scaling: Rescaling variables so that numerical magnitude does not distort learning.

PCA: An unsupervised linear projection preserving directions of greatest variance.

LDA: A supervised projection emphasizing between-class separation relative to within-class variation.

Clustering: Unsupervised grouping of observations according to similarity.

K-means: A partitioning algorithm minimizing within-cluster squared distances.

SVM: A margin-based classifier that can use linear or kernel decision boundaries.

Regularization: A constraint or penalty used to reduce overfitting.

Ensemble: A combination of multiple predictive models.

Calibration: Agreement between predicted probabilities and observed frequencies.

Cross-validation: Repeated training and validation over different data folds.

Data leakage: Use of information during training that would not be available at deployment.

Confusion matrix: A table showing predicted versus actual class counts.

Concept drift: Change over time in the data or relationship between inputs and labels.

Abstention: Declining to make an automatic decision and referring the case for review.

Review Questions

Short Answer Questions

1. Why can raw-pixel representations be sensitive to small translations?

2. What visual information is captured by HOG?

3. Distinguish PCA from LDA as dimensionality-reduction methods.

4. Why is grouped cross-validation important for image datasets containing video frames?

5. What does calibration mean in a probabilistic classifier?

Long Answer Questions

1. Explain the stages of a classical visual-recognition pipeline and the way errors can propagate between stages.

2. Compare HOG, LBP, local descriptors, colour histograms, and texture features in terms of the visual properties they represent.

3. Discuss linear models, SVMs, tree-based models, and probabilistic models as alternatives for image classification.

4. Explain how leakage, class imbalance, and dataset shift can produce misleading evaluation results.

5. Describe a complete error-analysis procedure for a deployed visual-recognition system.

Analytical / Application Questions

1. Design a feature and classifier pipeline for classifying fabric images into weave types. Justify preprocessing, descriptors, validation strategy, and metrics.

2. A classifier has 98% accuracy but very low recall for a rare safety defect. Analyze the likely cause and propose a revised evaluation and decision policy.

3. A clustering analysis separates images mainly by camera brand rather than object category. Explain why this occurred and recommend corrective actions.

22. Multiple Choice Questions with Answers

Question 1

A vision system uses PCA fitted on all images before the training and test split. What is the principal methodological problem?

A. PCA cannot be used with image features

B. The test set influences the learned representation, causing leakage

C. PCA always removes class-discriminative information

D. The classifier must be nonlinear after PCA

Correct Answer: B. The test set influences the learned representation, causing leakage

Question 2

Which feature is most directly designed to represent local edge orientation and silhouette structure?

A. HOG

B. Colour histogram

C. Naive Bayes

D. K-means

Correct Answer: A. HOG

Question 3

A dataset contains many video frames from the same person. Which validation design best estimates performance on unseen people?

A. Randomly split individual frames

B. Use the training accuracy

C. Use grouped splitting by person

D. Duplicate minority-class frames across folds

Correct Answer: C. Use grouped splitting by person

Question 4

Why might a colour histogram be insufficient for distinguishing two objects with identical colour distributions but different layouts?

A. It generally discards spatial arrangement

B. It cannot be standardized

C. It requires labelled clusters

D. It only works with binary images

Correct Answer: A. It generally discards spatial arrangement

Question 5

Which statement best describes an SVM?

A. It always estimates a generative probability distribution

B. It seeks a decision boundary with a large margin

C. It groups data without labels using centroids

D. It removes all correlated features before training

Correct Answer: B. It seeks a decision boundary with a large margin

Question 6

A manufacturing classifier performs well in the laboratory but poorly after a new camera is installed. This is most directly an example of:

A. Dataset shift

B. Perfect calibration

C. Stratification

D. Dimensionality collapse

Correct Answer: A. Dataset shift

Question 7

Which metric is especially useful when positive cases are rare and false positives matter?

A. Precision

B. Image width

C. Within-cluster variance only

D. Training loss alone

Correct Answer: A. Precision

Question 8

What is a major purpose of an abstention policy in visual recognition?

A. To force every uncertain sample into the majority class

B. To avoid collecting validation data

C. To route uncertain cases to human review

D. To eliminate the need for calibration

Correct Answer: C. To route uncertain cases to human review

Question 9

Which clustering method is explicitly capable of labelling isolated observations as noise?

A. DBSCAN

B. Linear regression

C. Logistic regression

D. PCA

Correct Answer: A. DBSCAN

Question 10

In the manufacturing case, why were grouped splits by production batch preferable to random image splits?

A. They guarantee perfect accuracy

B. They test whether the model generalizes across batches and reduce correlated-sample leakage

C. They eliminate the need for feature engineering

D. They ensure every class has equal size

Correct Answer: B. They test whether the model generalizes across batches and reduce correlated-sample leakage