Image Classification and Representation Learning

Introduction and Context

Image classification assigns an image to one or more semantic categories. A system may classify a radiograph as showing pneumonia, identify a vehicle type, or recognize plant disease. Modern classification is not merely a matter of selecting a final label: it requires learning useful representations—numerical features that preserve information relevant to the task while suppressing irrelevant variation such as lighting, position, scale, and background.

Earlier computer-vision systems depended on hand-designed descriptors such as SIFT, HOG, and color histograms followed by classifiers such as support vector machines. Convolutional neural networks (CNNs) transformed the field by learning feature extractors and classifiers jointly from data. More recently, vision transformers (ViTs) have modeled relationships between image patches using attention. Self-supervised learning, pretrained models, metric learning, and interpretability methods extend classification beyond conventional supervised training.

This chapter develops the concepts and practical design decisions needed to build reliable classification systems.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Explain image representations, classification objectives, and the role of inductive bias.

2. Describe CNN and vision-transformer architectures and compare their strengths.

3. Train, fine-tune, and evaluate pretrained image models appropriately.

4. Explain self-supervised learning, embeddings, and metric-learning objectives.

5. Diagnose overfitting, data leakage, class imbalance, and distribution shift.

6. Apply interpretability techniques critically rather than treating visual explanations as proof of correctness.

7. Design an ethically responsible image-classification pipeline.

1. The Classification Problem

Let an image be represented by x and its class label by y. In a K-class single-label problem, y belongs to {1, ..., K}. A model produces a vector of scores or logits z(x), one for each class. The softmax function converts these scores into probabilities:

p(y=k|x) = exp(z_k) / sum_{j=1}^{K} exp(z_j).

The probabilities are nonnegative and sum to one. The predicted class is usually the class with maximum probability. For a training example with true class y, the cross-entropy loss is

L = -log p(y|x).

For a dataset of N examples, the empirical objective is the mean loss, (1/N) sum_i L_i. Cross-entropy strongly penalizes assigning low probability to the correct class. It is appropriate when labels are treated as categorical observations and the model is trained by maximum likelihood. A probability of 0.9 for the correct class gives a loss of approximately 0.105, whereas a probability of 0.1 gives approximately 2.303.

In multilabel classification, several labels may be correct simultaneously. The model therefore uses one sigmoid output per label rather than a single softmax distribution. Binary cross-entropy is applied independently to each label. Choosing softmax for a multilabel task incorrectly forces class probabilities to compete.

A classifier is a mapping from images to decisions, but a representation learner maps an image to an embedding h(x) in a feature space. A linear or nonlinear prediction head can then use that embedding for classification, retrieval, clustering, or anomaly detection.

2. What Makes a Good Representation?

A useful representation should have:

• Discriminative power: images from different relevant classes should be separable.
• Invariance: irrelevant changes, such as moderate translation or illumination variation, should not alter the representation excessively.
• Equivariance where needed: spatial changes may need to be reflected predictably in intermediate features, especially for detection and segmentation.
• Compactness: the representation should avoid storing unnecessary detail.
• Transferability: features should support related tasks and domains.

These goals can conflict. Removing all sensitivity to position helps object recognition but harms tasks in which location is meaningful. A model trained to recognize species may learn background or camera-specific artifacts rather than biological characteristics. Consequently, representation quality must be judged in relation to the intended deployment task.

Inductive Bias

An inductive bias is a preference that helps a learner generalize beyond observed examples. CNNs encode local connectivity and translation-related structure. Vision transformers have weaker built-in locality but can learn long-range relationships through attention. Data augmentation, pretrained weights, architectural design, and regularization all introduce useful biases.

3. Convolutional Neural Networks

A two-dimensional convolution applies a learned kernel to local image neighborhoods. In practice, deep-learning libraries often implement cross-correlation rather than mathematical kernel reversal, but the operation is conventionally called convolution. For input feature map X and kernel W, a simplified output at location (i,j) is

Y(i,j) = sum_{u,v,c} W(u,v,c) X(i+u,j+v,c) + b,

where u and v index the kernel, c indexes input channels, and b is a bias. The output has multiple channels, each produced by a different learned kernel.

Important architectural concepts include:

• Stride: the step between neighboring kernel applications; larger stride reduces spatial resolution.
• Padding: additional border values that control output size.
• Receptive field: the region of the original image influencing a unit.
• Pooling: spatial aggregation, commonly by maximum or average, that reduces resolution.
• Normalization: operations such as batch normalization that stabilize optimization.
• Residual connection: addition of a block input to its transformed output, helping information and gradients propagate through deep networks.

If an input dimension is n, kernel size is f, padding is p on each side, and stride is s, the output dimension is

floor((n + 2p - f)/s) + 1.

This formula assumes a regular convolution and integer indexing. For example, an input of 32 pixels, a 3-pixel kernel, padding 1, and stride 1 produces floor((32+2-3)/1)+1 = 32 pixels. With stride 2, it produces 16 pixels.

A typical CNN consists of early layers that detect edges and textures, middle layers that combine local patterns into parts, and later layers that represent object-level structure. This progression is not guaranteed to be semantically clean, but it is a useful interpretive model.

CNN Design Families

LeNet established the practical value of convolutional networks for digit recognition. AlexNet demonstrated large-scale supervised learning with deeper networks, nonlinearities, data augmentation, and GPU computation. VGG emphasized repeated small convolutions. Inception used parallel filters of different effective scales. ResNet introduced residual blocks and enabled much deeper networks. EfficientNet systematically balanced network depth, width, and input resolution. Mobile-oriented networks use depthwise separable convolutions to reduce computation, making deployment on phones and embedded devices more feasible.

4. Vision Transformers

A vision transformer divides an image into fixed-size patches. Each flattened patch is projected into a vector, positional information is added, and the resulting sequence is processed by transformer encoder blocks. Self-attention allows each token to interact with other tokens.

For query matrix Q, key matrix K, and value matrix V, scaled dot-product attention is

Attention(Q,K,V) = softmax(QK^T / sqrt(d_k))V,

where d_k is the key-vector dimension. Scaling reduces the magnitude of dot products when the dimension is large, helping maintain useful softmax gradients.

A transformer encoder typically contains multi-head self-attention, a position-wise multilayer perceptron, residual connections, and normalization. A classification token or pooled token representation is passed to a classification head.

Compared with CNNs, ViTs can model global relationships directly and often scale effectively with large datasets and pretraining. However, they may require more data or stronger pretraining because they encode less locality by default. Hybrid architectures combine convolutional stems with attention. Hierarchical transformers, such as window-based models, reduce computational cost and create multiscale representations.

The attention mechanism has a quadratic interaction cost in the number of tokens in its standard form. Smaller patches increase the number of tokens and computational demand. Thus, image resolution, patch size, memory, and latency must be considered together.

5. Training, Transfer Learning, and Fine-Tuning

Training from random initialization is appropriate when a large, representative labeled dataset is available. In many practical settings, labels are expensive and the target dataset is small. Transfer learning begins with a model pretrained on a broad dataset and adapts it to the target task.

Common strategies are:

1. Feature extraction: freeze the backbone and train a new classification head.

2. Partial fine-tuning: unfreeze later layers while keeping early layers fixed.

3. Full fine-tuning: update the entire network using a relatively small learning rate.

4. Parameter-efficient adaptation: update a small set of additional or selected parameters.

Pretraining and target data must be examined for overlap. If test images or near-duplicates influenced pretraining, evaluation may be optimistic. Domain mismatch also matters: features learned from natural photographs may transfer imperfectly to microscopy or satellite imagery.

A sound workflow separates training, validation, and test data before model development. The validation set supports hyperparameter and threshold selection; the test set should be used only for final evaluation. Augmentation must be applied without allowing information from a future or related image to leak into training. For medical, industrial, or video data, splitting by patient, machine, or scene is often more appropriate than random image-level splitting.

Useful regularization methods include weight decay, dropout, early stopping, label smoothing, data augmentation, and mixup. Augmentations must preserve the label. Horizontal flipping may be valid for many animals but invalid where left-right orientation is clinically meaningful.

6. Evaluation and Calibration

Accuracy is the fraction of correct predictions and is suitable when classes and error costs are balanced. For binary classification, precision is TP/(TP+FP), while recall is TP/(TP+FN). The F1 score is the harmonic mean of precision and recall:

F1 = 2PR/(P+R),

provided P and R are defined and their sum is nonzero. A confusion matrix reveals which classes are confused. Macro-averaging gives each class equal weight; micro-averaging aggregates decisions and gives greater influence to frequent classes.

Top-k accuracy counts a prediction as correct when the true class is among the k highest-scoring classes. It can be useful in image search or human-in-the-loop systems but may be inappropriate when an automated system must make one definitive decision.

Receiver operating characteristic and precision-recall curves examine threshold behavior in binary or one-versus-rest settings. Area under a curve summarizes ranking performance, but it does not replace selecting an operating threshold based on actual costs.

A model is calibrated when predicted probabilities correspond approximately to observed frequencies. A group of predictions assigned probability 0.8 should be correct about 80% of the time under comparable conditions. Temperature scaling can improve calibration using a validation set, but calibration may deteriorate under distribution shift.

7. Self-Supervised Representation Learning

Self-supervised learning creates supervisory signals from the data itself, reducing reliance on manual labels. A pretext task may predict masked image content, discriminate transformed views, or reconstruct portions of an image. The learned encoder is later fine-tuned or evaluated with a linear probe.

Contrastive learning creates positive pairs, often two augmentations of the same image, and treats other images as negatives. A common objective encourages the positive pair to have high similarity and negatives to have lower similarity. The temperature parameter controls the sharpness of the similarity distribution. The exact usefulness of the objective depends heavily on augmentation choices, batch composition, and the definition of positive and negative pairs.

Non-contrastive methods avoid explicit negative examples by using mechanisms such as asymmetric predictors, stop-gradient operations, or teacher-student networks. Masked-image modeling hides patches and trains a model to infer missing information. These approaches may learn general features, but a pretraining objective can also emphasize details irrelevant to a downstream application.

Evaluation should include transfer performance, robustness, class efficiency, and domain relevance—not merely pretext-task accuracy.

8. Embeddings and Metric Learning

An embedding is a vector representation designed so that geometric relationships reflect semantic relationships. Similarity may be measured using a dot product or cosine similarity:

cos(a,b) = (a·b)/(||a|| ||b||),

for nonzero vectors a and b. Euclidean distance is another option. Feature normalization affects the interpretation of these measures.

Metric learning explicitly shapes the embedding space. In contrastive loss, a pair label indicates whether two examples are similar. In triplet learning, an anchor a, positive p, and negative n are used with a margin m:

L = max(0, d(a,p) - d(a,n) + m).

The loss is zero when the negative is sufficiently farther from the anchor than the positive by at least m. The choice of distance, margin, and mining strategy is important. Random negatives may be too easy; hard negatives may be informative but can also be mislabeled or destabilize training.

Embeddings support nearest-neighbor search, image retrieval, few-shot classification, duplicate detection, product recommendation, and open-set recognition. An embedding model should be evaluated on the retrieval or decision task of interest, not assumed to be universally semantic.

9. Interpretability and Model Diagnosis

Interpretability methods help investigate why a model produced a prediction. Saliency methods estimate sensitivity of the output to pixels. Grad-CAM uses gradients flowing into convolutional feature maps to produce a coarse localization map. Occlusion tests the effect of masking image regions. Integrated gradients accumulate attribution along a path from a baseline to the input.

These methods answer different questions and have limitations. A highlighted region may indicate correlation rather than causal evidence. Explanations can be unstable, resolution-limited, or affected by the baseline and visualization settings. A visually plausible heat map does not prove that the classifier used the medically or scientifically correct evidence.

Good practice includes testing explanation stability, comparing multiple methods, using counterfactual edits where feasible, and validating important findings with domain experts. Interpretability should complement quantitative error analysis, subgroup evaluation, and data auditing.

10. Worked Design Example: Wildlife Camera Classification

Consider a system that classifies camera-trap images into deer, fox, domestic dog, human, and empty-scene classes.

Problem analysis. Images contain severe class imbalance, night-time noise, motion blur, and correlated sequences from the same camera. A random image split could place near-identical frames in both training and test sets.

Approach. First, split by geographic site and time period. Inspect labels and create an explicit empty-scene class. Begin with a pretrained ResNet or vision transformer. Replace its classification head, train the head while the backbone is frozen, and then fine-tune selected layers with a lower learning rate. Use crops, mild brightness variation, and horizontal flips only after verifying that these transformations preserve class meaning.

Evaluation. Report per-class recall, macro-F1, the confusion matrix, and calibration. Dog-versus-fox errors may be more important than empty-scene errors if the system triggers ecological alerts. Evaluate separately for day and night, camera location, and weather.

Interpretability. Use Grad-CAM or occlusion to check whether the model attends to animals rather than timestamps, watermarks, or camera housings. If the model relies on a location-specific background, collect examples from new sites and retrain or rebalance the data.

Outcome. A successful system is not simply the one with the highest overall accuracy. It is the one that maintains acceptable error rates on new sites, communicates uncertainty, and supports ecological decisions without hiding systematic failure.

Learning points. Deployment conditions must shape data splitting, augmentation, evaluation, and monitoring. Representation learning improves performance only when the learned representation reflects the real task rather than shortcuts.

11. Challenges, Governance, and Best Practices

Important technical challenges include insufficient labels, noisy annotations, imbalance, domain shift, adversarial or natural corruptions, computational cost, and uncertain predictions. Models may fail on image formats, sensors, populations, or environments absent from training data.

Responsible development requires documenting data sources, consent and licensing, annotation procedures, known limitations, and intended use. Facial or biometric classification raises heightened privacy and civil-rights concerns. Medical and employment applications require domain-specific governance, security, auditability, and human oversight. Training data may encode demographic or geographic bias; subgroup performance should therefore be measured where legally and ethically appropriate.

Recommended practices are:

• define the operational decision and error costs before selecting metrics;
• prevent leakage through entity- or time-based splitting;
• establish a simple baseline before adopting a complex architecture;
• use pretrained models while checking licensing, provenance, and domain mismatch;
• tune thresholds on validation data, not the test set;
• report uncertainty, subgroup results, and failure cases;
• monitor post-deployment drift and create a retraining policy;
• retain human review for high-impact or ambiguous decisions.

Contemporary Developments

Current research combines large-scale image-text pretraining, multimodal embeddings, parameter-efficient adaptation, synthetic data, and efficient attention. Vision-language models can support zero-shot or prompt-based classification, but their outputs may be sensitive to wording and may inherit biases from web-scale data. Foundation models can reduce task-specific labeling requirements, yet they require careful domain validation and governance.

Research increasingly emphasizes robustness, data-centric improvement, uncertainty estimation, causal and concept-based explanations, energy efficiency, and evaluation beyond static benchmark accuracy. These developments reinforce a central principle: architecture matters, but data quality, task definition, and deployment context often matter more.

Chapter Summary

Image classification maps images to labels, while representation learning seeks feature spaces that preserve task-relevant structure. CNNs use locality and hierarchical feature extraction; vision transformers use patch tokens and self-attention to model global relationships. Pretrained models can reduce data and computation requirements, but transfer must account for domain mismatch and leakage.

Reliable systems require appropriate objectives, augmentation, data splitting, metrics, calibration, and error analysis. Self-supervised learning obtains representations from unlabeled data, while metric learning shapes distances for retrieval and few-shot applications. Interpretability methods can assist diagnosis but should not be treated as causal proof. Ethical deployment requires attention to privacy, bias, documentation, uncertainty, and human oversight.

Key Terms

Representation learning: Learning features that encode useful structure from data.

Embedding: A vector representation of an input in a learned feature space.

CNN: A neural network using convolutional operations and spatial inductive biases.

Vision transformer: An image model that processes image patches with transformer attention.

Logit: An unnormalized model score before conversion to probability.

Softmax: A function converting class scores into a probability distribution.

Cross-entropy: A loss that penalizes incorrect probability assignments.

Receptive field: The input region influencing a unit in a network.

Residual connection: A shortcut that adds an input to a transformed output.

Transfer learning: Reusing knowledge learned on one task or dataset.

Fine-tuning: Updating pretrained parameters on a target task.

Self-supervised learning: Learning from automatically constructed signals in unlabeled data.

Contrastive learning: Learning by bringing positive examples together and separating negatives.

Metric learning: Learning distances or similarities in an embedding space.

Data leakage: Unintended use of information from evaluation data during development.

Calibration: Agreement between predicted probabilities and observed frequencies.

Distribution shift: A change between training and deployment data distributions.

Grad-CAM: A gradient-based method for coarse visual attribution.

Macro-F1: The average of per-class F1 scores.

Open-set recognition: Handling inputs belonging to classes not represented during training.

Review Questions

Short Answer Questions

1. Why does a multilabel problem generally use sigmoid outputs rather than softmax?

2. Define transfer learning and distinguish feature extraction from fine-tuning.

3. What is the purpose of positional information in a vision transformer?

4. State two reasons why random image-level splitting can cause leakage.

5. What does calibration mean for a classifier?

Long Answer Questions

1. Compare CNNs and vision transformers in terms of inductive bias, data requirements, computational cost, and global context.

2. Explain how self-supervised learning can produce transferable image representations, including the role of augmentations.

3. Describe a complete workflow for training and evaluating a pretrained classifier on an imbalanced dataset.

4. Explain metric learning with a triplet-loss formulation and discuss hard-negative mining.

5. Discuss the strengths and limitations of saliency maps, Grad-CAM, and occlusion analysis.

Analytical / Application Questions

1. A wildlife classifier has high accuracy but poor recall for a rare species. Which data, metric, threshold, and sampling changes would you investigate, and why?

2. A medical image model performs well on one hospital’s test set but poorly at a second hospital. Analyze likely causes and propose a validation and adaptation plan.

3. Design an ethically responsible image-classification pipeline for a high-impact application. Specify data governance, model evaluation, interpretability, human oversight, and monitoring requirements.

Multiple Choice Questions with Answers

Question 1

Which output formulation is most appropriate for a single-label K-class classification problem?

A. Independent sigmoid outputs whose values need not sum to one

B. A softmax distribution over K classes

C. A single linear output with no decision rule

D. A reconstruction vector with one value per pixel

Correct Answer: B. A softmax distribution over K classes

Question 2

For an input dimension of 32, kernel size 3, padding 1, and stride 2, what is the convolution output dimension?

A. 14

B. 15

C. 16

D. 32

Correct Answer: C. 16

Question 3

What is a principal advantage of residual connections?

A. They remove the need for labels

B. They guarantee perfect calibration

C. They help information and gradients propagate through deep networks

D. They make every layer spatially larger

Correct Answer: C. They help information and gradients propagate through deep networks

Question 4

In standard self-attention, why is QK^T divided by the square root of the key dimension?

A. To reduce the magnitude of dot products and stabilize softmax behavior

B. To convert image patches into RGB values

C. To enforce that all attention weights are equal

D. To eliminate positional information

Correct Answer: A. To reduce the magnitude of dot products and stabilize softmax behavior

Question 5

Which practice best prevents leakage when multiple images come from the same patient?

A. Randomly assign every image independently

B. Split data by patient before training

C. Tune the test-set threshold repeatedly

D. Duplicate rare images across all splits

Correct Answer: B. Split data by patient before training

Question 6

What does a calibrated predicted probability of 0.8 approximately mean?

A. The model is correct on exactly 80% of every individual prediction

B. The image contains 80% foreground pixels

C. Among comparable predictions assigned 0.8, about 80% are correct

D. The model has 80 classes

Correct Answer: C. Among comparable predictions assigned 0.8, about 80% are correct

Question 7

In triplet loss, when is the loss zero?

A. When the negative is sufficiently farther from the anchor than the positive by the margin

B. Whenever the anchor and positive are identical in pixel space

C. Only when all embeddings have norm zero

D. When the classifier uses softmax

Correct Answer: A. When the negative is sufficiently farther from the anchor than the positive by the margin

Question 8

Which statement about Grad-CAM is most accurate?

A. It proves the highlighted region caused the prediction

B. It provides a coarse gradient-based attribution map

C. It replaces the need for quantitative evaluation

D. It can only be applied to raw pixel classifiers

Correct Answer: B. It provides a coarse gradient-based attribution map

Question 9

Why can macro-F1 be informative for imbalanced multiclass data?

A. It gives every class equal influence by averaging per-class F1 scores

B. It ignores all minority classes

C. It is identical to accuracy in every case

D. It measures only calibration

Correct Answer: A. It gives every class equal influence by averaging per-class F1 scores

Question 10

Which situation most strongly suggests distribution shift?

A. Training and deployment images come from identical sources and conditions

B. A model is evaluated once on its validation set

C. Deployment images come from a different hospital and sensor population

D. The model uses a residual block

Correct Answer: C. Deployment images come from a different hospital and sensor population