7. Deep Learning Foundations for Vision

Introduction and Context

Computer vision seeks to enable machines to extract useful information from images and video. Earlier chapters in a typical vision curriculum examine image formation, enhancement, segmentation, features, and geometric analysis. Those methods remain valuable, but many modern systems learn visual representations directly from data. Deep learning has therefore changed the dominant workflow: instead of designing all features manually, engineers construct models that learn increasingly abstract representations through multiple computational layers.

A deep vision model may begin with pixel intensities, discover local edges and textures, combine them into parts, and finally represent objects, scenes, or actions. This apparent progression is not guaranteed by hand; it emerges from architecture, data, objective functions, and optimization. Understanding these foundations is essential for selecting a model, diagnosing failure, estimating computational requirements, and judging whether a result is reliable.

This chapter develops the central ideas behind neural networks for vision. It explains convolutional neural networks (CNNs), receptive fields, activation and normalization, regularization and optimization, transfer learning, augmentation, representation learning, and vision transformers. It also considers practical deployment, professional decision-making, limitations, and responsible use.

Learning Objectives

After studying this chapter, a student should be able to:

1. Explain how a neural network learns a mapping from images to predictions.

2. Describe convolution, padding, stride, pooling, feature maps, and receptive fields.

3. Distinguish activation, normalization, optimization, and regularization.

4. Select suitable CNN or transformer-based architectures for different vision tasks.

5. Design a transfer-learning, augmentation, and evaluation strategy.

6. Diagnose overfitting, underfitting, optimization failure, and distribution shift.

7. Assess practical, ethical, and operational risks in deep vision systems.

From Pixels to Learned Representations

A neural network approximates a function y = f(x; θ), where x is an input, y is a target, and θ denotes learnable parameters. In supervised learning, parameters are adjusted to reduce a loss function over labelled examples. For classification, cross-entropy is commonly used:

L = −Σ_k y_k log(p_k),

where y_k is the target indicator and p_k is the predicted probability for class k. The loss is not merely a score reported after training; it supplies a signal for changing the parameters.

A fully connected layer computes z = Wx + b. The output of one layer becomes the input of the next, and a nonlinear activation is applied between layers. Without nonlinearities, a stack of linear layers is equivalent to a single linear transformation and cannot express the rich structure required for vision. Deep learning obtains its representational power from the composition of many transformations.

In image classification, the learned representation often progresses from general to specific. Early layers respond to oriented edges or colour contrasts; intermediate layers combine these responses into textures, contours, or object parts; later layers encode class-relevant arrangements. This is an interpretive tendency rather than a strict rule. Learned features depend on data, architecture, objective, and training procedure.

Essential terminology

A parameter is a value learned from data, such as a convolutional filter weight. A hyperparameter is selected by the practitioner, such as learning rate, batch size, depth, or augmentation strength. An epoch is one complete pass through the training set. A batch is a subset used for one parameter update. A feature map is the spatial output produced by a filter or channel. The logits are unnormalized class scores before the final softmax operation.

Training data are used to fit parameters, validation data to select configurations and monitor generalization, and test data to provide a final estimate. Leakage occurs when information from validation or test data influences training or model selection. It can produce deceptively high performance, especially in medical or industrial datasets containing multiple images from the same subject or machine.

Why Convolution Fits Visual Structure

Images contain strong local correlations: nearby pixels usually have more direct relationships than distant pixels. A convolutional layer exploits this structure by sliding a small kernel across an image. At each location it computes a weighted sum of the local patch, producing a response. The same weights are reused at all locations, which gives convolution two important properties:

• Local connectivity: each response depends on a limited neighbourhood.
• Parameter sharing: the same pattern detector can operate throughout the image.

If an input has height H, width W, kernel size K, padding P, and stride S, the output spatial dimension is commonly

floor((H − K + 2P)/S) + 1.

Padding preserves or controls spatial dimensions, while stride determines how far the kernel moves. A stride greater than one reduces resolution and increases the spatial area represented by subsequent units. Channels allow the layer to process colour bands or learned feature maps. A convolutional layer typically has many filters, each producing one output channel.

Pooling layers, such as maximum or average pooling, summarize local regions. They can reduce computation and provide limited tolerance to small translations, although strided convolution can perform a similar downsampling role while learning the reduction. Excessive downsampling may discard small objects and precise boundaries, so architecture must reflect the task.

Receptive fields and hierarchy

The receptive field of a unit is the region of the original image that can influence it. A 3 × 3 convolution initially has a small receptive field. Stacking two such layers gives a theoretical receptive field of 5 × 5, while also inserting an additional nonlinear transformation. Downsampling expands the rate at which receptive fields grow. Theoretical receptive field and effective receptive field are not identical: in practice, influence is often concentrated near the centre rather than uniformly distributed.

Receptive-field reasoning is important in design. A model detecting a small defect requires sufficient local resolution, whereas scene classification requires broad context. Dilated convolution increases receptive field without proportionally increasing kernel size by inserting gaps between kernel elements. Skip connections, used prominently in residual networks, allow information and gradients to pass more directly across layers. They make very deep networks easier to optimize and support multi-scale representations.

Activations, Normalization, and Information Flow

The rectified linear unit, ReLU(z) = max(0, z), is computationally simple and reduces saturation for positive inputs. Its limitation is the possibility of permanently inactive units when updates place them in the negative region. Variants such as leaky ReLU retain a small negative slope. Smooth alternatives include GELU, frequently used in transformer architectures.

Normalization controls the distribution of intermediate activations. Batch normalization estimates statistics over a mini-batch and maintains running estimates for inference. It can stabilize optimization and permit higher learning rates, but behaviour may become unreliable with very small or highly heterogeneous batches. Layer normalization normalizes across features within an individual example and is common in transformers because it does not depend on batch statistics. Group normalization is often useful for vision tasks with small batches.

Normalization is not a substitute for good data preparation or sound architecture. It may also interact with quantization, domain shift, and transfer learning. Practitioners should distinguish training-mode and inference-mode behaviour and verify that the correct statistics are used during deployment.

Optimization: How Networks Learn

Training usually proceeds through forward propagation, loss computation, backpropagation, and parameter updating. Backpropagation applies the chain rule to calculate the gradient of the loss with respect to every parameter. An optimizer then uses these gradients. Stochastic gradient descent updates parameters approximately as

θ_{t+1} = θ_t − η∇L(θ_t),

where η is the learning rate. Momentum accumulates a fraction of previous updates and can reduce oscillation. Adam adapts step sizes using estimates of first and second gradient moments and is often convenient for initial experiments, although stochastic gradient descent may generalize strongly in some supervised settings.

The learning-rate schedule is frequently more influential than changing optimizers. Warm-up can prevent unstable early updates; cosine decay, step decay, or one-cycle schedules reduce the learning rate as training progresses. Weight decay penalizes large weights and is related to, but not identical in implementation to, L2 regularization. Gradient clipping limits unusually large updates, useful in unstable or recurrent computations.

A learning curve is a diagnostic instrument. High training and validation error suggests underfitting, insufficient capacity, poor optimization, or inadequate features. Low training error with substantially higher validation error indicates overfitting. A widening gap may be addressed through more data, augmentation, regularization, reduced capacity, or early stopping. However, validation performance must be interpreted alongside class imbalance, confidence calibration, and subgroup performance.

Regularization and Data Augmentation

Regularization aims to improve generalization rather than training accuracy. Dropout randomly suppresses activations during training, encouraging distributed representations. Stochastic depth randomly bypasses layers in deep residual models. Weight decay, early stopping, label smoothing, and architectural constraints are other forms of regularization.

Data augmentation creates transformed training examples that preserve the target. Common image operations include horizontal flipping, cropping, resizing, colour perturbation, blur, noise, and small geometric changes. More structured approaches include mixup, which interpolates pairs of images and labels, and CutMix, which combines image regions while adjusting labels. Augmentation must reflect the application: a horizontal flip may be valid for animals but invalid when left-right orientation is clinically meaningful; aggressive colour changes may be harmful in pathology or remote sensing.

Augmentation does not create genuinely new independent subjects. It cannot correct biased sampling, missing classes, or systematic labelling errors. Augmentations should be applied consistently with the task and evaluated through ablation studies rather than adopted merely because they are fashionable.

Transfer Learning and Representation Learning

Training a large model from random initialization requires substantial data and computation. Transfer learning begins with a model trained on a source dataset, then adapts its representation to a target task. A common procedure is to replace the final classification head, freeze much of the backbone, train the new head, and then fine-tune selected or all layers using a smaller learning rate.

Transfer is effective when source and target domains share useful visual structure, but it is not automatic. A model trained on natural photographs may transfer less effectively to microscopy, satellite imagery, thermal cameras, or documents. Domain mismatch can cause negative transfer. Strategies include domain-specific pretraining, gradual unfreezing, discriminative learning rates, and self-supervised pretraining on unlabelled target-domain images.

Representation learning asks the model to discover features useful for prediction or later tasks. Supervised learning shapes representations around labels. Self-supervised methods construct pretext or contrastive objectives from the data itself, such as predicting masked image content or bringing related views closer in embedding space. Foundation vision models can then be adapted through fine-tuning, linear probing, prompting, or parameter-efficient methods. Their generality does not remove the need for domain validation.

CNNs and Vision Transformers

CNNs encode locality and translation-aware weight sharing as architectural inductive biases. They are computationally efficient for many image sizes and often perform well when labelled data are limited. Residual networks, densely connected networks, encoder–decoder models, and feature pyramids illustrate important CNN design patterns. Encoder–decoder architectures are particularly useful for segmentation because the encoder captures semantic context while the decoder restores spatial detail, often with skip connections.

Vision transformers (ViTs) divide an image into patches, convert patches into vectors, add positional information, and process the resulting sequence using self-attention. Self-attention allows a token to interact with distant tokens, making global relationships accessible early in the network. A simplified attention operation is

Attention(Q,K,V) = softmax(QKᵀ/√d)V,

where Q, K, and V are learned query, key, and value representations and d is the key dimension.

Transformers can require substantial data or pretraining because they impose fewer visual assumptions than CNNs. Hierarchical transformers and hybrid CNN-transformer models address computational and multi-scale concerns. The choice between architectures should depend on data volume, image resolution, latency, memory, annotation availability, and the importance of local detail—not on model popularity alone.

A Practical Development Workflow

A responsible project begins with task definition. Classification, detection, segmentation, retrieval, depth estimation, and generative reconstruction require different labels, losses, and evaluation measures. The data split should reflect deployment conditions. If images from one patient, camera, or production batch are correlated, splitting individual images may inflate performance; grouping by source is safer.

The next step is a baseline: a simple model, transparent preprocessing, and a reproducible training configuration. Baselines reveal whether complexity is justified. Evaluation should match operational costs. Accuracy may be inadequate for imbalanced defect detection; precision, recall, F1, area under the precision–recall curve, intersection over union, or calibration measures may be more informative.

Deployment introduces constraints absent from laboratory training. Quantization reduces numerical precision and can reduce memory and latency, but accuracy must be checked. Pruning removes parameters or computation, while knowledge distillation trains a smaller student to imitate a larger teacher. Edge systems require profiling of end-to-end latency, energy, memory, and failure recovery. Monitoring should detect changes in image quality, class frequencies, confidence, and error patterns.

Illustrative Case Study: Visual Inspection of Welds

Background. A manufacturing company uses cameras to inspect welds on metal components. Defects are rare, and lighting differs between production lines.

Situation. A prototype reports 98 percent accuracy, but operators observe missed cracks. Investigation shows that most images are normal and that images from the same production batch appear in both training and validation sets.

Analysis. Accuracy is dominated by the majority class. The split also permits leakage of batch-specific visual cues. Cracks are small, so aggressive resizing and pooling may erase evidence. Lighting variation indicates a domain-shift risk.

Approach. The team creates a group-wise split by production batch, reports recall and precision for defects, and examines a precision–recall curve. It uses a CNN with a feature pyramid and retains higher-resolution features. Transfer learning supplies an initial backbone, while moderate brightness and contrast augmentation model expected illumination changes. Class-weighted loss and targeted sampling address imbalance. Engineers inspect false negatives and use calibrated thresholds rather than accepting the default probability cutoff. A held-out production-line evaluation is conducted before deployment.

Outcome. The revised system has lower overall accuracy but substantially better defect recall and a more credible estimate of performance under deployment conditions. It is deployed as a decision-support tool with human review for uncertain cases, and monitoring is established for camera drift and lighting changes.

Learning points. The case demonstrates that architecture, data splitting, metrics, threshold selection, and operational workflow are inseparable. A higher headline score can represent a worse industrial system when the evaluation objective is misaligned with the cost of failure.

Limitations, Risks, and Professional Responsibilities

Deep models can be sensitive to distribution shift, adversarial perturbations, compression, blur, sensor changes, and rare classes. Saliency visualizations may assist diagnosis but should not be treated as proof of causal reasoning. Confidence scores are not automatically probabilities; calibration must be measured and, where necessary, improved.

Bias can enter through under-represented populations, unequal image quality, proxy labels, or historical decisions embedded in annotations. In high-impact domains, teams should document data provenance, consent and permissible use, annotation procedures, known exclusions, and subgroup performance. Privacy protection may require access control, de-identification, secure storage, and carefully designed retention policies. Copyright and licensing restrictions apply to training data and pretrained models.

Human oversight must be meaningful rather than ceremonial. Users need to know the model’s intended scope, uncertainty, and escalation procedure. Reproducibility requires versioned data, code, configurations, random seeds where feasible, and records of model changes. Governance should include pre-deployment validation, incident reporting, periodic re-evaluation, and a process for withdrawing a model that no longer meets requirements.

Emerging Directions and Best Practices

Current research combines large-scale self-supervised learning, multimodal vision-language models, efficient adaptation, synthetic data, neural compression, and improved robustness evaluation. Video models extend spatial representations with temporal modelling. Promptable segmentation and open-vocabulary detection broaden the set of categories that can be handled without retraining a fixed classifier. These developments increase capability but also raise concerns about computational cost, data ownership, hallucinated outputs, and evaluation beyond benchmark datasets.

Effective practice can be summarized as follows:

• Define the operational decision and its cost before choosing a model.
• Split data by the unit that can cause dependence, not merely by file name.
• Establish a reproducible baseline and inspect errors, not only aggregate scores.
• Match receptive field, resolution, and architecture to the visual evidence required.
• Use augmentation and transfer learning only when justified by the domain.
• Tune thresholds and assess calibration when predictions drive decisions.
• Evaluate robustness, subgroups, latency, memory, and maintenance requirements.
• Treat deployment as a monitored socio-technical system rather than a one-time training exercise.

Chapter Summary

Deep learning for vision learns layered representations through differentiable transformations optimized against a loss function. CNNs exploit locality and parameter sharing, while receptive fields explain how units acquire context. Activations provide nonlinearity; normalization stabilizes information flow; optimization determines how parameters are updated; and regularization improves generalization. Data augmentation, transfer learning, and self-supervised representation learning are central strategies when data or labels are limited. Vision transformers use patch tokens and self-attention to model long-range relationships, offering an alternative to CNN inductive biases. Successful systems require careful data splitting, task-appropriate metrics, error analysis, deployment profiling, monitoring, and ethical governance.

Key Terms

Parameter: A learnable value adjusted during training.

Hyperparameter: A configuration chosen by the practitioner rather than learned directly.

Activation function: A nonlinear transformation applied within a neural network.

Convolution: A local, weight-sharing operation that produces feature responses.

Feature map: A spatial array of responses produced by a filter or channel.

Stride: The spatial step by which a convolutional kernel moves.

Padding: Added border values used to control spatial dimensions.

Receptive field: The input region capable of influencing a particular unit.

Pooling: Local aggregation used to summarize or downsample features.

Backpropagation: Computation of parameter gradients using the chain rule.

Learning rate: The scale of an optimizer’s parameter update.

Normalization: Transformation that controls activation statistics.

Regularization: A method intended to improve generalization and reduce overfitting.

Data augmentation: Label-preserving transformation or synthesis of training examples.

Transfer learning: Adaptation of knowledge learned on one task or domain to another.

Representation learning: Automatic discovery of features useful for prediction or transfer.

Self-supervised learning: Learning from targets constructed from the data itself.

Vision transformer: An image model that processes patch tokens using self-attention.

Distribution shift: A change between training conditions and deployment conditions.

Calibration: Agreement between predicted confidence and observed likelihood.

Review Questions

Short Answer Questions

1. Why are nonlinear activation functions necessary in a deep neural network?

2. Define a receptive field and explain why it matters in vision architecture design.

3. Distinguish training, validation, and test data.

4. What problem does parameter sharing solve in convolutional networks?

5. Why might layer normalization be preferred to batch normalization in a transformer?

Long Answer Questions

1. Explain the operation of a convolutional layer, including kernel, stride, padding, channels, and feature maps.

2. Discuss how backpropagation, gradient descent, learning-rate schedules, and regularization interact during training.

3. Compare CNNs and vision transformers in terms of inductive bias, data requirements, computation, and long-range context.

4. Describe a transfer-learning workflow for a small, domain-specific image dataset and identify possible failure modes.

5. Explain how data leakage, class imbalance, and distribution shift can produce misleading evaluation results.

Analytical / Application Questions

1. A defect occupies only a few pixels, but a proposed model repeatedly misses it after resizing. How would you analyze and redesign the system?

2. A model achieves high validation accuracy but performs poorly on a new camera. Propose an investigation covering data, architecture, metrics, and deployment.

3. Design an evaluation and governance plan for a medical image classifier whose false negatives are more costly than false positives.

22. Multiple Choice Questions with Answers

Question 1

A practitioner observes low training error but substantially higher validation error. Which response most directly addresses the likely problem?

A. Increase model capacity without changing the data

B. Use suitable regularization or augmentation and inspect the data split

C. Remove the validation set

D. Increase the classification threshold only

Correct Answer: B. Use suitable regularization or augmentation and inspect the data split

Question 2

Why does parameter sharing make convolution especially suitable for images?

A. It ensures every image has identical labels

B. It allows a learned local pattern detector to operate at multiple locations

C. It eliminates the need for nonlinear activations

D. It guarantees invariance to every geometric transformation

Correct Answer: B. It allows a learned local pattern detector to operate at multiple locations

Question 3

A small defect disappears after repeated downsampling. Which architectural change is most appropriate?

A. Use higher-resolution or multi-scale features

B. Increase only the final softmax temperature

C. Remove all convolutional layers

D. Use a smaller validation set

Correct Answer: A. Use higher-resolution or multi-scale features

Question 4

Which statement best distinguishes batch normalization from layer normalization?

A. Batch normalization uses batch statistics, whereas layer normalization operates within an individual example

B. Layer normalization can only be used for image labels

C. Batch normalization is an activation function

D. They are identical during both training and inference

Correct Answer: A. Batch normalization uses batch statistics, whereas layer normalization operates within an individual example

Question 5

In the weld-inspection case, why was group-wise splitting by production batch important?

A. It increases the number of classes

B. It prevents correlated batch-specific information from leaking across evaluation sets

C. It guarantees perfect calibration

D. It removes the need for augmentation

Correct Answer: B. It prevents correlated batch-specific information from leaking across evaluation sets

Question 6

What is the principal purpose of a learning-rate schedule?

A. To change the ground-truth labels during training

B. To control the scale of parameter updates over the course of optimization

C. To increase image resolution at inference

D. To replace the loss function

Correct Answer: B. To control the scale of parameter updates over the course of optimization

Question 7

Which situation is most likely to produce negative transfer?

A. The source and target domains share useful visual structure

B. A natural-image model is applied to a substantially different sensor domain without validation

C. A new classification head is added to a pretrained model

D. The backbone is fine-tuned using a smaller learning rate

Correct Answer: B. A natural-image model is applied to a substantially different sensor domain without validation

Question 8

What is a defining mechanism of a vision transformer?

A. It processes image patches as tokens and uses self-attention

B. It cannot represent relationships between distant regions

C. It uses only fixed, hand-designed edge detectors

D. It requires pooling after every layer

Correct Answer: A. It processes image patches as tokens and uses self-attention

Question 9

A classifier has high accuracy because 98% of examples are normal, but it misses many defects. Which metric is particularly important to examine?

A. Defect-class recall and the precision–recall relationship

B. Only the total number of model parameters

C. Training image file size

D. The mean pixel value alone

Correct Answer: A. Defect-class recall and the precision–recall relationship

Question 10

Which practice best supports responsible deployment of a high-impact vision model?

A. Deploy once benchmark accuracy is high and stop evaluation

B. Hide uncertainty from users to prevent hesitation

C. Document data provenance, monitor drift, assess subgroups, and provide human escalation

D. Use the same decision threshold in every operating environment without testing

Correct Answer: C. Document data provenance, monitor drift, assess subgroups, and provide human escalation