Computer Vision with Convolutional Networks

Introduction and Context

Computer vision enables machines to interpret images and video. Applications include medical diagnosis, autonomous driving, industrial inspection, retail analytics, biometric systems, agriculture, and remote sensing. Although traditional computer-vision systems depend heavily on manually designed features, convolutional neural networks (CNNs) learn useful visual representations directly from data.

A modern vision system normally contains four stages: data acquisition, preprocessing and annotation, model training, and deployment with monitoring. A successful system is not determined only by classification accuracy. It must also be robust to changes in lighting, viewpoint, camera quality, and population; efficient enough for its hardware; and appropriate for the risks of its application.

This chapter develops the principles and practical implementation of CNNs for image classification, object detection, and semantic segmentation using TensorFlow and PyTorch. It also examines augmentation, transfer learning, pretrained architectures, evaluation, efficiency, robustness, and responsible deployment.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain how convolution, padding, stride, pooling, and receptive fields support visual representation learning.

2. Calculate the spatial dimensions and parameter count of common CNN layers.

3. Build and train an image classifier in TensorFlow or PyTorch.

4. Distinguish classification, object detection, semantic segmentation, and instance segmentation.

5. Apply data augmentation, transfer learning, and pretrained architectures appropriately.

6. Evaluate accuracy, precision, recall, IoU, mAP, calibration, robustness, and computational efficiency.

7. Identify ethical, safety, privacy, and deployment issues in computer-vision systems.

1. Images as Structured Data

A digital image is represented as a tensor. A colour image commonly has height H, width W, and three channels for red, green, and blue. Pixel values may be stored as integers from 0 to 255 or normalized to a range such as [0,1]. A batch of images is represented as either (N,H,W,C), used commonly by TensorFlow, or (N,C,H,W), used commonly by PyTorch.

Images contain strong local structure. Nearby pixels are usually more related than distant pixels, and the same visual pattern may occur at different positions. CNNs exploit these properties through local connectivity and weight sharing. Unlike a fully connected network, which assigns an independent weight to every input-output pair, a convolutional layer applies the same small set of filters across the image.

Important terminology includes:

• Feature map: The output produced by a filter or channel.
• Kernel or filter: A learned tensor that detects a local pattern.
• Channel: A separate image or feature dimension, such as red, green, blue, or a learned feature channel.
• Activation: The output of a layer after applying a nonlinear function.
• Receptive field: The region of the original image that can influence a particular activation.

2. The Convolutional Network

2.1 Convolution

For a two-dimensional single-channel cross-correlation operation, the output at position (i,j) can be written as

Y(i,j) = b + Σu=0 to Kh−1 Σv=0 to Kw−1 X(i+u,j+v)K(u,v),

where X is the input, K is a kernel of height Kh and width Kw, b is a bias, and Y is the output. Deep-learning libraries generally call this operation convolution even though the kernel is not mathematically flipped as in classical signal processing.

For an input with Cin channels and an output with Cout filters, the parameter count is

Kh × Kw × Cin × Cout + Cout.

The final term represents one bias per output channel. This formula assumes a standard dense convolution, not grouped or depthwise convolution.

A convolution layer has three important geometric settings:

• Stride: The number of pixels by which the filter moves. Larger stride reduces spatial resolution.
• Padding: Extra pixels added around an input. “Valid” padding adds none; “same” padding is selected to preserve spatial dimensions when stride is one.
• Dilation: Spacing between kernel elements. Dilation enlarges the receptive field without proportionally increasing the number of parameters.

For input size H × W, kernel size K, padding P on each side, stride S, and dilation D, the output height is

Hout = floor((H + 2P − D(K−1) − 1)/S + 1),

with an analogous expression for width. The formula assumes symmetric padding and a regular convolution.

Worked Example 1: Output Size and Parameters

Suppose an input is 32 × 32 × 3 and a layer uses 16 filters of size 3 × 3, stride 1, and padding 1. The output spatial size is

floor((32 + 2(1) − 1(3−1) − 1)/1 + 1) = 32.

Thus the output is 32 × 32 × 16. The parameter count is

3 × 3 × 3 × 16 + 16 = 448.

The network learns 448 parameters, not 32 × 32 × 16 independent sets of weights, because each filter is shared over all positions.

2.2 Nonlinearity and Pooling

After convolution, a nonlinear activation is needed. ReLU is defined as

ReLU(x) = max(0,x).

It is computationally simple and reduces the optimization problems associated with repeatedly composing linear operations. Variants such as Leaky ReLU, GELU, and SiLU may improve optimization in particular architectures.

Pooling reduces spatial resolution and can provide limited translation tolerance. Max pooling selects the largest value in a local window, while average pooling computes the mean. Modern architectures often use strided convolutions or global average pooling instead of extensive pooling. Global average pooling averages each final feature map over its spatial positions and produces one value per channel, reducing parameters compared with a large fully connected layer.

2.3 Hierarchical Representation Learning

Early layers often learn edges, colour contrasts, and simple textures. Intermediate layers combine these into motifs such as corners, curves, or repeated patterns. Deeper layers represent object parts and category-specific configurations. This hierarchy is not guaranteed to correspond perfectly to human concepts, but it is a useful interpretation of learned representations.

Residual connections, introduced in architectures such as ResNet, allow a block to learn a residual transformation:

Y = F(X) + X,

provided X and F(X) have compatible dimensions. If dimensions differ, a projection, often a 1 × 1 convolution, is used. Residual connections improve gradient flow and make very deep networks easier to optimize.

Batch normalization normalizes activations using statistics estimated from a mini-batch during training and stored running statistics during inference. It can stabilize optimization, although its behaviour may become unreliable with very small batches or strong domain shifts. Layer normalization, group normalization, and careful initialization are alternatives in some settings.

3. The Main Computer-Vision Tasks

3.1 Image Classification

Classification assigns one or more labels to an image. In single-label, multi-class classification, exactly one class is usually selected. The final layer produces logits z1,...,zC, converted to probabilities by softmax:

p(c|x) = exp(zc) / Σk exp(zk).

The cross-entropy loss for a target class y is

L = −log p(y|x).

This loss assumes mutually exclusive classes. For multi-label classification, where several labels may be true simultaneously, independent sigmoid outputs and binary cross-entropy are generally more appropriate.

3.2 Object Detection

Detection identifies both what objects are present and where they occur. A prediction normally contains a class, confidence score, and bounding box. Two-stage detectors, such as Faster R-CNN, first generate or propose regions and then classify and refine them. One-stage detectors, such as SSD and YOLO-family models, predict boxes and classes directly and generally offer lower latency.

Intersection over Union (IoU) measures overlap between predicted box Bp and ground-truth box Bg:

IoU = area(Bp ∩ Bg) / area(Bp ∪ Bg).

IoU ranges from 0 to 1. A detection is often considered correct when its class is correct and IoU exceeds a selected threshold, such as 0.5. The threshold must be reported because changing it changes measured performance.

Non-maximum suppression (NMS) removes redundant predictions. The highest-scoring box is retained, and boxes with excessive IoU with it are suppressed, usually within the same class. NMS can fail when objects overlap heavily, so soft-NMS or learned alternatives may be useful.

3.3 Segmentation

Semantic segmentation assigns a class to every pixel, while instance segmentation additionally distinguishes separate objects belonging to the same class. Panoptic segmentation combines semantic and instance interpretations.

A common segmentation metric is mean IoU. For class c,

IoUc = TPc / (TPc + FPc + FNc),

where TP, FP, and FN are pixel-level true positives, false positives, and false negatives. Mean IoU is the arithmetic mean across classes, often excluding a designated ignore class. Dice similarity is

Dice = 2TP / (2TP + FP + FN).

Dice and IoU are related but not identical; the choice should be consistent across experiments.

4. Training a CNN

A reliable training pipeline begins with a clean split into training, validation, and test data. Splitting must occur at the appropriate independent unit. For example, images from the same patient, video, or machine should not be scattered across all splits, because this causes leakage and inflated performance.

Typical steps are:

1. Decode images and verify labels.

2. Resize or crop according to the architecture and task.

3. Normalize using statistics appropriate to the model, especially when using pretrained weights.

4. Apply training-only augmentation.

5. Construct mini-batches.

6. Compute predictions and loss.

7. Backpropagate gradients and update parameters.

8. Evaluate on validation data without random training augmentation.

9. Select the model using validation performance and report final test performance once.

The optimizer updates parameters using gradients of the training loss. Adam adapts a learning rate for each parameter, while stochastic gradient descent with momentum often provides strong generalization when well tuned. Learning-rate schedules, weight decay, early stopping, and checkpointing are standard practices.

Overfitting occurs when training performance improves while validation performance stagnates or declines. Remedies include more data, augmentation, weight decay, dropout, reduced model capacity, label correction, and transfer learning. Underfitting may require a larger model, longer training, better features, or reduced regularization.

Minimal PyTorch Classifier

```python

import torch

from torch import nn

class SmallCNN(nn.Module):

def __init__(self, classes=10):

super().__init__()

self.features = nn.Sequential(

nn.Conv2d(3, 32, 3, padding=1), nn.ReLU(),

nn.MaxPool2d(2),

nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(),

nn.AdaptiveAvgPool2d((1, 1))

)

self.classifier = nn.Linear(64, classes)

def forward(self, x):

x = self.features(x)

return self.classifier(x.flatten(1))

model = SmallCNN(classes=10)

loss_fn = nn.CrossEntropyLoss()

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

```

`CrossEntropyLoss` expects class-index targets and internally combines a log-softmax operation with negative log likelihood. Therefore, raw logits should be passed to it rather than probabilities produced by a separate softmax.

TensorFlow Equivalent Pattern

```python

import tensorflow as tf

model = tf.keras.Sequential([

tf.keras.layers.Input((128, 128, 3)),

tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu'),

tf.keras.layers.MaxPooling2D(),

tf.keras.layers.Conv2D(64, 3, padding='same', activation='relu'),

tf.keras.layers.GlobalAveragePooling2D(),

tf.keras.layers.Dense(5)

])

model.compile(

optimizer=tf.keras.optimizers.Adam(1e-3),

loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),

metrics=['accuracy']

)

```

The label format and `from_logits` setting must agree with the final layer. A mismatch is a frequent implementation error.

5. Data Augmentation and Robustness

Data augmentation creates altered training examples that preserve the intended label. Common transformations include random crops, horizontal flips, small rotations, colour jitter, scaling, blur, and random erasing. Augmentation should reflect plausible variation in deployment. A vertical flip may be valid for some satellite images but harmful for traffic signs or medical images.

Augmentation can be applied at the image level, in feature space, or through approaches such as MixUp and CutMix. MixUp forms a weighted combination of two images and their labels; it can improve calibration and reduce memorization, but may be unsuitable when mixed images have physically meaningless interpretations.

Robustness should be evaluated using realistic distribution shifts: lighting changes, compression, blur, sensor changes, weather, demographic variation, and unseen environments. A model that performs well on an identically distributed test set may still fail in production. Stress tests should be designed before deployment rather than after a failure occurs.

Calibration measures whether predicted probabilities correspond to observed frequencies. A model that says “90% confidence” should be correct approximately 90% of the time among such predictions. Reliability diagrams, expected calibration error, temperature scaling, and class-specific analysis are useful tools. Accuracy alone does not establish trustworthy confidence.

6. Transfer Learning and Pretrained Architectures

Training a deep vision model from random initialization requires substantial labelled data and computation. Transfer learning begins with a model trained on a large source dataset and adapts it to a target task.

Two common strategies are:

• Feature extraction: Freeze most or all pretrained layers and train a new task-specific head.
• Fine-tuning: Unfreeze selected layers and continue training with a smaller learning rate.

A typical procedure is to replace the final classifier, train the new head while the backbone is frozen, then gradually unfreeze later layers. Batch-normalization statistics require special care, especially with small target datasets. The source and target domains should be compared; pretraining on natural images may transfer less effectively to specialised medical or scientific imagery than to ordinary photographs.

Important architectures include:

• LeNet-style networks: Early demonstrations of convolution for character recognition.
• AlexNet: Demonstrated the effectiveness of deep CNNs, GPUs, ReLU activations, and large-scale data.
• VGG: Used simple repeated 3 × 3 convolutions but was computationally expensive.
• Inception: Used multiple filter scales and factorized operations for efficiency.
• ResNet: Used residual connections to support very deep networks.
• MobileNet: Used depthwise separable convolutions for mobile deployment.
• EfficientNet: Balanced network depth, width, and input resolution through compound scaling.
• Vision Transformer: Uses tokenized image patches and attention; it is not a CNN, but is an important contemporary alternative.

A depthwise separable convolution divides a standard convolution into a depthwise spatial convolution followed by a pointwise 1 × 1 convolution. For kernel size K, input channels Cin, and output channels Cout, its parameter count is approximately

K²Cin + CinCout,

compared with K²CinCout for a standard convolution, excluding biases. The reduction is substantial when both channel counts are large, although actual latency also depends on hardware and implementation.

7. Evaluation and Model Selection

For classification, report a confusion matrix and metrics appropriate to the problem. Accuracy is the fraction of correct predictions, but it can be misleading with class imbalance. Precision and recall for a selected positive class are

Precision = TP/(TP+FP),

Recall = TP/(TP+FN),

provided the denominators are nonzero. The F1 score is the harmonic mean:

F1 = 2(Precision × Recall)/(Precision + Recall).

Macro-averaging gives equal weight to classes, whereas micro-averaging aggregates decisions before calculating a metric. Balanced accuracy averages recall across classes and is often useful for imbalanced classification.

For detection, precision-recall curves and mean average precision (mAP) are standard. mAP depends on the IoU criterion, class averaging, confidence ranking, and treatment of difficult or ignored objects. Therefore, a statement such as “mAP is 0.72” is incomplete without its evaluation protocol.

For segmentation, report mean IoU or Dice, per-class scores, and boundary quality where boundaries matter. Small objects may contribute little to aggregate pixel metrics, so object-level and size-stratified analyses are valuable.

Performance must also include latency, throughput, memory, energy use, and model size. FLOPs are a theoretical operation estimate and do not directly equal runtime. Benchmark the complete pipeline on target hardware, including preprocessing and postprocessing.

8. Applied Case: Illustrative Industrial Surface Inspection

Background

A manufacturer produces metal panels and wishes to detect scratches, dents, and coating defects using an overhead camera. Manual inspection is slow and inconsistent, while missed defects can lead to expensive downstream failures.

Problem

The company has 18,000 labelled images from two production lines. Defects are rare, lighting differs between lines, and some scratches are only a few pixels wide. A random image split would be risky because consecutive frames from the same production batch are highly similar.

Analysis

The team first creates a batch-level split: training data from earlier batches, validation data from later batches on both lines, and a final test set containing a held-out production period. Labels include bounding boxes for defects and pixel masks for a subset of severe defects. The deployment objective prioritizes recall because missed defects are more costly than additional manual review.

Approach

The team compares three systems. A ResNet-based classifier determines whether a panel is defective. A one-stage detector locates defects and supports operator review. A lightweight segmentation network provides masks for defects requiring precise repair. Training augmentation includes brightness changes, blur, small translations, and realistic sensor noise, but not arbitrary flips because panel orientation has operational meaning.

A pretrained backbone is first frozen while the detection and segmentation heads are trained. Later, selected backbone layers are fine-tuned with a lower learning rate. The team reports per-line recall, precision, IoU, inference latency, and false alarms per production hour. It also tests performance on the second production line and under reduced illumination.

Outcome

The detector substantially improves throughput, but the first version misses narrow scratches because its input resolution is too low. Increasing resolution improves recall but exceeds the edge device’s latency budget. The final design uses a two-stage process: a fast detector identifies candidate regions, and a small high-resolution classifier or segmenter examines only those regions. Human review remains active for low-confidence predictions.

Learning Points

The case demonstrates that model choice is a systems decision. Dataset splitting, defect prevalence, asymmetric costs, resolution, hardware, and human workflow all influence success. A high test accuracy from a random split would not have been sufficient evidence of production readiness.

9. Deployment, Governance, and Ethical Issues

Vision systems may process faces, bodies, licence plates, medical images, workplaces, or private homes. Organisations must establish a lawful purpose, data-minimization policy, retention period, access controls, and secure storage. Consent and notice may be required, depending on jurisdiction and use.

Bias can arise from under-represented populations, different image quality across groups, label subjectivity, or historical decisions encoded in the data. Evaluation should be stratified by relevant groups where lawful and ethically justified. Group comparisons must be interpreted carefully: differences may reflect sampling, measurement error, or real environmental variation.

High-stakes applications require human oversight, audit logs, incident procedures, explainability appropriate to the user, and a mechanism for correction or appeal. Saliency maps can help investigate model behaviour, but they are not definitive causal explanations. Privacy-preserving techniques, controlled access, encryption, and retention limits should be treated as engineering requirements rather than optional additions.

10. Contemporary Developments and Best Practices

Current computer vision increasingly combines CNNs with attention, vision transformers, multimodal models, self-supervised pretraining, foundation models, and synthetic data. These developments can reduce annotation requirements but introduce new risks, including domain mismatch, opaque training data, hallucinated interpretations, and high computational cost.

Best practices include:

1. Define the operational objective and error costs before selecting a metric.

2. Inspect labels and document annotation uncertainty.

3. Split data by the real unit of independence.

4. Establish a simple baseline before using a large architecture.

5. Match augmentation to physical reality.

6. Use pretrained models when data or computation is limited, while checking domain compatibility.

7. Track experiments, random seeds, software versions, and preprocessing decisions.

8. Evaluate calibration, subgroup performance, robustness, latency, and energy use.

9. Test on genuinely new environments and monitor post-deployment drift.

10. Keep humans involved when errors can cause significant harm.

Challenges and Limitations

CNNs are data-hungry and may learn shortcuts such as backgrounds, watermarks, camera artefacts, or hospital-specific acquisition patterns. They can be vulnerable to adversarial perturbations and distribution shifts. Annotation is expensive, especially for pixel masks and rare events. Class imbalance can cause a model to ignore minority categories. Compression, sensor failure, occlusion, and domain changes can degrade performance.

Interpretability remains limited: visual explanations may show where a model responds without proving why the decision is correct. Large models can be expensive to train and deploy. Finally, benchmark improvements do not necessarily translate into social or commercial value; reliable integration with users, processes, and governance is essential.

Chapter Summary

CNNs learn spatially structured representations using local filters, shared parameters, nonlinear activations, and hierarchical feature extraction. Convolution output size depends on kernel size, padding, stride, and dilation, while parameter count depends on kernel dimensions and channel numbers. Classification predicts image-level labels; detection predicts objects and bounding boxes; segmentation assigns labels to pixels.

Effective computer-vision development requires careful dataset construction, augmentation, optimization, transfer learning, and evaluation. Accuracy must be supplemented by precision, recall, F1, IoU, mAP, calibration, robustness, latency, memory, and energy measurements. Pretrained architectures such as ResNet, MobileNet, and EfficientNet provide practical starting points, while attention-based models offer important alternatives.

A production system must be evaluated under realistic distribution shifts and integrated with human workflows. Privacy, fairness, security, transparency, and accountability are essential, particularly in high-stakes domains. The central professional lesson is that computer vision is not merely a model-training exercise; it is the design of a dependable socio-technical system.

Key Terms

Convolution: A local weighted operation that applies shared filters across an image.

Kernel: A learned set of weights used by a convolutional layer.

Stride: The step size with which a kernel moves across an input.

Padding: Added border values used to control output dimensions.

Receptive field: The input region influencing a particular activation.

Feature map: A spatial map produced by a filter.

Pooling: A downsampling operation such as maximum or average pooling.

Residual connection: A shortcut that adds a block’s input to its transformed output.

Data augmentation: Label-preserving transformation used to increase training variation.

Transfer learning: Reusing knowledge learned from a source task or dataset.

Fine-tuning: Updating pretrained parameters on a target task.

Classification: Assigning one or more labels to an image.

Object detection: Predicting object categories and bounding boxes.

Semantic segmentation: Assigning a class to every pixel.

Instance segmentation: Assigning pixels to classes while separating object instances.

IoU: Intersection area divided by union area between predicted and reference regions.

Precision: The proportion of positive predictions that are correct.

Recall: The proportion of actual positives that are detected.

Calibration: Agreement between predicted confidence and observed correctness.

Distribution shift: A change between training conditions and deployment conditions.

Review Questions

Short Answer Questions

1. Why does parameter sharing make CNNs more efficient than fully connected image networks?

2. What is the difference between semantic and instance segmentation?

3. When is sigmoid output preferable to softmax output?

4. Why should training augmentation normally not be applied randomly during validation?

5. What does IoU measure in object detection and segmentation?

Long Answer Questions

1. Explain the roles of convolution, padding, stride, pooling, activation functions, and residual connections in a CNN.

2. Describe a complete, leakage-resistant workflow for training and evaluating an image classifier.

3. Compare two-stage and one-stage object detectors, including their typical trade-offs.

4. Explain transfer learning and discuss when freezing layers or fine-tuning layers is appropriate.

5. Discuss why accuracy, by itself, is inadequate for evaluating a deployed vision system.

Analytical / Application Questions

1. A 64 × 64 × 3 image enters a convolutional layer with 32 filters of size 5 × 5, stride 2, and padding 2. Calculate the output spatial dimensions and trainable parameter count.

2. A medical classifier has high accuracy but low recall for a rare disease. Propose a revised evaluation and training strategy, explaining the likely trade-offs.

3. Design an evaluation plan for a defect detector deployed on two factories with different cameras and lighting. Specify data splitting, metrics, robustness tests, and deployment monitoring.

Multiple Choice Questions with Answers

Question 1

An input of size 32 × 32 × 3 passes through 16 filters of size 3 × 3 with stride 1 and padding 1. What is the output shape?

A. 30 × 30 × 16

B. 32 × 32 × 16

C. 32 × 32 × 3

D. 16 × 16 × 16

Correct Answer: B. 32 × 32 × 16

Question 2

How many trainable parameters are in a convolutional layer with 3 × 3 kernels, 3 input channels, and 16 output channels, including biases?

A. 432

B. 448

C. 480

D. 1,440

Correct Answer: B. 448

Question 3

Which output activation is generally appropriate for mutually exclusive multi-class classification?

A. Softmax

B. Independent sigmoid only

C. ReLU

D. Linear thresholding before training

Correct Answer: A. Softmax

Question 4

What does IoU measure?

A. The ratio of correct classes to all classes

B. The overlap divided by the union of two regions

C. The average training loss per epoch

D. The number of filters in a network

Correct Answer: B. The overlap divided by the union of two regions

Question 5

Which task predicts both object categories and bounding boxes?

A. Image classification

B. Object detection

C. Semantic normalization

D. Global pooling

Correct Answer: B. Object detection

Question 6

Why is a patient-level split important in a medical-image dataset?

A. It increases image resolution

B. It prevents related images from causing data leakage across splits

C. It guarantees perfect calibration

D. It removes the need for validation data

Correct Answer: B. It prevents related images from causing data leakage across splits

Question 7

Which technique reuses a model trained on a large source dataset?

A. Transfer learning

B. Non-maximum suppression

C. Random guessing

D. One-hot decoding

Correct Answer: A. Transfer learning

Question 8

If false negatives are especially costly, which metric should receive particular attention?

A. Recall

B. Parameter count only

C. Image width

D. Training batch index

Correct Answer: A. Recall

Question 9

What is a principal advantage of depthwise separable convolution?

A. It always increases accuracy

B. It can reduce parameters and computation compared with standard convolution

C. It eliminates the need for labels

D. It guarantees invariance to every transformation

Correct Answer: B. It can reduce parameters and computation compared with standard convolution

Question 10

Which practice best evaluates robustness to deployment conditions?

A. Testing only on training images

B. Using realistic changes in lighting, sensors, environments, or image quality

C. Reporting only the largest class accuracy

D. Removing all difficult examples from the test set

Correct Answer: B. Using realistic changes in lighting, sensors, environments, or image quality