Convolutional Neural Networks and Computer Vision

Introduction and Context

Computer vision enables machines to interpret images and video. Unlike ordinary tabular data, an image contains structured spatial relationships: neighbouring pixels are usually more related than distant pixels, and the same visual pattern may appear at different locations. Convolutional neural networks (CNNs) exploit this structure through local connectivity, shared parameters and hierarchical feature learning.

CNNs are central to image classification, object detection, semantic and instance segmentation, medical imaging, industrial inspection, autonomous systems and visual search. Their practical importance has increased through transfer learning, residual architectures, hardware acceleration and large-scale pretrained models.

This chapter develops the principles required to design, train, evaluate and deploy CNN-based computer-vision systems. It also considers limitations, data governance, interpretability and responsible use.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain how convolution, activation, padding and pooling transform image representations.
2. Calculate CNN output dimensions and parameter counts.
3. Design simple CNN architectures for classification.
4. Explain residual connections and their role in training deep networks.
5. Select and adapt pretrained models using transfer learning.
6. Distinguish classification, detection, semantic segmentation and instance segmentation.
7. Apply CNNs to visual inspection and evaluate their performance appropriately.
8. Identify operational, ethical and technical limitations of computer-vision systems.

1. Images as Structured Numerical Data

A grayscale image can be represented as a two-dimensional array of intensity values. A colour image normally has three channels—red, green and blue—and is represented as a tensor with height, width and channels. A batch of images adds a fourth dimension. For example, a batch of 32 RGB images of size 224 × 224 has shape 32 × 224 × 224 × 3 in a channels-last convention.

Pixel values may be stored as integers from 0 to 255, but neural networks generally perform better when inputs are scaled or standardised. Common choices include dividing by 255 to obtain values in [0, 1], or standardising each channel using a training-set mean μ and standard deviation σ:

Here, x is a pixel value and x′ is the transformed value. The same preprocessing must be used during training, validation and deployment. Statistics must be calculated from the training set only to avoid data leakage.

A CNN does not usually treat each pixel as an unrelated feature. Instead, it learns local patterns such as edges, corners and textures in early layers, then combines them into parts and objects in deeper layers.

2. The Convolution Operation

A convolutional layer applies learnable filters, also called kernels, to local regions of an input. In deep-learning libraries, the operation is technically often cross-correlation rather than mathematical convolution because the kernel is not flipped; the term convolution remains standard in machine learning.

For a two-dimensional single-channel input, the output at location (i, j) may be written as:

The symbols are defined as follows:

For multi-channel input, the filter spans all input channels. If the input has C_in channels, the parameter count for one filter is K_h × K_w × C_in weights plus one bias. With C_out filters, the total parameter count is:

This formula assumes one bias per output channel and no grouped or depthwise convolution.

Why Parameter Sharing Matters

A fully connected layer connecting a 224 × 224 × 3 image to 1,000 units would require more than 150 million weights. A 3 × 3 convolution with 64 filters requires:

The same filter is applied at many spatial positions. This reduces parameters and gives the network a useful form of translation equivariance: if a pattern moves in the input, its feature response tends to move correspondingly. Pooling, striding and later classification layers may reduce exact positional information, so CNNs are not perfectly translation invariant.

Stride and Output Size

Stride determines how far the filter moves between positions. For an input dimension N, kernel size K, padding P on each side and stride S, the output dimension is:

For rectangular inputs, the formula is applied independently to height and width. The floor operation is required when the parameters do not produce an integer number of positions. The formula assumes dilation of one and symmetric padding represented by P.

Worked example. An input feature map is 32 × 32. A 5 × 5 kernel uses padding 2 and stride 1:

Thus, the spatial dimensions are preserved. If stride were 2, the output would be 16 × 16 because:

3. Padding, Activation and Pooling

Padding

Without padding, repeated convolution gradually shrinks feature maps and discards information near boundaries. Common schemes include:

• Valid padding: no added pixels; spatial dimensions usually decrease.
• Same padding: padding is selected so that stride-1 convolution preserves spatial dimensions.
• Zero padding: added pixels have value zero; other padding modes include reflection and replication.

For an odd K × K kernel and stride 1, same padding commonly uses P = (K − 1)/2. This relationship is exact for odd kernels under the usual symmetric-padding convention.

Padding introduces assumptions about what exists beyond the image boundary. Zero padding is convenient but may create artificial edges. Reflection padding can sometimes reduce this effect, although its suitability depends on the application.

Nonlinear Activation

A convolution without a nonlinear activation would result in a linear transformation. Stacking linear transformations would still be equivalent to one linear transformation. Activations therefore allow the network to represent complex functions.

The rectified linear unit (ReLU) is:

ReLU is computationally simple and generally supports effective gradient flow for positive inputs. However, units can become permanently inactive when their inputs remain negative; this is known as the dying-ReLU problem. Alternatives include Leaky ReLU, GELU and SiLU.

A typical block is convolution → normalisation → activation. Batch normalisation uses batch statistics during training and moving estimates during inference. It can stabilise optimisation, but its behaviour may become unreliable with very small batch sizes. Group normalisation is often more suitable when batches are small.

Pooling

Pooling summarises local regions and reduces spatial resolution. Max pooling selects the largest value:

Average pooling computes the arithmetic mean. Max pooling can preserve strong evidence for a feature, while average pooling retains a broader measure of activation. Pooling reduces computation and may provide limited robustness to small translations, but it also discards spatial detail.

Strided convolution can perform downsampling while learning the transformation. It is often preferred in modern architectures when a learnable reduction is useful.

4. Hierarchical Feature Learning

A shallow CNN may learn oriented edges. Intermediate layers combine edges into corners, curves, textures and object parts. Deeper layers represent more abstract combinations. This hierarchy is not manually programmed; it emerges through optimisation against the training objective.

A classification CNN commonly contains:

1. An input and preprocessing stage.
2. Several convolutional blocks.
3. Downsampling stages that reduce spatial size and increase channel depth.
4. A feature aggregation stage, often global average pooling.
5. A classification head.

Global average pooling averages each final feature map over its spatial dimensions. If the final tensor has height H, width W and C channels, it produces C values:

This often uses fewer parameters than flattening the entire feature map and can reduce overfitting. A final dense layer produces class logits. For mutually exclusive classes, softmax converts logits z_c into probabilities:

The categorical cross-entropy loss for a one-hot target y is:

This loss assumes one correct class per image and probabilities generated by softmax. For multi-label classification, where several classes may be present simultaneously, independent sigmoid outputs and binary cross-entropy are more appropriate.

5. Residual Connections and Deep Network Design

As networks become deeper, optimisation may become difficult. A deeper network can exhibit degradation: training accuracy may become worse even though the deeper model contains the shallower model in principle. Vanishing gradients, unstable optimisation and poorly conditioned transformations contribute to this difficulty.

A residual block learns a residual function F(x) and adds the input through an identity shortcut:

Here, x is the block input, W represents the block parameters and F is usually a sequence of convolutions, normalisation and activations. If the input and output shapes differ, a projection shortcut—often a 1 × 1 convolution with suitable stride—is used:

The addition requires matching tensor dimensions. Residual connections provide a direct pathway for information and gradients. They do not guarantee perfect optimisation, but they make very deep networks more practical. ResNet is the best-known family based on this principle.

Practical Architecture Choices

Design should be guided by data, compute and task requirements rather than depth alone.

The receptive field is the region of the original image that can influence a unit. It grows with stacked kernels, strides and dilation. A large theoretical receptive field does not necessarily mean that the model uses all contextual information effectively; data and optimisation also matter.

6. Transfer Learning

Training a deep CNN from random initialisation requires substantial labelled data and computation. Transfer learning begins with a model pretrained on a large source dataset and adapts it to a target task.

Two standard strategies are:

1. Feature extraction: freeze most or all backbone layers, compute learned representations and train a new task-specific head.
2. Fine-tuning: unfreeze some later layers and continue training with a small learning rate.

A reliable workflow is to replace the original classification head, freeze the backbone initially, train the new head, then gradually unfreeze selected layers if validation performance improves. Batch-normalisation statistics require care when batch sizes are small or when the target domain differs strongly from the source domain.

Transfer learning works best when source and target visual statistics are sufficiently related. A model pretrained on ordinary photographs may transfer effectively to animals or vehicles, but less directly to specialised microscopy, infrared imagery or industrial sensors. Domain shift can require stronger augmentation, domain-specific pretraining or training from scratch.

Data Augmentation

Augmentation creates plausible variations such as horizontal flips, crops, rotations, colour changes, noise and blur. It should reflect real-world variation. A vertical flip is inappropriate for many road scenes but may be valid for some texture-inspection tasks. Augmentation must not change the class label or violate physical constraints.

For detection and segmentation, image transformations must also transform bounding boxes and masks. Augmentation applied only to the image creates incorrect labels.

Avoiding Leakage

Near-duplicate images, frames from the same video and images of the same physical object should not be distributed carelessly across training and test sets. Otherwise, performance estimates may be unrealistically high. Splits should reflect deployment conditions, such as time, location, patient or production batch.

7. Computer-Vision Tasks and Model Heads

Image Classification

Classification assigns one or more labels to an entire image. A single-label problem uses one class among C alternatives; a multi-label problem permits several independent labels. Accuracy can be misleading when classes are imbalanced. Precision, recall, F1 score and class-specific confusion matrices should also be considered.

Object Detection

Detection predicts both object categories and locations, commonly as bounding boxes. A box may be represented by coordinates (x_min, y_min, x_max, y_max). Intersection over Union (IoU) measures overlap between a predicted box B_p and ground-truth box B_g:

IoU ranges from 0 to 1. Detection systems generally produce many candidate boxes, which are filtered using confidence thresholds and non-maximum suppression (NMS). NMS retains a high-confidence box and suppresses highly overlapping lower-confidence boxes. One-stage detectors predict locations and classes directly, while two-stage detectors first generate candidate regions and then refine them.

Semantic Segmentation

Semantic segmentation assigns a class to every pixel, such as road, sky or tumour. It does not distinguish between two objects of the same class. Pixel accuracy may be dominated by large background regions, so mean intersection over union is often more informative:

Here, TP_c, FP_c and FN_c refer to true positives, false positives and false negatives for class c. Mean IoU is the arithmetic mean of IoU_c across the selected classes.

Instance Segmentation

Instance segmentation identifies each object separately, even when objects share a class. It combines detection and pixel-level masks and is useful for counting, robotics and medical or industrial measurement.

Visual Inspection

Industrial visual inspection commonly detects scratches, dents, missing components, contamination or assembly errors. The challenge is often not model selection but data quality, rare-defect sampling, changing illumination, camera calibration and defining acceptable tolerances. A system may use classification for pass/fail decisions, detection for locating defects or segmentation for measuring defect area.

8. Training and Evaluation Workflow

A professional workflow includes:

1. Define the operational decision and acceptable error types.
2. Establish data ownership, consent, labelling rules and retention policies.
3. Inspect class balance, image quality, duplicates and possible leakage.
4. Create deployment-representative training, validation and test splits.
5. Establish a simple baseline.
6. Train with augmentation and suitable optimisation settings.
7. Tune thresholds on validation data, not on the final test set.
8. Evaluate overall and subgroup performance.
9. Test robustness under lighting, camera, location and time variation.
10. Deploy with monitoring, human review and a rollback plan.

For binary classification:

• Precision = TP/(TP + FP): among predicted positives, how many are correct.
• Recall = TP/(TP + FN): among actual positives, how many are found.
• F1 score = 2PR/(P + R), where P is precision and R is recall.

Threshold selection should reflect costs. In safety inspection, missing a dangerous defect may be much worse than generating a false alarm. In such cases, high recall may be prioritised, with human review handling additional alerts.

Calibration is also important. A confidence score of 0.9 should correspond approximately to a 90% success rate for predictions in the relevant operating population. Accuracy alone does not establish calibration.

9. Worked Design Example

Consider a 128 × 128 RGB image and the following classification backbone:

• Conv1: 3 × 3 kernel, 32 filters, stride 1, same padding.
• 2 × 2 max pooling, stride 2.
• Conv2: 3 × 3 kernel, 64 filters, stride 1, same padding.
• 2 × 2 max pooling, stride 2.
• Global average pooling.
• Dense output layer for 5 classes.

The first convolution preserves spatial dimensions, producing 128 × 128 × 32. Pooling gives 64 × 64 × 32. The second same-padded convolution produces 64 × 64 × 64. The second pooling operation gives 32 × 32 × 64. Global average pooling produces a vector of length 64, and the output layer produces five logits.

Parameter counts are:

The total trainable parameter count, excluding normalisation layers, is 19,717. This design is small enough for a baseline and can be expanded if validation evidence indicates underfitting. It should not be expanded merely because a larger model is available.

10. Illustrative Case: Automated Surface-Defect Inspection

Background

A manufacturer of coated metal panels wants to identify scratches, pinholes and coating bubbles before shipping. Images are captured by a fixed camera and controlled lighting system.

Problem

Defects occupy a small fraction of each image and are less common than acceptable surfaces. Some defects vary in appearance, while reflections can resemble scratches. A model that reports high overall accuracy but misses rare defects is unacceptable.

Analysis

The problem requires defect localisation as well as classification because operators need to verify and repair the affected area. A random image split could be misleading if multiple images of the same panel appear in both training and test sets. The split should be performed by production batch or physical panel.

Approach

An illustrative solution uses a pretrained CNN backbone with a detection or segmentation head. The backbone is initially frozen, then partially fine-tuned using a low learning rate. Augmentation includes modest brightness variation, small translations and realistic blur. Labels are reviewed by at least two trained inspectors, with disagreements adjudicated under a written policy.

The evaluation reports recall for each defect type, precision, false alarms per panel, localisation IoU and performance by production batch. The decision threshold is selected to achieve a required minimum recall. Predictions below a confidence threshold are sent to human review rather than automatically accepted.

Outcome

In a production pilot, the system should be judged not only by test metrics but also by reduced inspection time, missed-defect rate, operator workload and stability across cameras and shifts. If performance declines after a lighting change, monitoring should trigger recalibration or retraining.

Learning Points

The case demonstrates that computer vision is a socio-technical system. Model architecture matters, but data governance, label quality, process control, threshold design and human workflows are equally important.

11. Limitations, Ethics and Governance

CNNs can fail under distribution shift: unfamiliar cameras, weather, lighting, image compression, demographic variation or new defect types may reduce performance. Adversarial perturbations and spurious correlations can also produce confident errors. Saliency maps may assist investigation, but they are not proof that a model used a particular causal feature.

High-stakes applications require documented intended use, validation boundaries, human oversight and incident procedures. Facial analysis, medical imaging and workplace monitoring may involve sensitive personal data and legal restrictions. Organisations should establish lawful data collection, access control, retention limits, audit logs and secure model deployment.

Fairness evaluation should compare error rates across relevant groups or operating conditions, but subgroup definitions must be appropriate and privacy-preserving. A model should not be deployed solely because its average score is high if it performs poorly for a consequential subgroup.

In industrial environments, safety decisions should have clearly defined authority. Automation should not silently override established safety procedures. Model cards, dataset documentation and change-control records improve accountability.

12. Contemporary Developments and Best Practices

Modern vision systems increasingly use attention mechanisms, vision transformers, self-supervised pretraining, multimodal models and foundation models. CNNs remain important because they are efficient, well understood and effective on many edge and real-time workloads. Hybrid architectures combine convolutional inductive biases with attention-based global context.

Best practices include:

• Start with a transparent baseline and a deployment-relevant metric.
• Use pretrained weights when source and target domains are reasonably related.
• Freeze, then progressively fine-tune; use lower learning rates for pretrained layers.
• Keep preprocessing identical across training and deployment.
• Split data by the unit that can create dependence, not merely by file name.
• Inspect false positives and false negatives, not only aggregate scores.
• Measure latency, memory, energy use and reliability in the target environment.
• Monitor data drift, confidence distributions and subgroup performance.
• Preserve human escalation routes for uncertain or high-impact cases.
• Retrain only after verifying that new labels are accurate and representative.

Chapter Summary

Convolutional neural networks learn spatially organised representations through local filters, shared parameters and nonlinear transformations. Kernel size, stride and padding determine feature-map dimensions, while pooling or strided convolution reduces spatial resolution. Parameter sharing makes CNNs substantially more efficient than fully connected image models.

Deep CNNs build hierarchical features from edges and textures to object parts and semantic patterns. Residual connections improve the optimisation of deep networks by providing identity or projection shortcuts. Transfer learning allows pretrained representations to be adapted efficiently, but domain shift, preprocessing consistency and data leakage must be managed carefully.

Classification predicts image-level labels; detection predicts categories and bounding boxes; semantic segmentation assigns classes to pixels; and instance segmentation separates individual objects. Evaluation must use metrics suited to the task and costs, including precision, recall, IoU and mean IoU. In real deployments, data quality, threshold selection, monitoring, human oversight, privacy and governance are as important as architecture.

Key Terms

Convolution: A local, shared-weight operation that produces feature maps from input data.

Kernel or filter: A learnable tensor applied across local input regions.

Feature map: The spatial output produced by a convolutional filter.

Stride: The number of pixels by which a kernel moves between applications.

Padding: Values added around an input boundary before convolution.

ReLU: An activation function defined as max(0, z).

Pooling: A local aggregation operation that reduces spatial resolution.

Receptive field: The region of the original input that can influence a unit.

Residual connection: A shortcut that adds a block input to its transformed output.

Global average pooling: Spatial averaging that converts each feature map to one value.

Logit: An unnormalised score produced before a probability transformation.

Softmax: A transformation from class logits to mutually exclusive class probabilities.

Transfer learning: Adapting a model or representation learned from another dataset or task.

Fine-tuning: Updating pretrained parameters on a target dataset.

Object detection: Prediction of object categories and their locations.

Semantic segmentation: Assigning a class label to every pixel.

Instance segmentation: Producing separate pixel masks for individual objects.

Intersection over Union: Overlap area divided by union area for predicted and true regions.

Data leakage: Unintended use of information from evaluation data during model development.

Distribution shift: A change between training data conditions and deployment conditions.

Review Questions

5 Short Answer Questions

1. Why does parameter sharing make CNNs efficient for image processing?
2. State the output-dimension formula for a convolution and define its variables.
3. What is the difference between valid and same padding?
4. Why are residual connections useful in deep networks?
5. Distinguish semantic segmentation from instance segmentation.

5 Long Answer Questions

1. Explain the operation of a convolutional block, including convolution, activation, padding and pooling.
2. Discuss the advantages and limitations of transfer learning for a specialised visual-inspection dataset.
3. Compare classification, object detection and segmentation in terms of outputs and suitable evaluation metrics.
4. Explain how data leakage can occur in image datasets and propose a robust splitting strategy.
5. Discuss the technical and governance requirements for deploying a CNN in a high-impact application.

3 Analytical / Application Questions

1. Design a CNN for classifying five types of manufactured components. Justify the input preprocessing, downsampling strategy, use of transfer learning and evaluation metrics.
2. A detector has high precision but low recall for safety-critical defects. Analyse possible causes and recommend changes to data collection, threshold selection and human review.
3. A model performs well in laboratory images but poorly on images from a new factory camera. Propose an investigation and adaptation plan addressing domain shift, calibration, validation and monitoring.

Multiple Choice Questions with Answers

Question 1

For an input dimension of 32, kernel size 5, padding 2 and stride 1, what is the output dimension?

A. 28

B. 30

C. 32

D. 36

Correct Answer: C. 32

Question 2

How many parameters, including one bias per filter, are in a convolution with a 3 × 3 kernel, 3 input channels and 64 filters?

A. 576

B. 1,728

C. 1,792

D. 18,496

Correct Answer: C. 1,792

Question 3

Which operation most directly reduces spatial resolution while retaining the strongest local activation?

A. Max pooling

B. Softmax

C. Batch splitting

D. One-hot encoding

Correct Answer: A. Max pooling

Question 4

What is the main purpose of a residual connection?

A. To remove all nonlinearities

B. To provide a shortcut for information and gradients

C. To convert logits into probabilities

D. To guarantee translation invariance

Correct Answer: B. To provide a shortcut for information and gradients

Question 5

Which output is most characteristic of object detection?

A. One class label for the whole image only

B. A class label for every pixel without object identities

C. Class labels and bounding-box coordinates

D. Only a reconstructed image

Correct Answer: C. Class labels and bounding-box coordinates

Question 6

Which metric measures overlap between a predicted region and a ground-truth region?

A. Recall

B. IoU

C. Latency

D. Calibration slope

Correct Answer: B. IoU

Question 7

For a multi-label image in which several classes may be present simultaneously, which output strategy is generally appropriate?

A. One softmax distribution over mutually exclusive classes

B. Independent sigmoid outputs

C. A single linear output with no loss

D. Max pooling over class labels

Correct Answer: B. Independent sigmoid outputs

Question 8

What is the recommended initial approach when adapting a pretrained CNN to a small target dataset?

A. Freeze the backbone and train a new task-specific head

B. Randomly delete all pretrained layers

C. Use the test set for threshold tuning

D. Disable input normalisation

Correct Answer: A. Freeze the backbone and train a new task-specific head

Question 9

If a binary inspection system has many missed defects, which metric is especially important to examine?

A. Recall

B. Parameter count only

C. Image width only

D. Training batch index

Correct Answer: A. Recall

Question 10

Why should images from the same physical object usually remain within one dataset split?

A. To increase image compression

B. To prevent dependence and overly optimistic evaluation

C. To guarantee every class has equal size

D. To eliminate the need for validation data

Correct Answer: B. To prevent dependence and overly optimistic evaluation