Convolutional Networks and Computer Vision

Introduction and Context

Computer vision seeks to enable machines to interpret images and video. A modern vision system may classify an image, locate multiple objects, assign a class to every pixel, estimate depth, recognize actions, or combine visual evidence with language. Convolutional neural networks (CNNs) were central to the transformation of these tasks from manually engineered pipelines into representation-learning systems.

An image is not merely a vector of unrelated numbers. Nearby pixels usually have related colours, edges, textures, and shapes. CNNs exploit this spatial structure through local connectivity, shared parameters, hierarchical feature extraction, and progressively larger receptive fields. Although vision transformers and hybrid architectures are now important, CNN principles remain essential for designing, training, evaluating, and auditing practical vision systems.

This chapter develops CNN-based solutions for classification, object detection, semantic and instance segmentation, and broader image understanding. It also addresses transfer learning, augmentation, modern architectures, deployment constraints, and responsible use.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Explain why convolution, pooling, and hierarchical representations are effective for images.
2. Calculate output dimensions and parameter counts for common CNN layers.
3. Design training pipelines for image classification, detection, and segmentation.
4. Select appropriate losses and evaluation metrics for different vision tasks.
5. Apply transfer learning and augmentation without introducing data leakage.
6. Compare CNN families with modern convolutional and transformer-based architectures.
7. Diagnose errors, distribution shifts, fairness risks, and deployment constraints.

1. Images as Structured Data

A colour image can be represented as a tensor X ∈ ℝ^H×W×C, where H is height, W is width, and C is the number of channels. For standard RGB imagery, C = 3 and pixel values may be stored as integers from 0 to 255 or scaled to a floating-point interval such as [0, 1]. A batch adds another dimension, producing a tensor of shape N × H × W × C or N × C × H × W, depending on the software convention.

The spatial arrangement matters. A small edge detector applied at one location is often useful at another location. This motivates parameter sharing: the same kernel is reused across the image. It reduces the number of learned parameters and gives a useful form of translation equivariance. Equivariance means that translating an input tends to translate the resulting feature map, although padding, stride, pooling, and boundary effects can weaken this property. It is not the same as translation invariance, which means that the final prediction remains unchanged under a relevant translation.

2. The Convolutional Operation

For a two-dimensional single-channel input X and kernel K, a cross-correlation operation, commonly called convolution in deep-learning libraries, is:

Here, K has height k_h and width k_w, b is a bias, and y is the output feature map. Mathematical signal processing often flips the kernel for a strict convolution; most neural-network implementations use cross-correlation. This distinction should be understood when reproducing results across disciplines.

For a multi-channel input with C_in channels and C_out filters, the output at filter r is:

The parameter count, including one bias per output channel, is:

This assumes dense connectivity between every input and output channel. A 3 × 3 convolution with 64 input and 128 output channels therefore has 128 × (64 × 9 + 1) = 73,856 trainable parameters.

Output Dimensions

For input height H, kernel size K, padding P, and stride S, the output height is:

The corresponding width formula is obtained by replacing H with W. This formula assumes uniform padding and stride, integer indexing, and no dilation. With dilation D, the effective kernel size is K_eff = D(K − 1) + 1, so K is replaced by K_eff.

For example, an input of 224 × 224, a 3 × 3 kernel, padding 1, and stride 1 produces:

H_out = ⌊(224 + 2 − 3)/1⌋ + 1 = 224.

Thus, “same” padding preserves spatial dimensions in this particular odd-kernel, unit-stride case. It does not universally preserve dimensions when stride differs from one.

Stride, Padding, and Dilation

Stride controls how far the kernel moves. Larger stride reduces spatial resolution and computation but may discard fine detail. Padding adds boundary values, usually zeros, so that edge information is not rapidly lost. Dilation inserts gaps between kernel elements, increasing the receptive field without proportionally increasing the number of parameters. Dilation is particularly useful in segmentation, where retaining resolution is important.

Nonlinearity and Normalization

A convolution alone is a linear transformation. CNNs therefore insert nonlinear functions such as ReLU:

ReLU is computationally simple but can produce inactive units when inputs remain negative. Alternatives include GELU, SiLU/Swish, and leaky ReLU. Batch normalization normalizes activations using minibatch statistics during training and running estimates during inference. Layer normalization and group normalization can be more stable for small batches or dense prediction tasks. Normalization is not a substitute for sound preprocessing: the normalization used during training must be consistently applied at inference.

3. Pooling and Hierarchical Representation

Pooling aggregates local activations. Max pooling selects the largest activation in a window; average pooling computes its mean. Pooling can reduce computation and create limited tolerance to small translations, but it also removes spatial information. Many modern networks use strided convolutions or learned downsampling instead of, or alongside, pooling.

A CNN often learns a hierarchy:

• early layers: edges, colour contrasts, and simple orientations;
• intermediate layers: corners, textures, and parts;
• deep layers: object configurations and semantic patterns.

The receptive field is the region of the original image that can influence a unit. The theoretical receptive field grows with kernel size, stride, and depth, while the effective receptive field may be smaller because of learned weights and nonlinear interactions. A large receptive field is valuable for recognizing context, but excessive downsampling may harm small-object localization.

Figure: A typical CNN transforms pixels into increasingly abstract representations before task-specific prediction.

4. CNN Architectures and Design Patterns

LeNet demonstrated the viability of convolutional classification on handwritten digits. AlexNet showed that deep CNNs trained with GPUs, ReLU activations, data augmentation, and dropout could produce a major improvement on large-scale image recognition. VGG emphasized repeated 3 × 3 convolutions. Inception used parallel filters and factorization to obtain multi-scale processing. ResNet introduced residual connections:

Here x is a block input and F is a learned residual mapping. The shortcut improves gradient propagation and permits very deep networks. DenseNet connects a layer to many earlier layers, encouraging feature reuse. EfficientNet scales network depth, width, and input resolution jointly, while MobileNet uses depthwise-separable convolution for resource-constrained devices.

A standard convolution has cost proportional to k_h × k_w × C_in × C_out per spatial location. A depthwise-separable convolution replaces it with a depthwise convolution and a 1 × 1 pointwise convolution. Its parameter count is:

This is substantially smaller than a standard convolution when the kernel is larger than 1 × 1. The saving may reduce accuracy or representational capacity, so architecture selection should be validated on the target task rather than inferred from parameter count alone.

5. Image Classification

Classification assigns one or more labels to an image. In single-label, multi-class classification, the final layer produces logits z_1,…,z_C. Softmax converts them into probabilities:

The probabilities are non-negative and sum to one. Cross-entropy for a one-hot target y is:

For a single correct class t, this becomes L = −log(p_t). The formula assumes mutually exclusive classes. Multi-label problems instead commonly use independent sigmoid outputs and binary cross-entropy, because several labels may be true simultaneously.

Accuracy can be misleading with class imbalance. Precision, recall, F1 score, balanced accuracy, calibration error, and class-specific confusion matrices should be considered. Thresholds should be selected using validation data and an explicit cost structure, not silently inherited from a default of 0.5.

6. Object Detection

Detection predicts both a class and a bounding box for each object. A box may be represented by corner coordinates (x_min, y_min, x_max, y_max), centre and dimensions (x, y, w, h), or normalized coordinates. Intersection over Union (IoU) measures overlap:

IoU lies between 0 and 1 for valid finite boxes. Detection systems typically use a classification loss, a box-regression loss such as smooth L1 or an IoU-based loss, and sometimes an objectness loss.

Two broad families are one-stage detectors, which predict dense candidates directly, and two-stage detectors, which first generate proposals and then refine them. One-stage systems generally favour speed; two-stage systems may provide strong accuracy, especially in difficult settings. Non-maximum suppression (NMS) removes redundant boxes: predictions are sorted by confidence, and boxes with IoU above a selected threshold with a retained box are suppressed. NMS thresholds affect recall and duplicate detections and must be evaluated rather than assumed.

Mean average precision (mAP) summarizes precision–recall performance across classes and, depending on the benchmark, IoU thresholds. A reported mAP is meaningful only when its IoU convention, confidence handling, class averaging, and dataset split are specified.

7. Image Segmentation

Semantic segmentation assigns a class to every pixel, so different objects of the same class receive the same label. Instance segmentation separates individual objects. Panoptic segmentation combines semantic background regions with distinct object instances.

A common overlap metric is the Dice coefficient:

where A is the predicted foreground set and B is the ground-truth foreground set. The equivalent Jaccard index is IoU. In implementations, a small smoothing constant may be added to numerator and denominator to avoid division by zero; its value must be documented.

Encoder–decoder networks reduce spatial resolution to obtain semantic context and then restore resolution. U-Net adds skip connections from encoder stages to decoder stages, preserving fine boundaries. Feature Pyramid Networks combine multi-scale features and are useful for objects of different sizes. Fully convolutional networks replace classification heads with spatial prediction layers, allowing variable image sizes in suitable implementations.

Segmentation requires careful mask alignment. Resizing a categorical mask with bilinear interpolation creates invalid intermediate class values; nearest-neighbour interpolation is generally appropriate for discrete labels. Images and masks must undergo geometrically identical transformations.

8. Transfer Learning and Augmentation

Training a large vision model from random initialization requires substantial labelled data and compute. Transfer learning begins with a model pretrained on a source dataset, replaces or adapts its task head, and fine-tunes it on the target domain.

A robust workflow is:

1. Define train, validation, and test splits before inspecting performance.
2. Replace the source head with a target-specific head.
3. Freeze most of the backbone initially and train the new head.
4. Unfreeze selected deeper layers gradually when the target domain differs.
5. Use a smaller learning rate for pretrained layers than for the new head.
6. Compare against a domain-appropriate baseline and monitor calibration.

Freezing is not automatically correct: a medical image domain, for example, may differ substantially from natural photographs. Pretraining can also transfer unwanted biases or spurious features.

Augmentation should model plausible variation. Useful operations include crop, resize, horizontal flip when semantically valid, colour perturbation, blur, noise, MixUp, and CutMix. Vertical flipping is inappropriate for many scenes; colour changes may be harmful where colour is diagnostic. Augmentation must be applied only to training data, except for explicitly defined test-time augmentation. If an augmentation changes labels, it must also transform the labels.

Data leakage is a critical risk. Near-duplicate images, frames from the same video, images from the same patient, or multiple views of one physical object must not be distributed across train and test sets. Splitting by independent entity is often more valid than splitting by file.

9. Training, Regularization, and Evaluation

A supervised training objective is commonly written as:

Here θ denotes model parameters, f_θ the model, (x_i, y_i) a training example, L the task loss, Ω a regularizer, and λ ≥ 0 its strength. The expression assumes independent examples for the empirical average, although practical datasets may contain correlated samples.

Optimizers such as stochastic gradient descent with momentum and Adam update parameters using minibatch gradients. Learning-rate schedules, weight decay, early stopping, and mixed-precision training influence both convergence and generalization. Weight decay should be distinguished conceptually from arbitrary L2 penalties because their interaction with adaptive optimizers can differ.

Training diagnostics should include learning curves, per-class errors, examples of false positives and negatives, confidence histograms, and performance by subgroup or acquisition condition. A model with high aggregate accuracy may fail systematically on low-light images, minority groups, rare classes, or small objects.

Calibration measures whether predicted probabilities correspond to empirical frequencies. A model that assigns 0.9 confidence to many incorrect cases is unsafe even if its ranking performance is acceptable. Temperature scaling on validation data can improve calibration without changing the ranking, but it does not correct dataset shift.

10. An Illustrative Case: Automated Defect Detection

Background. A manufacturing company wants to detect surface defects on metal panels using line-scan or industrial camera images. Defects are rare, vary in size, and have high financial consequences when missed.

Problem. A prototype classifier achieves 98% accuracy, yet operators report frequent missed cracks. Inspection images from the same production run also appear in both training and test sets, inflating the estimate.

Analysis. The correct task is not merely image-level classification. A panel may contain several defects, and operators need location and severity. The data should be split by production batch or physical panel, not by nearly identical crops. Class imbalance requires recall, precision–recall curves, false-negative cost, and performance by defect type.

Approach. The team uses a pretrained backbone and an object-detection head, with a segmentation branch for elongated cracks. Rare-defect oversampling is restricted to training data. Photometric augmentation models lighting variation, while geometric augmentation preserves the panel coordinate system. Small-object performance is improved with multi-scale features. The system is evaluated at the operating threshold selected with quality engineers, not at a generic threshold.

Outcome. Offline performance is reported using per-class recall, precision, mAP at documented IoU thresholds, segmentation Dice, calibration, and inference latency. A shadow deployment runs beside human inspection. Images with low confidence or unfamiliar appearance are routed to an operator, and newly confirmed examples enter a controlled retraining process.

Learning points. The application illustrates that model choice, label design, splitting strategy, threshold selection, human workflow, and monitoring can matter as much as backbone accuracy. It also shows why a classifier should not be deployed when the operational decision requires localization.

11. Modern Vision Architectures and Image Understanding

Convolutional models remain attractive because of locality, efficient kernels, mature deployment tools, and strong inductive biases for spatial data. Vision transformers divide an image into patches and use self-attention to model long-range relationships. Hybrid systems combine convolutional stems or feature pyramids with attention. ConvNeXt demonstrates that carefully modernized convolutional designs can remain competitive with transformer-inspired systems.

Attention can be valuable when relationships between distant regions matter, but its memory and computational requirements may grow substantially with image or token count. The best architecture depends on data volume, resolution, latency, hardware, and task structure. Architecture comparisons should control training budget, augmentation, preprocessing, and evaluation protocol; otherwise apparent superiority may reflect an experimental confound.

Image understanding increasingly combines vision with language. Contrastive image–text pretraining can support zero-shot classification or retrieval, while multimodal models can answer questions about images. Such systems introduce additional risks: generated descriptions may hallucinate content, encode social stereotypes, or expose sensitive information. Grounded evaluation and abstention mechanisms are essential.

12. Deployment, Governance, and Limitations

A production vision system must satisfy more than benchmark accuracy. Important constraints include memory, throughput, latency, energy consumption, camera variation, quantization error, model update procedures, and failure recovery. Pruning, knowledge distillation, lower-precision inference, and architecture-specific accelerators may reduce cost, but each requires revalidation.

Computer vision can process biometric, workplace, medical, or public-space data. Governance should address lawful collection, consent where required, purpose limitation, retention, access controls, audit logs, and secure deletion. Facial recognition and other high-impact applications require especially strict legal and institutional review. A model should not be used to infer protected traits or sensitive states merely because visual correlations make prediction possible.

Human oversight must be meaningful rather than symbolic. Operators need clear uncertainty signals, escalation rules, and the authority to reject automated recommendations. Documentation should include dataset provenance, annotation guidelines, known limitations, subgroup results, intended use, and out-of-distribution behaviour. Monitoring should detect changes in image quality, class frequency, confidence, and error patterns.

Best-Practice Design Checklist

• Define the operational decision before selecting a model.
• Match the output type to the task: class, box, mask, or structured relation.
• Establish entity-level data splits and document leakage controls.
• Use augmentation that reflects plausible deployment variation.
• Preserve label geometry during transformations.
• Report task-appropriate metrics, uncertainty, calibration, and subgroup results.
• Use transfer learning while checking source–target domain mismatch.
• Compare accuracy with latency, memory, energy, and maintenance cost.
• Test failure cases and out-of-distribution inputs explicitly.
• Provide governance, human escalation, and an auditable update process.

Chapter Summary

CNNs exploit spatial locality and parameter sharing to transform images into hierarchical representations. Convolutional output size depends on input size, kernel, padding, stride, and dilation; parameter count depends on channel connectivity and kernel dimensions. Residual, separable, multi-scale, and encoder–decoder designs extend the basic CNN for deeper, faster, and denser prediction systems.

Classification uses probability models and losses suited to mutually exclusive or multi-label outputs. Detection adds localization and requires overlap-aware metrics such as IoU and mAP. Segmentation predicts pixel-level structure and benefits from skip connections, careful mask interpolation, and Dice or IoU evaluation. Transfer learning and augmentation improve data efficiency when applied without leakage and with domain validity.

Professional deployment requires error analysis, calibration, subgroup evaluation, monitoring, privacy protection, human oversight, and explicit documentation. Modern CNNs coexist with transformers and multimodal models; architecture selection should be evidence-based and constrained by the application.

Key Terms

Convolution: A local, shared-weight operation that produces feature maps.

Cross-correlation: The kernel operation normally implemented as “convolution” in deep-learning libraries without kernel flipping.

Kernel: A learnable filter applied across spatial positions.

Feature map: An activation tensor produced by a layer.

Stride: The spatial step between successive kernel applications.

Padding: Added boundary values used to control spatial dimensions.

Dilation: Spacing between kernel elements that enlarges the receptive field.

Receptive field: The input region that can influence a unit.

Residual connection: A shortcut that adds a block input to its learned transformation.

Transfer learning: Reusing knowledge learned on a source task or dataset.

Data augmentation: Label-preserving or label-adjusted transformation used to improve generalization.

Logit: An unnormalized output score before a probability transformation.

IoU: Intersection area divided by union area for two regions or boxes.

Non-maximum suppression: A procedure for removing overlapping duplicate detections.

Semantic segmentation: Assigning a class label to each pixel.

Instance segmentation: Separating individual object instances at pixel level.

Calibration: Agreement between predicted confidence and empirical correctness.

Distribution shift: A change between training conditions and deployment conditions.

Out-of-distribution input: An input substantially unlike the data used for training.

Review Questions

Short Answer Questions

1. Why does parameter sharing make CNNs more suitable than a fully connected layer for many images?
2. Distinguish translation equivariance from translation invariance.
3. What is the purpose of a residual connection?
4. Why is nearest-neighbour interpolation commonly used for categorical segmentation masks?
5. State one reason accuracy can be misleading in defect detection.

Long Answer Questions

1. Explain the effects of kernel size, stride, padding, and dilation on feature-map dimensions and receptive fields.
2. Compare one-stage and two-stage object detectors, including their practical trade-offs.
3. Describe a transfer-learning workflow for a small target dataset and discuss when freezing the backbone may fail.
4. Explain how an encoder–decoder network with skip connections supports semantic segmentation.
5. Discuss the technical and governance requirements for deploying a CNN in a high-impact setting.

Analytical / Application Questions

1. A 128 × 128 RGB image passes through a convolution with 32 filters of size 5 × 5, stride 2, and padding 2. Calculate the output spatial dimensions and trainable parameter count, including biases. State your assumptions.
2. Design an evaluation plan for a rare-defect detector in which false negatives cost five times as much as false positives. Which metrics, threshold procedure, split strategy, and human-escalation policy would you use?
3. A segmentation model performs well on bright laboratory images but poorly on night-time camera images. Diagnose possible causes and propose an experiment that distinguishes domain shift, label inconsistency, and inadequate augmentation.

Multiple Choice Questions with Answers

Question 1

For an input of 224 × 224, kernel size 3, padding 1, and stride 1, what is the output spatial size?

A. 222 × 222

B. 223 × 223

C. 224 × 224

D. 226 × 226

Correct Answer: C. 224 × 224

Question 2

How many parameters, including biases, are in a convolution with 64 input channels, 128 output channels, and a 3 × 3 kernel?

A. 18,432

B. 73,856

C. 73,728

D. 8,320

Correct Answer: B. 73,856

Question 3

Which loss setup is most appropriate when several labels may simultaneously be true for one image?

A. Softmax with single-class cross-entropy

B. Independent sigmoid outputs with binary cross-entropy

C. Mean squared error on bounding-box coordinates only

D. Dice loss with no output per label

Correct Answer: B. Independent sigmoid outputs with binary cross-entropy

Question 4

What does IoU measure in object detection?

A. The ratio of intersection area to union area

B. The ratio of predicted confidence to class count

C. The number of detected objects per image

D. The ratio of true positives to all pixels

Correct Answer: A. The ratio of intersection area to union area

Question 5

Which task assigns labels to individual object instances at pixel level?

A. Image classification

B. Semantic segmentation only

C. Instance segmentation

D. Image retrieval

Correct Answer: C. Instance segmentation

Question 6

Why are skip connections useful in U-Net-like segmentation models?

A. They remove the need for labels

B. They preserve fine spatial information from earlier encoder stages

C. They guarantee perfect calibration

D. They convert all masks into bounding boxes

Correct Answer: B. They preserve fine spatial information from earlier encoder stages

Question 7

Which practice best prevents leakage when images come from multiple frames of the same video?

A. Randomly distribute all frames across splits

B. Split data by independent video or recording entity

C. Apply stronger augmentation to the test set

D. Remove the validation set

Correct Answer: B. Split data by independent video or recording entity

Question 8

A detector outputs two highly overlapping boxes for one object. Which procedure is commonly used to retain one and suppress the other?

A. Batch normalization

B. Nearest-neighbour interpolation

C. Non-maximum suppression

D. Global average pooling

Correct Answer: C. Non-maximum suppression

Question 9

Which statement about strict mathematical convolution and common deep-learning implementations is correct?

A. They are always identical in indexing

B. Deep-learning libraries commonly use cross-correlation without flipping the kernel

C. Deep-learning libraries never use learned kernels

D. Strict convolution cannot produce feature maps

Correct Answer: B. Deep-learning libraries commonly use cross-correlation without flipping the kernel

Question 10

A 128 × 128 RGB input uses 32 filters of size 5 × 5, stride 2, and padding 2. What are the output size and parameter count?

A. 64 × 64 × 32 and 2,432 parameters

B. 63 × 63 × 32 and 2,400 parameters

C. 64 × 64 × 16 and 2,432 parameters

D. 128 × 128 × 32 and 2,432 parameters

Correct Answer: A. 64 × 64 × 32 and 2,432 parameters