Deep Learning and Contemporary Vision AI

Introduction and Context

Machine vision has moved from hand-designed image features toward learned representations. In a conventional system, an engineer may select edges, corners, texture measures, or geometric descriptors and then train a classifier. Deep learning instead learns increasingly abstract representations directly from images, usually through convolutional neural networks (CNNs) or attention-based vision transformers (ViTs). This shift has enabled reliable classification, object detection, semantic and instance segmentation, pose estimation, optical character recognition (OCR), and visual anomaly detection.

For mechatronics engineers, vision AI is not merely a software topic. A deployed system must connect optics, illumination, sensors, mechanics, real-time computation, robot control, safety, data management, and maintenance. A highly accurate model can still fail if the camera is misaligned, the lighting changes, latency is excessive, or the training data do not represent the production environment.

This chapter develops the principles needed to design, evaluate, and govern contemporary vision systems. It emphasizes transfer learning, augmentation, synthetic data, foundation models, explainability, and the distinction between benchmark performance and dependable operation.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Explain how CNNs and transformers learn visual representations.
2. Select suitable architectures for classification, detection, segmentation, pose estimation, OCR, and anomaly detection.
3. Apply transfer learning, augmentation, class balancing, and synthetic data appropriately.
4. Evaluate models using task-appropriate metrics and operating constraints.
5. Design a vision-AI pipeline that includes data, hardware, deployment, and monitoring.
6. Interpret explainability methods without confusing explanation with proof of causation.
7. Identify safety, privacy, bias, security, and governance risks.

1. From Pixels to Learned Representations

An image can be represented as a tensor X ∈ R^H×W×C, where H and W are image height and width in pixels and C is the number of channels. A grayscale image commonly has C=1, whereas an RGB image has C=3. Pixel values are often normalized to a range such as [0, 1] or standardized using the training-set mean and standard deviation.

A learning system approximates a function f_θ(X), where θ denotes trainable parameters. In supervised classification, the model predicts a probability distribution over K classes. With one-hot target vector y, the cross-entropy loss is:

Here p_k is the predicted probability for class k. The expression assumes p_k>0 and ∑_k p_k=1. For a single correct class, the loss is the negative logarithm of its predicted probability. Training seeks parameters that minimize average loss over the training data, usually by stochastic gradient descent or a related optimizer.

Convolutional neural networks

A convolution layer applies learned filters locally. For a two-dimensional single-channel example, a discrete cross-correlation operation is:

W is a filter of size r×s, b is a bias, and Z is the resulting feature map. In many machine-learning texts this operation is called convolution, although strict mathematical convolution reverses the kernel. Stride changes the sampling step; padding controls border treatment; multiple filters produce multiple output channels.

Early layers often learn edges and color transitions. Intermediate layers combine these into textures and parts, while later layers represent objects or task-specific patterns. Pooling or strided convolution can reduce spatial resolution and increase the effective receptive field. Residual connections, as used in ResNet-like architectures, help optimization by providing an identity pathway:

The tensors must have compatible dimensions; a projection is used when channel count or spatial size differs.

CNNs are computationally efficient and naturally exploit local spatial structure. They remain strong choices for industrial inspection, especially when image appearance is stable, latency is important, and training data are limited.

Vision transformers

A vision transformer divides an image into patches, flattens each patch, and maps it to an embedding. If an image has dimensions H×W and square patch size P, the number of patches, assuming divisibility, is:

A transformer adds positional information because self-attention alone does not intrinsically encode patch order. For query, key, and value matrices Q, K, and V, scaled dot-product attention is:

d_k is the key-vector dimension. The scaling reduces excessively large dot products and improves numerical behavior. Multi-head attention uses several learned projections, allowing the model to capture different relationships.

Transformers can model long-range relationships more directly than local convolutions, but they often require substantial data or effective pretraining. Hybrid models combine convolutional inductive bias with attention. The correct choice depends on data scale, compute budget, image resolution, latency, and the spatial structure of the task—not on the fact that one architecture is newer.

2. Learning Strategy: Data, Loss, and Generalization

A dataset should be divided into training, validation, and test sets. The test set must remain isolated until final evaluation. Images from the same physical part, video sequence, or production batch should generally remain in one split; random frame-level splitting can create leakage and produce unrealistically high results.

Overfitting occurs when a model fits training examples but performs poorly on unseen cases. Common controls include weight decay, dropout, early stopping, data augmentation, smaller models, and more representative data. A useful conceptual decomposition is:

• Bias: systematic error caused by an overly restrictive model or inadequate features.
• Variance: sensitivity to the particular training sample.
• Distribution shift: change between training and deployment conditions, such as new lighting, camera optics, materials, or defect types.

Transfer learning

Transfer learning starts with a model pretrained on a large dataset or self-supervised corpus. The final task-specific head is replaced, and the network is first trained with the backbone frozen. Selected layers may then be unfrozen for fine-tuning using a lower learning rate. This approach is valuable when labelled industrial data are scarce.

Pretraining is not automatically beneficial. A model trained on natural images may have weak representations for thermal images, microscopic surfaces, X-ray data, or highly repetitive manufactured parts. Fine-tuning must also preserve a validation set from the target domain. Domain-specific pretraining or self-supervised learning may be preferable.

Augmentation

Augmentation creates plausible training variations. Geometric transformations include translation, cropping, rotation, and scale changes. Photometric transformations include brightness, contrast, blur, noise, and color changes. CutMix and MixUp combine samples, but they may be inappropriate when object boundaries or physical appearance are important.

An augmentation is valid only if it preserves the task label. Horizontal flipping is unsafe when left-right orientation matters; aggressive rotation is unsafe for gravity-dependent assemblies; artificial blur may hide the defect being detected. Augmentation should model expected variation rather than manufacture unrealistic images.

Synthetic data

Synthetic images can provide rare defect examples, controlled annotations, and safer data generation. They may be produced using CAD rendering, simulation, procedural textures, or generative models. Their main risk is the simulation-to-reality gap: unrealistic textures, reflections, object contacts, noise, and lighting can cause a model to learn synthetic cues.

A practical strategy is domain randomization—varying backgrounds, illumination, textures, camera pose, and object parameters—followed by fine-tuning on real images. Synthetic data should be audited visually and quantitatively, and real-world acceptance tests remain essential.

3. Vision Tasks and Appropriate Metrics

Classification

Classification assigns one or more labels to an image or region. Single-label multiclass classification uses one mutually exclusive class; multilabel classification allows several independent labels. Accuracy is:

For a particular positive class, precision and recall are:

TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives. These metrics assume a defined decision threshold and must be interpreted with class prevalence. F1 score is the harmonic mean of precision and recall:

Accuracy can be misleading for rare defects. A model that labels every item “normal” may achieve 99% accuracy when only 1% are defective, while detecting no defects.

Object detection

A detector predicts class labels and bounding boxes. Intersection over Union (IoU) measures overlap between predicted box A and ground-truth box B:

IoU ranges from 0 to 1. A detection is commonly considered a match when IoU exceeds a specified threshold, but the threshold must reflect the application. Mean average precision (mAP) summarizes precision-recall performance across classes and, in many benchmarks, across IoU thresholds. Reports must state the exact convention because mAP implementations differ.

Single-stage detectors are usually fast and suitable for real-time robotics. Two-stage detectors may offer strong accuracy for difficult scenes but at higher computational cost. Non-maximum suppression removes highly overlapping duplicate boxes, though crowded scenes require specialized methods.

Segmentation

Semantic segmentation assigns a class to every pixel; instance segmentation additionally distinguishes separate objects of the same class. The intersection-over-union formula also applies to masks. Mean IoU averages IoU across classes, while Dice similarity is:

Small defects can be severely penalized by a few misplaced pixels. Boundary quality, defect-level recall, and connected-component errors may therefore be more informative than a single average score.

Pose estimation

Pose systems estimate keypoints, skeletons, or six-degree-of-freedom object pose. Keypoint accuracy may use a normalized distance between predicted and true points. The normalization factor should be defined, such as person scale or object diameter. For robotic manipulation, pixel accuracy alone is insufficient: camera calibration, depth uncertainty, coordinate transformation, and control latency determine whether a grasp succeeds.

OCR and document understanding

OCR commonly consists of text detection, rectification, recognition, and post-processing. Character error rate (CER) can be expressed as:

where S, D, and I are substitutions, deletions, and insertions required to transform the predicted string into the reference string, and N is the number of reference characters. Word error rate uses the same idea at word level. An OCR system for traceability should also validate syntax, checksums, and permitted character sets rather than trusting visual confidence alone.

Anomaly detection

Anomaly detection is useful when defective examples are rare or unknown. A model may learn normal images through an autoencoder, feature distribution, one-class objective, or pretrained embedding. An anomaly score must be calibrated on representative normal and abnormal validation data. “Unseen” does not necessarily mean “defective”: a new but acceptable product may be flagged because it lies outside the training distribution.

4. Foundation Models and Contemporary Architectures

Foundation vision models are pretrained on broad image, text, or multimodal data and adapted through fine-tuning, prompting, adapters, or retrieval. Vision-language models can associate images with textual concepts and may support open-vocabulary detection or visual inspection assistance. Their flexibility is valuable for prototyping, but their outputs can be unstable, difficult to calibrate, and unsuitable as the sole safety decision-maker.

Segmentation foundation models can produce masks from points, boxes, or textual prompts. They reduce annotation effort, but prompt dependence and domain mismatch remain important. A useful engineering pattern is human-in-the-loop annotation: a foundation model proposes labels, a qualified person corrects them, and a task-specific model is trained and tested on verified data.

Model compression is often required at the edge. Quantization reduces numerical precision, pruning removes parameters or operations, and knowledge distillation trains a smaller student model from a larger teacher. Each can reduce memory and latency but may reduce accuracy, especially for small objects and rare classes. The compressed model must be evaluated on the deployment hardware, not only on a workstation.

5. Explainability, Reliability, and Evaluation

Explainable AI methods include saliency maps, gradient-weighted activation maps, occlusion tests, feature visualization, counterfactual examples, and example-based explanations. They can reveal that a classifier attends to a background, watermark, fixture, or illumination artifact rather than the intended defect.

An explanation is a diagnostic aid, not proof that the highlighted region caused the decision. Saliency methods can be sensitive to model architecture, smoothing, and visualization settings. Engineers should compare multiple methods and test explanations with controlled perturbations.

Evaluation should include:

1. Discrimination: Can the model separate classes or localize objects?
2. Calibration: Do confidence values correspond to observed frequencies?
3. Robustness: Does performance survive expected changes in lighting, pose, contamination, and sensor noise?
4. Operational performance: Are latency, throughput, memory, and uptime acceptable?
5. Safety performance: Are dangerous false negatives sufficiently rare?
6. Maintainability: Can data drift be detected and the model updated under configuration control?

For a binary classifier, a confusion matrix should be reported with the operating threshold. Receiver operating characteristic curves summarize true-positive versus false-positive rates across thresholds, while precision-recall curves are often more informative for rare defects. Confidence intervals or repeated production trials are preferable to presenting a single score without uncertainty.

6. Deployment in a Mechatronic System

A complete vision-AI system includes illumination, lens, sensor, trigger, image acquisition, preprocessing, inference, decision logic, actuator interface, logging, and human procedures. A typical pipeline is:

Figure: A closed-loop industrial vision-AI deployment pipeline.

The inference threshold should be selected from the cost of errors. If a missed defect is much more costly than a false alarm, the threshold may be lowered to increase recall, with human review or a second inspection stage managing the additional alarms. Safety-related functions should use appropriate fail-safe design and independent verification; a neural network should not silently override a hard safety interlock.

7. Illustrative Case: Vision Inspection of Welded Assemblies

Background

A fictional automotive supplier inspects welded brackets on a conveyor. Defects include missing welds, excessive spatter, and incorrect bracket orientation. The existing rule-based system fails when surface finish and illumination vary.

Problem

Only 420 defective images are available among 42,000 normal images. Defect prevalence is therefore approximately 1%. A random image split would be dangerous because consecutive frames from the same production run are visually correlated.

Analysis

The team defines the unit of generalization as a production batch and creates batch-level training, validation, and test splits. It records camera exposure, lens, lighting state, part variant, and operator decision. False negatives are assigned greater cost because an undetected bad weld can cause field failure.

Approach and solution

A pretrained CNN is fine-tuned for image-level screening, while a detector localizes the bracket and weld regions. Mild brightness and translation augmentation reflect expected variation. Horizontal flipping is rejected because connector orientation is meaningful. Synthetic CAD renderings provide orientation examples, but real images are required for weld texture. Grad-CAM and occlusion tests identify a background fixture that the first model used spuriously; the fixture is redesigned and background variation is added.

The final pipeline uses a high-recall threshold followed by a human review queue for low-confidence images. The system logs image identifiers, model version, confidence, decision, and operator override. Performance is reported separately by part variant and lighting condition, rather than only as one aggregate accuracy.

Outcome

In an acceptance trial, the model reduces manual inspection workload while preserving the required defect recall. The manufacturer does not deploy the model as an autonomous safety authority: rejected parts are held, and a controlled escalation procedure handles uncertain cases. Periodic sampling of accepted parts monitors drift.

Learning points

The central improvement came not from selecting the newest architecture, but from defining the split correctly, improving image formation, removing shortcut cues, selecting a threshold based on risk, and integrating the model with human and mechanical processes.

8. Ethical, Security, and Governance Considerations

Vision systems can collect faces, worker behavior, product designs, and proprietary processes. Data minimization, access control, retention limits, encryption, and documented purpose are essential. Applicable privacy and workplace regulations must be considered before deployment.

Bias can arise from under-represented operators, skin tones, product variants, suppliers, or environmental conditions. Performance should be stratified across relevant groups and conditions. Dataset provenance, annotation instructions, consent or legal basis, and model versions should be documented.

Adversarial patches, image tampering, camera obstruction, and data poisoning are security risks. Physical access to cameras and update interfaces should be controlled. A model card or system dossier should state intended use, limitations, training data characteristics, metrics, known failure modes, and change history. Human accountability must remain clear: automation changes decisions, but it does not eliminate engineering responsibility.

Best-Practice Design Procedure

1. Define the operational decision, error costs, latency, and acceptance criteria.
2. Control optics, field of view, focus, exposure, and illumination before increasing model complexity.
3. Collect representative data, including difficult normal cases and failure modes.
4. Split data by physical unit, batch, site, or time to prevent leakage.
5. Establish a simple baseline before evaluating complex architectures.
6. Use transfer learning and augmentation only when their assumptions fit the domain.
7. Select metrics and thresholds according to consequences, prevalence, and review capacity.
8. Test robustness, calibration, latency, cybersecurity, and human interaction.
9. Validate on deployment hardware and under production conditions.
10. Monitor drift, investigate overrides, and retrain through controlled change management.

Chapter Summary

Contemporary vision AI learns visual representations using CNNs, transformers, or hybrid architectures. CNNs exploit local structure and are often efficient; transformers model long-range relationships but commonly benefit from large-scale pretraining. Successful systems depend on representative data, leakage-free evaluation, suitable augmentation, transfer learning, and careful handling of synthetic data.

Classification, detection, segmentation, pose estimation, OCR, and anomaly detection require different outputs and metrics. Accuracy alone is inadequate for imbalanced or safety-sensitive problems. Precision, recall, F1, IoU, Dice, mAP, CER, calibration, robustness, and operational latency should be selected according to the decision.

Foundation models can accelerate annotation and support open-vocabulary tasks, but domain mismatch and unreliable confidence require validation. Explainability can expose shortcut learning but is not causal proof. Deployment is a socio-technical engineering problem involving optics, computation, actuators, people, safety, privacy, security, and monitoring. Dependable performance comes from disciplined system design rather than architecture choice alone.

Key Terms

CNN: Neural network using learned local filters to process spatial data.

Vision transformer: Transformer architecture that processes image patches with attention.

Transfer learning: Reusing a pretrained model for a new target task or domain.

Data augmentation: Label-preserving transformation used to increase training variation.

Synthetic data: Artificially generated data used for training or testing.

Feature map: Spatial array of activations produced by a neural-network layer.

Receptive field: Input region that can influence a particular activation.

Attention: Mechanism that weights relationships among tokens or image patches.

Classification: Assigning one or more labels to an image or region.

Object detection: Predicting object classes and bounding boxes.

Semantic segmentation: Assigning a class to every image pixel.

Instance segmentation: Producing separate masks for individual object instances.

Pose estimation: Estimating keypoints, body configuration, or object pose.

OCR: Optical character recognition from images.

Anomaly detection: Identifying observations that differ from learned normal behavior.

IoU: Ratio of intersection area to union area for two regions.

Precision: Fraction of positive predictions that are correct.

Recall: Fraction of actual positives that are detected.

Calibration: Agreement between predicted confidence and observed correctness.

Distribution shift: Change between training conditions and deployment conditions.

Review Questions

Short Answer Questions

1. Why can accuracy be misleading for a rare-defect inspection problem?
2. Distinguish semantic segmentation from instance segmentation.
3. What is the purpose of a validation set?
4. State two risks associated with synthetic training data.
5. Why should a vision model be evaluated on deployment hardware?

Long Answer Questions

1. Compare CNNs and vision transformers in terms of inductive bias, data requirements, computation, and industrial applications.
2. Explain how transfer learning, augmentation, and regularization can reduce overfitting, including situations in which each may fail.
3. Describe a complete evaluation plan for an object-detection system used on a production line.
4. Explain why OCR confidence alone should not determine acceptance of a serialized product.
5. Discuss the role and limitations of explainable AI in diagnosing shortcut learning.

Analytical / Application Questions

1. A factory has 10,000 parts, of which 100 are defective. A classifier produces 90 true positives, 20 false positives, 10 false negatives, and 9,880 true negatives. Calculate precision, recall, and F1 score, and explain whether accuracy alone gives a satisfactory assessment.
2. Design a vision system for robot picking of mixed parts. Specify the camera arrangement, task type, model family, pose representation, latency requirements, validation data, and failure response.
3. An anomaly detector flags many new products after a lighting upgrade, although operators find them acceptable. Diagnose possible causes and propose a controlled investigation and remediation plan.

Multiple Choice Questions with Answers

Question 1

Which metric directly measures the overlap between a predicted bounding box and its ground-truth box?

A. Intersection over Union

B. Character error rate

C. Recall

D. Calibration error

Correct Answer: A. Intersection over Union

Question 2

Why is batch-level splitting often preferable to random frame-level splitting for production video data?

A. It increases image resolution

B. It prevents correlated frames from leaking across splits

C. It removes the need for labels

D. It guarantees perfect calibration

Correct Answer: B. It prevents correlated frames from leaking across splits

Question 3

In scaled dot-product attention, what is the purpose of dividing by √d_k?

A. To increase image brightness

B. To remove positional information

C. To control the scale of dot products for stable softmax behavior

D. To convert pixels into bounding boxes

Correct Answer: C. To control the scale of dot products for stable softmax behavior

Question 4

Which augmentation is potentially invalid when left-right orientation carries meaning?

A. Small brightness variation

B. Moderate sensor noise

C. Horizontal flipping

D. Slight translation

Correct Answer: C. Horizontal flipping

Question 5

A model has TP = 90 and FP = 20. What is its precision?

A. 0.692

B. 0.818

C. 0.900

D. 0.980

Correct Answer: B. 0.818

Question 6

What is a principal risk of relying exclusively on synthetic images?

A. The model cannot have trainable parameters

B. Synthetic images always have too few pixels

C. The simulation-to-reality gap may cause the model to learn unrealistic cues

D. Synthetic data cannot contain labels

Correct Answer: C. The simulation-to-reality gap may cause the model to learn unrealistic cues

Question 7

Which task requires distinguishing separate objects that belong to the same class?

A. Image-level classification

B. Semantic segmentation

C. Instance segmentation

D. Image normalization

Correct Answer: C. Instance segmentation

Question 8

What does calibration describe in a classification system?

A. Whether confidence values correspond to observed correctness frequencies

B. Whether the lens is physically focused

C. Whether every class has equal size

D. Whether an image has three channels

Correct Answer: A. Whether confidence values correspond to observed correctness frequencies

Question 9

Which statement about saliency maps is most accurate?

A. They prove the highlighted pixels caused the decision

B. They can help diagnose shortcut features but require careful interpretation

C. They replace test-set evaluation

D. They guarantee fairness across populations

Correct Answer: B. They can help diagnose shortcut features but require careful interpretation

Question 10

For an OCR system used for traceability, why should recognized text be checked against syntax or checksums?

A. Visual confidence alone may not detect a plausible but invalid identifier

B. Checksums increase camera frame rate

C. Syntax checks remove the need for training data

D. OCR cannot output character strings

Correct Answer: A. Visual confidence alone may not detect a plausible but invalid identifier