Foundations of Computer Vision

Introduction and Context

Computer vision is the field concerned with enabling machines to acquire, process, interpret, and act upon visual information. A computer-vision system may receive images, video, depth maps, thermal measurements, or other sensor data and transform them into descriptions, predictions, or decisions. The central challenge is that visual data are rich but ambiguous: the same object can appear different because of illumination, viewpoint, scale, occlusion, motion, or background clutter.

Computer vision is therefore not merely the study of digital images. It combines sensing, geometry, signal processing, statistical inference, machine learning, artificial intelligence, and domain knowledge. A useful abstraction is:

visual world → sensor → image or video representation → computational analysis → interpretation → action

For example, an autonomous vehicle may capture camera images, identify road boundaries and pedestrians, estimate their positions and motion, and then use those estimates for braking or steering. A medical system may analyze a scan to highlight a suspicious region, while a manufacturing system may identify surface defects.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Define computer vision and explain its relationship to image processing, computer graphics, machine learning, and human vision.

2. Describe major historical developments in computer vision.

3. Use essential terminology involving images, features, objects, scenes, models, and predictions.

4. Classify computer-vision tasks and distinguish classification, detection, segmentation, recognition, and estimation.

5. Explain a typical computer-vision workflow from data acquisition to deployment.

6. Identify important applications, limitations, ethical risks, and professional responsibilities.

7. Interpret basic evaluation measures and reason about vision-system design choices.

1. What Computer Vision Seeks to Do

An image is a two-dimensional arrangement of measurements. In a grayscale image, each pixel commonly stores an intensity value; in a color image, each pixel may store three channel values such as red, green, and blue. A digital image can be represented as a function I(x,y), where x and y denote spatial coordinates and I denotes measured intensity. A color image is more accurately represented as I(x,y,c), where c identifies a color channel.

The image is not the scene itself. It is a projection or measurement of a three-dimensional world through a sensing process. Consequently, different scenes may produce similar images, and the same scene may produce very different images. This is called the inverse-problem character of vision: the system attempts to infer causes from effects.

A vision system may perform several levels of interpretation:

• Sensing: acquiring an image or video stream.
• Enhancement: improving visibility or correcting sensor artifacts.
• Measurement: extracting edges, distances, shapes, or motion.
• Recognition: assigning meaning to patterns or objects.
• Decision-making: selecting an action based on visual evidence.

Not every system requires all levels. A barcode reader may need reliable pattern decoding but little semantic understanding. A surgical-navigation system may require geometric reconstruction, tracking, and uncertainty management.

2. Historical Development

The field developed through several overlapping periods rather than through a single invention.

2.1 Early foundations

Early work in the 1950s and 1960s explored image formation, pattern recognition, edge detection, and symbolic descriptions of scenes. Researchers investigated whether geometric primitives such as lines, circles, and corners could be extracted and combined into object models. Computing resources were limited, so systems were often designed for constrained environments.

2.2 Model-based and geometric vision

During the 1970s and 1980s, researchers developed mathematical models for camera projection, stereo vision, motion, shape, and object recognition. The goal was frequently to recover three-dimensional structure from two-dimensional images. Approaches based on edges, regions, contours, and manually designed features were influential. These methods remain important in robotics, calibration, metrology, and scientific imaging.

2.3 Statistical learning

From the 1990s onward, computer vision increasingly incorporated statistical pattern recognition. Local descriptors, classifiers, probabilistic models, and benchmark datasets supported more general recognition systems. A common pipeline extracted hand-designed features and passed them to a classifier such as a support vector machine or a decision-tree ensemble.

2.4 Deep learning

The 2010s saw rapid progress from convolutional neural networks trained on large labeled datasets and accelerated hardware. Deep models learned representations directly from data, often outperforming hand-engineered pipelines on image classification, detection, and segmentation benchmarks. Later developments included vision transformers, multimodal models, self-supervised learning, foundation models, and systems that combine visual and language information.

Progress has not eliminated fundamental difficulties. A model that performs well on a benchmark may fail under distribution shift, unusual weather, unfamiliar equipment, demographic imbalance, adversarial manipulation, or a new operational environment. Modern practice therefore emphasizes not only accuracy, but also robustness, calibration, interpretability, privacy, safety, and lifecycle monitoring.

3. Relationship to Neighboring Fields

3.1 Image processing

Image processing focuses primarily on transforming images or extracting measurements from them. Typical operations include denoising, sharpening, contrast adjustment, restoration, compression, filtering, and geometric transformation. Image processing may be used without assigning semantic meaning. For example, resizing an image is image processing; identifying a person in it is a computer-vision task.

The boundary is not absolute. Edge detection is traditionally image processing but is also a component of many vision systems. A useful distinction is one of emphasis: image processing asks, “How can the image be improved or measured?” Computer vision asks, “What does the visual data represent, and what can be inferred or done from it?”

3.2 Computer graphics

Computer graphics is primarily a forward problem: given a scene description, lighting model, materials, and camera, generate an image. Computer vision is primarily an inverse problem: given an image, infer scene properties. Graphics and vision are complementary. Graphics can produce synthetic training data, simulate sensors, and visualize reconstructed scenes. Vision can estimate the parameters needed to build a graphical representation.

3.3 Machine learning

Machine learning provides methods for learning relationships from data. Computer vision provides the visual domain, task definitions, representations, and evaluation context. Not all vision is machine learning: calibration and geometric reconstruction can use explicit models. Not all machine learning is vision: it is also applied to language, finance, recommendation, and scientific data.

In supervised learning, a model is trained using inputs paired with labels. In unsupervised or self-supervised learning, useful representations are learned without manually assigning every target label. In deployment, the model is only one part of a larger system involving sensors, preprocessing, decision thresholds, interfaces, and human oversight.

3.4 Human vision and biological perception

Human vision motivates many computational questions, but a computer-vision system is not necessarily a model of the human visual system. Biological vision uses complex neural structures, attention, prior knowledge, active movement, and multimodal interaction. Humans can recognize objects despite substantial variation, yet they also exhibit systematic biases and visual illusions. Computer systems may exceed human performance on narrowly defined tasks while lacking common-sense understanding and flexible context awareness.

4. Essential Terminology

A pixel is a discrete image sample. Resolution describes the number of samples or the spatial detail represented; it should not be confused with physical accuracy. A frame is one image in a video sequence. The field of view is the angular extent visible to a camera. Frame rate is the number of frames captured or processed per second.

An object is an entity of interest, such as a vehicle or cell. A scene is the broader visual environment. An instance is one occurrence of an object category. A class or category is a group sharing a specified semantic label.

A feature is a measurable property useful for analysis. Features may be low-level, such as intensity gradients and corners, or learned, such as internal neural-network representations. A descriptor is a numerical representation of an image region, object, or scene. A model is a computational or mathematical representation used to explain, predict, or classify data.

A label is the target annotation associated with an example. Ground truth is the reference annotation or measurement used for training or evaluation, although it may itself contain uncertainty or error. Inference is the process of applying a trained or specified model to new data. Training, validation, and test data should be separated to reduce overly optimistic evaluation.

False positives are incorrect positive predictions; false negatives are missed positive cases. Latency is the time required to produce a result. Throughput is the amount of data processed per unit time. Robustness concerns performance under relevant variation or disturbance. Domain shift occurs when deployment data differ from training data.

5. Taxonomy of Computer-Vision Tasks

Task categories often overlap, and a practical system may combine several of them.

5.1 Image classification

Classification assigns one or more labels to an entire image. In single-label classification, one principal class is selected. In multi-label classification, several classes may be present, such as “road,” “car,” and “pedestrian.” Classification does not normally provide the location of each object.

5.2 Object detection

Detection identifies object instances and estimates their locations, commonly using rectangular bounding boxes. A detection generally includes a class label, a box, and a confidence score. Detection is appropriate when the system needs to know both what objects exist and approximately where they are.

5.3 Semantic segmentation

Semantic segmentation assigns a class to each pixel, but pixels belonging to separate objects of the same class receive the same class label. It is useful for road scenes, agricultural regions, and medical structures.

5.4 Instance segmentation

Instance segmentation assigns pixels to individual object instances. Two people standing next to one another are represented as two separate masks, even though both have the class “person.”

5.5 Panoptic segmentation

Panoptic segmentation combines semantic and instance perspectives. It represents both countable objects, such as cars, and amorphous background regions, such as sky or road.

5.6 Localization and keypoint estimation

Localization may identify the position of a known object or estimate its bounding region. Keypoint estimation predicts meaningful points, such as joints of a human body, corners of a manufactured part, or anatomical landmarks.

5.7 Tracking and motion analysis

Tracking associates observations over time. A tracker may maintain an identity for each object, estimate velocity, and handle temporary occlusion. Optical flow estimates apparent motion of image regions between frames. Tracking is especially important in surveillance, robotics, sports analysis, and autonomous systems.

5.8 Depth, three-dimensional reconstruction, and pose

Depth estimation predicts distance from the camera. Stereo vision uses images from different viewpoints; the displacement between corresponding points is called disparity. Under a simplified rectified stereo model, depth is often expressed as

Z = fB/d,

where Z is depth, f is focal length expressed in compatible image units, B is the baseline between cameras, and d is disparity. This relationship assumes corresponding points have been correctly identified and that the camera geometry has been calibrated. As disparity decreases, depth uncertainty generally increases, particularly for distant objects.

Three-dimensional reconstruction estimates scene geometry from one or more images. Pose estimation determines the position and orientation of an object or camera. These tasks support augmented reality, robotics, inspection, and navigation.

5.9 Image retrieval, captioning, and visual question answering

Image retrieval searches for visually or semantically similar images. Captioning generates a textual description. Visual question answering combines image analysis with language reasoning to answer a question about an image. These tasks illustrate the growing connection between vision and multimodal AI, but generated language should not automatically be treated as verified fact.

6. A Typical Computer-Vision Workflow

A robust workflow begins with the operational problem rather than with a preferred algorithm.

Stage 1: Problem formulation

Define the decision to be supported, the target population, acceptable errors, latency requirements, operating conditions, and consequences of failure. “Recognize defects” is incomplete; a useful specification states which defects, at what size, under what illumination, and with what required recall.

Stage 2: Data acquisition and governance

Select sensors, viewpoints, illumination, resolution, and sampling frequency. Record relevant metadata. Obtain consent and establish lawful data-use procedures when people or sensitive environments are captured. Data should represent expected operating conditions, including difficult cases.

Stage 3: Annotation and quality control

Choose labels suitable for the task: image labels, boxes, masks, keypoints, tracks, or numerical measurements. Annotation guidelines should define ambiguous cases. Inter-annotator disagreement can reveal genuine uncertainty rather than merely poor labeling.

Stage 4: Preprocessing and representation

Typical operations include calibration, resizing, normalization, color conversion, denoising, augmentation, and region selection. Preprocessing must be applied consistently and must not leak information from the test set into training.

Stage 5: Model selection and training

Choose an explicit geometric method, classical machine-learning model, deep neural network, or hybrid approach according to data, constraints, explainability, and risk. Training should use suitable loss functions and regularization. Hyperparameters should be selected using validation data, not the final test set.

Stage 6: Evaluation

Evaluate with task-appropriate measures and report uncertainty or confidence intervals when feasible. Test performance should be broken down by relevant conditions, groups, environments, and failure modes. A single average score can conceal serious weaknesses.

Stage 7: Deployment and monitoring

Deployment includes hardware, software, interfaces, security, version control, and fallback procedures. Monitor data drift, calibration, latency, failure rates, and changes in the environment. A deployed model requires maintenance; it is not a finished product at the moment of training.

7. Evaluation Concepts and a Worked Example

For binary decisions, let TP denote true positives, TN true negatives, FP false positives, and FN false negatives. Precision and recall are defined as

Precision = TP/(TP + FP),

Recall = TP/(TP + FN),

provided the denominators are nonzero. Precision measures the proportion of predicted positives that are correct. Recall measures the proportion of actual positives that are detected.

The intersection over union (IoU), often used for boxes or masks, is

IoU = |P ∩ G| / |P ∪ G|,

where P is the predicted region and G is the ground-truth region. Since |P ∪ G| = |P| + |G| − |P ∩ G|, the formula can also be written as

IoU = intersection/(prediction area + ground-truth area − intersection).

Worked example. Suppose a predicted box has area 80 square units, the ground-truth box has area 100 square units, and their intersection is 60 square units. The union is 80 + 100 − 60 = 120 square units. Therefore,

IoU = 60/120 = 0.50.

This value is neither 0.60 nor 0.75. The subtraction of the intersection is essential because the overlapping area would otherwise be counted twice.

For a model producing confidence scores, changing the decision threshold usually creates a trade-off between precision and recall. The appropriate threshold depends on the cost of missed detections versus false alarms. In safety-critical inspection, a high recall requirement may be prioritized, followed by human review of flagged cases.

8. Applications and Professional Relevance

Computer vision is used in:

• Healthcare: image reconstruction, lesion measurement, cell analysis, surgical guidance, and radiological decision support.
• Manufacturing: defect inspection, dimensional measurement, robot guidance, and quality control.
• Transportation: traffic monitoring, driver assistance, pedestrian detection, mapping, and autonomous navigation.
• Agriculture: crop classification, disease detection, yield estimation, and precision spraying.
• Retail and logistics: inventory analysis, package sorting, shelf monitoring, and warehouse robotics.
• Security and public services: access control, identity verification, anomaly detection, and infrastructure inspection.
• Science and environmental monitoring: microscopy, astronomy, wildlife surveys, climate observation, and geological analysis.
• Media and accessibility: image search, caption generation, visual effects, and assistive technologies.

Professionals must translate business or scientific goals into measurable vision requirements. They must also understand the entire system: an accurate detector may still be unsuitable if its latency is excessive, its false alarms overwhelm staff, or its sensor fails under realistic conditions.

9. Illustrative Case Study: Automated Surface Inspection

Background. A manufacturer produces painted metal panels. Human inspectors examine each panel for scratches, incomplete coating, and dents. Inspection is slow and results vary with fatigue and lighting.

Problem. The company wants an automated system that flags likely defects without allowing defective panels to pass unnoticed. Panels move on a conveyor, and their orientation varies slightly.

Analysis. The task is not simply image classification. The system must locate defects and distinguish them from harmless texture variation. Missed defects are more costly than additional inspections, but excessive false alarms reduce productivity. The lighting must be stabilized because reflections can resemble scratches.

Approach. The engineering team installs diffuse, controlled illumination and a fixed camera. It collects images from multiple production batches, including known defect types and difficult non-defect examples. Defects are annotated with masks because area and shape matter. A segmentation model is trained, while a rule-based size filter removes extremely small isolated responses caused by sensor noise. Data are divided by production batch rather than by randomly mixing adjacent frames, reducing the risk that nearly identical images appear in both training and testing.

Evaluation reports recall by defect type, false alarms per panel, processing latency, and performance under changes in paint color. Borderline cases are sent to a human inspector. The deployed system stores examples of uncertain predictions for periodic review, with access controls protecting production information.

Outcome. The system improves consistency and reduces routine inspection workload, but it does not eliminate human responsibility. New coatings require validation, and cameras must be cleaned and recalibrated. The system is accepted as decision support rather than as an unquestionable authority.

Learning points. Sensor design can be as important as model architecture. The task definition determines annotation and metrics. Evaluation must reflect production structure. Human review, monitoring, and maintenance are part of the solution.

10. Challenges, Limitations, and Responsible Use

Ambiguity and incomplete information

Occlusion, shadows, reflections, low resolution, and viewpoint changes make visual inference uncertain. A confidence score is not a guarantee of correctness. Systems should expose uncertainty where decisions have serious consequences.

Dataset bias and distribution shift

If training data underrepresent environments or populations, performance may be unequal. Random splits can also exaggerate performance when neighboring video frames are highly similar. Evaluation should reflect deployment conditions and should include subgroup and stress testing.

Privacy and surveillance

Images may contain faces, biometrics, license plates, homes, workplaces, or medical information. Organizations should minimize collection, define retention periods, control access, document purpose, and comply with applicable privacy and data-protection law. Consent and notice may be required, especially for biometric identification.

Safety and accountability

A vision system used in medicine, transportation, employment, or public safety can affect rights and physical well-being. Responsibility should not be delegated vaguely to “the algorithm.” There must be clear ownership, escalation procedures, audit logs, human oversight where appropriate, and mechanisms for correction or appeal.

Security and adversarial behavior

Attackers may manipulate images, presentation conditions, data pipelines, or model interfaces. Threat modeling, access control, secure updates, input validation, and adversarial testing are necessary for high-consequence deployments.

Explainability and documentation

Documentation should state intended use, excluded use, training-data characteristics, evaluation conditions, known failure modes, thresholds, and monitoring plans. A saliency visualization may help investigation, but it is not automatically a causal explanation.

11. Contemporary Developments and Best Practices

Current research includes self-supervised representation learning, vision transformers, multimodal foundation models, 3D neural representations, synthetic data, continual learning, event-based cameras, and efficient models for edge devices. These technologies can reduce annotation requirements or broaden capabilities, but they also introduce concerns about computational cost, provenance, copyright, hallucinated descriptions, and difficult-to-audit training data.

Good practice includes:

1. Start with a precise use case and measurable acceptance criteria.

2. Design sensing and illumination before optimizing the model.

3. Establish annotation standards and measure label quality.

4. Separate training, validation, and test data by meaningful units such as person, site, time, or production batch.

5. Report multiple metrics and analyze failures, not only average accuracy.

6. Test robustness under realistic changes in lighting, viewpoint, hardware, and population.

7. Calibrate confidence scores when decisions depend on probability estimates.

8. Keep humans informed and provide safe fallback procedures for high-risk applications.

9. Monitor deployed performance and document model and data versions.

10. Treat privacy, fairness, security, and accessibility as design requirements.

Chapter Summary

Computer vision infers useful information about the visual world from images, video, and related sensor data. It is an inverse-problem discipline connected to image processing, computer graphics, machine learning, geometry, and biological vision. Its tasks range from enhancement and measurement to classification, detection, segmentation, tracking, depth estimation, reconstruction, pose estimation, and multimodal interpretation.

A practical workflow proceeds from problem formulation and responsible data acquisition through annotation, preprocessing, model development, evaluation, deployment, and monitoring. Metrics must match the task: precision and recall characterize binary decisions, while IoU measures overlap between predicted and reference regions. The corrected worked example demonstrates that areas of 80 and 100 with intersection 60 produce IoU 0.50 because the union is 120.

Computer vision supports healthcare, manufacturing, transport, agriculture, science, and accessibility. Nevertheless, ambiguity, domain shift, bias, privacy concerns, security threats, and safety risks limit automatic interpretation. Effective professionals therefore combine technical modeling with careful sensing, meaningful evaluation, governance, human oversight, and lifecycle management.

Key Terms

Pixel: A discrete sample representing image information at a spatial location.

Image: A structured measurement of a visual scene, commonly represented as an array of pixels.

Frame: One image in a video sequence.

Feature: A measurable or learned property useful for analysis.

Descriptor: A numerical representation of an image region, object, or scene.

Ground truth: A reference annotation or measurement used for training or evaluation.

Inference: Applying a model or computational procedure to new data.

Classification: Assigning one or more semantic labels to an image or input.

Object detection: Predicting object categories and their locations, often with bounding boxes.

Semantic segmentation: Assigning a class label to each pixel without separating same-class instances.

Instance segmentation: Assigning pixels to individual object instances.

Tracking: Associating observations of objects across time.

Optical flow: Estimated apparent motion of image points or regions between frames.

Depth: Distance from the sensor to a scene point along a defined measurement direction.

Pose estimation: Estimating an object’s or camera’s position and orientation.

Precision: The fraction of predicted positives that are correct.

Recall: The fraction of actual positives that are detected.

Intersection over union: The intersection area divided by the union area of predicted and reference regions.

Domain shift: A difference between training-data conditions and deployment-data conditions.

Latency: The time required to produce a computational result.

Review Questions

Short Answer Questions

1. Define computer vision and explain why it is commonly described as an inverse problem.

2. Distinguish image processing from computer vision using one example of each.

3. What is the difference between semantic segmentation and instance segmentation?

4. Define precision, recall, and intersection over union.

5. List four stages in a responsible computer-vision workflow.

Long Answer Questions

1. Discuss the historical development of computer vision from geometric methods to deep learning, including continuing limitations.

2. Compare computer vision with computer graphics, machine learning, and human vision.

3. Explain the major task categories in computer vision and describe an application for each.

4. Design a complete workflow for a vision system used in manufacturing inspection.

5. Analyze the ethical, privacy, safety, and governance issues involved in deploying vision systems in public environments.

Analytical / Application Questions

1. A predicted segmentation mask has area 150 pixels, a reference mask has area 120 pixels, and their intersection is 90 pixels. Calculate the IoU and explain the calculation.

2. A medical screening system detects 92 of 100 patients with a condition and incorrectly flags 18 patients without it. Calculate recall and precision, and discuss which additional information is needed to evaluate the system fully.

3. A pedestrian detector performs well in daylight but poorly at night and on rainy roads. Propose a data, sensor, evaluation, and deployment strategy to investigate and reduce this domain shift.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes computer vision from basic image processing?

A. Computer vision is concerned only with color images.

B. Computer vision seeks to infer meaning or properties of a scene, whereas image processing often transforms or measures image data.

C. Computer vision never uses mathematical models.

D. Image processing always requires machine learning.

Correct Answer: B. Computer vision seeks to infer meaning or properties of a scene, whereas image processing often transforms or measures image data.

Question 2

Which task assigns a semantic class to every pixel but does not distinguish separate objects of the same class?

A. Image classification

B. Object detection

C. Semantic segmentation

D. Object tracking

Correct Answer: C. Semantic segmentation

Question 3

A predicted region has area 80, a ground-truth region has area 100, and their intersection has area 60. What is the IoU?

A. 0.40

B. 0.50

C. 0.60

D. 0.75

Correct Answer: B. 0.50

Question 4

In the simplified rectified stereo relation Z = fB/d, what generally happens to estimated depth as disparity decreases while f and B remain fixed?

A. Estimated depth decreases.

B. Estimated depth remains exactly constant.

C. Estimated depth increases.

D. Depth becomes equal to disparity.

Correct Answer: C. Estimated depth increases.

Question 5

Which measure is the fraction of predicted positive cases that are actually positive?

A. Recall

B. Precision

C. Specificity

D. Latency

Correct Answer: B. Precision

Question 6

What is the principal purpose of separating training, validation, and test data?

A. To increase image resolution

B. To prevent evaluation from being overly optimistic and to support model selection responsibly

C. To guarantee that labels contain no errors

D. To eliminate the need for deployment monitoring

Correct Answer: B. To prevent evaluation from being overly optimistic and to support model selection responsibly

Question 7

Which task estimates the position and orientation of an object or camera?

A. Pose estimation

B. Image compression

C. Color correction

D. Semantic labeling only

Correct Answer: A. Pose estimation

Question 8

Why can controlled illumination be important in industrial visual inspection?

A. It guarantees perfect classification regardless of the model.

B. It reduces visual variation that could otherwise resemble defects or hide them.

C. It eliminates the need for ground-truth annotations.

D. It makes all images three-dimensional.

Correct Answer: B. It reduces visual variation that could otherwise resemble defects or hide them.

Question 9

A system detects 45 of 60 actual positive cases. What is its recall?

A. 0.25

B. 0.60

C. 0.75

D. 1.33

Correct Answer: C. 0.75

Question 10

Which practice is most appropriate for a high-consequence deployed vision system?

A. Use only the average benchmark score and ignore failure cases.

B. Allow the model to change without version records.

C. Monitor performance, document limitations, and provide appropriate human oversight or fallback procedures.

D. Collect as many personal images as possible without purpose limitation.

Correct Answer: C. Monitor performance, document limitations, and provide appropriate human oversight or fallback procedures.