Robot Perception and Machine Vision

Introduction

A robot can move only as intelligently as it can sense and interpret its environment. Robot perception converts measurements from cameras and other sensors into estimates of objects, surfaces, motion, and spatial relationships. Machine vision is the branch of robot perception concerned primarily with image-based sensing, especially visible-light and infrared cameras.

A camera does not measure objects directly. It records projections of three-dimensional surfaces onto a two-dimensional image plane. Perception therefore involves an inference problem: the robot must estimate scene properties from incomplete, noisy, and often ambiguous measurements. This chapter develops the geometric, algorithmic, and statistical foundations required to design such systems.

The subject is important in contemporary mechanical engineering because vision is now central to industrial inspection, autonomous vehicles, warehouse robotics, agricultural machines, medical devices, and human–robot collaboration. Effective systems require more than a neural network. They require appropriate sensing, calibration, image processing, coordinate transformations, uncertainty analysis, validation data, and safe integration with control systems.

Learning objectives

After studying this chapter, the reader should be able to:

1. Explain image formation using the pinhole camera model.
2. Transform points among camera, robot, and world coordinate frames.
3. Perform and evaluate intrinsic and extrinsic camera calibration.
4. Select suitable image-processing and feature-extraction methods.
5. Compare object detection, segmentation, recognition, and pose estimation.
6. Explain stereo, depth, structured-light, and time-of-flight perception.
7. Assess deep-learning vision systems using appropriate metrics and datasets.
8. Propagate uncertainty and identify sources of perception failure.
9. Design a vision pipeline for a practical robotic application.

1. Perception as an estimation pipeline

A robot-vision system can be represented as a sequence:

scene → illumination and optics → image acquisition → preprocessing → features or learned representation → recognition and localization → uncertainty estimate → planning and control

The output may be a class label, image coordinates, a 3D position, an object pose, a surface model, or a tracking estimate. Each output has a confidence or uncertainty that should influence the robot's behavior. A grasp planner, for example, should treat a high-confidence pose on a rigid object differently from an uncertain pose obtained under glare or occlusion.

Important terminology includes:

• Pixel: a discrete image sample, usually indexed by row and column.
• Resolution: the number of pixels in an image.
• Field of view: the angular extent visible to the camera.
• Region of interest: a selected image area processed in detail.
• Detection: locating instances of objects and assigning classes.
• Recognition: identifying what an object or scene is.
• Segmentation: assigning a class or instance label to pixels.
• Localization: estimating an object's position.
• Pose estimation: estimating position and orientation.
• Tracking: estimating how an object or camera state changes over time.

A useful engineering distinction is between measurement, interpretation, and decision. The image is a measurement; object recognition is interpretation; grasping or collision avoidance is a decision. Errors can arise at all three levels.

2. Camera geometry

2.1 The pinhole model

In the ideal pinhole model, a 3D point in camera coordinates is P_c=(X,Y,Z)^T, with Z>0. Its perspective projection onto image coordinates is

Here, x,y are pixel coordinates, f_x,f_y are focal lengths measured in pixels, and c_x,c_y are the principal-point coordinates. The focal lengths incorporate physical focal length and pixel scale. The equations assume an ideal pinhole camera and no lens distortion.

The homogeneous form is

where s is a nonzero scale factor, γ is the skew parameter, and K is the intrinsic matrix. The usual rectangular-pixel camera has γ=0. The matrix maps camera-coordinate points to pixels; it does not by itself provide depth.

A 3D point expressed in a world frame is first transformed into the camera frame:

where R_cw is a 3 × 3 rotation matrix and t_cw is a translation vector. Both must use consistent length units. The complete projection is therefore s p=K(R_cw P_w+t_cw).

A rotation matrix must satisfy R^TR=I and det(R)=+1. Robot software commonly represents the same rigid transformation by a homogeneous matrix:

The inverse transformation is

This inverse is valid because a rigid transformation preserves distances and has an orthonormal rotation matrix.

Figure: Perspective projection maps a 3D point along an optical ray to an image point.

2.2 Distortion and field of view

Real lenses introduce radial and tangential distortion. A common radial model uses normalized coordinates x_n=X/Z and y_n=Y/Z, radius r²=x_n²+y_n², and coefficients k₁,k₂,k₃. The distorted coordinates include radial terms such as

Tangential terms may be added for decentered lenses. Calibration software estimates these parameters and supplies an undistortion operation. Distortion correction is especially important for accurate measurement near image boundaries.

For a sensor dimension w and focal length f, the horizontal field of view is

This assumes a rectilinear lens and describes the angle subtended by the sensor width. A wider field of view increases coverage but generally reduces pixels per object and may increase distortion.

3. Calibration and coordinate frames

3.1 Intrinsic calibration

Intrinsic calibration estimates focal lengths, principal point, skew, and distortion. A planar checkerboard is moved through multiple orientations. Known board points are projected into the image, and parameters are selected to minimize reprojection error:

Here, p_i is a measured image point, P_i is a known calibration-board point, π is the distorted projection function, d contains distortion parameters, and θ contains all unknown parameters. The sum covers points and board views. This is a nonlinear least-squares objective; good initial estimates and diverse views improve reliability.

A low average reprojection error is useful but not sufficient. Errors may be systematically concentrated at the image edges, or the board may be too small to constrain distortion. Calibration should therefore be inspected spatially and repeated after changing lens focus, zoom, resolution, or camera mounting.

3.2 Hand–eye and robot–camera calibration

A robot may carry the camera (eye-in-hand) or observe it from a fixed location (eye-to-hand). Calibration determines the rigid transform between camera and robot frames. In hand–eye calibration, a standard formulation is

where A_i is a measured robot motion, B_i is a measured camera or target motion, and X is the unknown fixed camera–robot transform. The motions must be sufficiently varied; repeated rotations about nearly one axis create poorly conditioned estimates.

Frame conventions must be documented explicitly. A common practical error is confusing “camera pose in the robot frame” with “robot pose in the camera frame.” Unit tests should transform known points forward and backward and verify that the recovered coordinates agree within tolerance.

4. Image formation and preprocessing

Image quality depends on illumination, exposure, optics, sensor noise, motion, and compression. Exposure controls the amount of light accumulated, while gain amplifies both signal and noise. Saturated pixels contain little recoverable intensity information. Mechanical designers should therefore address lighting before attempting sophisticated software.

Typical preprocessing operations include:

• white-balance or colour correction;
• conversion between RGB, HSV, grayscale, and other colour spaces;
• denoising using Gaussian, median, or bilateral filters;
• contrast normalization and histogram equalization;
• geometric rectification;
• morphological erosion, dilation, opening, and closing;
• masking irrelevant regions.

A Gaussian filter smooths high-frequency noise but also blurs edges. A median filter is often effective against isolated impulse noise. Morphological operations use a structuring element and are useful for removing small regions or closing narrow gaps, but their scale must match the image resolution and object geometry.

Edges identify rapid intensity changes. Gradient operators estimate local change, while the Canny method combines smoothing, gradient calculation, nonmaximum suppression, thresholding, and edge linking. Edge detection is valuable for geometric inspection but can fail under shadows, texture, or weak contrast.

Thresholding can create a binary mask. A global threshold uses one intensity value; adaptive thresholding varies the threshold spatially. Segmentation based only on colour is efficient in controlled lighting but is vulnerable to illumination changes, reflections, and object colour similarity.

5. Features, matching, and motion

A feature is a measurable image structure useful for correspondence or recognition. Corners are locations where intensity changes in two directions. Local descriptors encode the neighbourhood around a keypoint so that corresponding points can be matched between images.

Feature methods differ in invariance and computational cost. SIFT-like descriptors are relatively robust to scale and rotation. ORB provides a faster binary descriptor suitable for many embedded applications. Template matching is simple when appearance and scale are controlled, but it is less robust to viewpoint and deformation.

False matches should be rejected. The ratio test compares the nearest and second-nearest descriptor distances. Geometric verification can use RANSAC, which repeatedly fits a model from minimal samples and selects the model with the largest inlier set. RANSAC is robust to outliers, but its result depends on the inlier threshold, sample count, and scene geometry.

For a moving monocular camera, two views constrain image correspondences through the epipolar relationship:

where x₁ and x₂ are homogeneous image points and F is the fundamental matrix. The equation assumes corresponding points arise from the same static 3D scene and ideal point correspondences after appropriate modelling. It constrains a match to an epipolar line; it does not by itself provide an absolute metric scale.

6. Object recognition and pose estimation

A vision system should distinguish among four tasks:

1. Image classification: assigns one or more labels to the image.
2. Object detection: returns bounding boxes and classes.
3. Semantic segmentation: labels each pixel by class.
4. Instance segmentation: separates individual objects of the same class.

For manipulation, a bounding box may be insufficient. The robot may require a 6D pose, consisting of three translations and three rotational degrees of freedom. Pose estimation can use known geometric models, keypoints, depth data, or learned pose networks. Occlusion, reflective surfaces, symmetry, and textureless geometry are major difficulties.

A rigid transformation preserves object geometry. If object points are P_i and measured points are q_i, rigid registration seeks R,t minimizing

subject to R^TR=I and det(R)=1. This least-squares problem can be solved by singular-value decomposition when correspondences are known. Iterative closest point (ICP) alternates correspondence assignment and transformation estimation; it requires a reasonable initial alignment and can converge to a wrong local minimum.

7. Three-dimensional perception

7.1 Stereo vision

A calibrated stereo pair estimates depth from disparity. For parallel cameras with focal length f, baseline B, and disparity d=u_L−u_R,

Here, Z and B use the same length unit, while f and d are in pixels. The formula assumes rectified, parallel cameras and correctly matched points. Depth uncertainty increases rapidly as disparity becomes small. Differentiating gives the approximate sensitivity

Thus, distant objects are intrinsically harder to measure accurately. Increasing baseline or focal length improves depth sensitivity but may reduce overlap or field of view.

7.2 Other depth sensors

Structured-light cameras project a known pattern and infer depth from its deformation. They can provide dense depth at short range but may fail outdoors or under strong ambient light. Time-of-flight cameras estimate distance from the travel time or phase of emitted light; multipath reflections and surface reflectivity can cause errors. LiDAR provides accurate range geometry over useful distances but may have lower image resolution and higher cost.

Sensor selection depends on working distance, required accuracy, surface properties, environmental lighting, latency, computational resources, and safety. Combining RGB and depth can improve object recognition and geometric localization, but calibration and synchronization become additional engineering requirements.

8. Deep-learning vision

Convolutional neural networks learn hierarchical image representations from labelled examples. Modern detectors generally predict object locations and class probabilities, while segmentation networks predict pixel-level labels. Vision transformers and multimodal models are increasingly used, but the engineering principles remain: define the task, collect representative data, train without leakage, validate on unseen conditions, and monitor deployment performance.

For a binary classifier, precision and recall are

where TP, FP, and FN denote true positives, false positives, and false negatives. The F1 score is

These metrics are meaningful only with a clearly defined positive class and decision threshold. Detection systems commonly use intersection over union (IoU):

A detection is counted as correct when IoU exceeds a stated threshold and the class is correct. Mean average precision summarizes precision–recall performance across classes and thresholds, but it should not replace application-specific measures such as missed-grasp rate, localization error, latency, and safety response time.

Dataset limitations are often more important than model architecture. Problems include class imbalance, duplicated frames across training and test sets, inaccurate labels, insufficient examples of rare failures, domain shift, camera changes, seasonal variation, and synthetic-to-real differences. Splitting data by video frame can overestimate performance because adjacent frames are highly correlated. A stronger test set is separated by scene, object instance, operator, location, and time where relevant.

Data augmentation can simulate modest changes in brightness, scale, crop, blur, and orientation. It cannot reliably reproduce every real-world effect, and physically implausible augmentation may harm learning. A deployment system should log uncertain and failed cases, protect personal data, and periodically evaluate for performance drift.

9. Uncertainty and reliability

Vision uncertainty has several sources: sensor noise, calibration error, quantization, occlusion, ambiguous appearance, model approximation, and changing environments. A covariance matrix represents uncertainty in multiple estimated variables. For a differentiable function y=g(x), a first-order approximation gives

where Σ_x and Σ_y are covariance matrices and J_g is the Jacobian of g evaluated at the estimate. This approximation assumes errors are small enough for local linearization and describes random uncertainty; systematic bias must be assessed separately.

For stereo depth Z=fB/d, if disparity uncertainty is σ_d, then approximately

assuming focal length and baseline errors are negligible and disparity error is small. In a real system, calibration and correspondence errors should also be included.

A safe robot should not silently treat low confidence as certainty. Useful responses include requesting another view, slowing down, illuminating the scene, choosing a conservative grasp, or stopping. Confidence thresholds must be selected using the costs of false positives and false negatives, not merely by maximizing an abstract benchmark score.

10. Illustrative case: vision-guided bin picking

Background

Consider an illustrative robotic cell in which a six-axis manipulator picks metal fittings from a bin and places them into trays. The parts have moderate shape similarity, occasional overlap, and specular surfaces. A wrist-mounted RGB-D camera observes the bin.

Problem

The robot must identify each fitting, estimate a collision-free grasp pose, and complete the cycle within 12 seconds. Failure can occur through false detection, inaccurate depth near reflective surfaces, occlusion, or collision with neighbouring parts.

Analysis

A 2D detector alone cannot determine whether a visible part is reachable or whether its grasp surface is occluded. Depth enables geometric filtering, but reflective metal produces invalid or biased measurements. The camera must also be calibrated to the robot flange, and the combined latency of image capture, inference, and robot motion must be measured.

Approach and solution

The cell uses diffuse lighting and a matte bin to reduce reflections. Intrinsic calibration is performed at the operating resolution, followed by hand–eye calibration. The software pipeline is:

1. capture synchronized colour and depth images;
2. mask the bin boundary and remove invalid depth;
3. detect candidate parts with an instance-segmentation model;
4. reject masks with low confidence or implausible size;
5. estimate a 3D pose from valid depth and a fitted geometric model;
6. propagate the camera-to-robot transform;
7. score candidate grasps using clearance, pose uncertainty, and reachability;
8. approach slowly and verify grasp success with force or gripper sensing.

The dataset contains parts at different orientations, overlaps, fill levels, lighting conditions, and camera viewpoints. Entire scenes, rather than individual frames, are separated between training and testing. Performance is evaluated using detection precision and recall, 3D position error, orientation error, cycle time, and successful picks per hour.

Outcome

The system achieves reliable picking for nonreflective parts but requests a second view for uncertain reflective surfaces. This reduces throughput slightly while preventing unsafe or unsuccessful picks. The example illustrates that perception quality is a property of the complete sensing, calibration, data, algorithm, and control system—not of the neural network alone.

Learning points

• Lighting and fixture design can be more effective than algorithmic complexity.
• Calibration must be validated in the actual working volume.
• Application metrics should include operational success, not only image metrics.
• Uncertainty should affect planning and robot speed.
• Dataset diversity must represent deployment conditions and failure cases.

11. Challenges, ethics, and professional practice

Vision systems fail under occlusion, glare, fog, dust, motion blur, poor exposure, unusual objects, and distribution shift. Mechanical vibration can change calibration. Thermal expansion can alter camera mounting. Network delays can make an apparently accurate observation obsolete.

Best practice includes rigid mounting, controlled illumination, exposure monitoring, timestamp synchronization, calibration checks, versioned datasets, independent test scenes, quantitative acceptance criteria, and fault-safe behavior. Perception should be integrated with conventional sensors such as force, proximity, encoders, and safety scanners rather than assumed to be infallible.

Cameras may collect images of workers or members of the public. Engineers must consider lawful data collection, informed notice where required, access control, retention limits, anonymization, cybersecurity, and bias. Facial or biometric identification requires especially careful governance. Safety-related systems should follow applicable machinery, functional-safety, and data-protection requirements; the exact standard depends on jurisdiction and application.

Contemporary developments include event cameras for high-speed motion, neural implicit 3D representations, foundation vision models, self-supervised representation learning, vision-language-action systems, and edge inference hardware. These technologies can reduce labelling effort or latency, but they also introduce new concerns about explainability, computational demand, model updates, and validation outside the training distribution. Emerging methods should be evaluated against a conventional baseline and a clearly defined operational requirement.

Chapter Summary

Robot perception estimates useful properties of the physical world from imperfect sensor measurements. The pinhole model explains perspective projection, while intrinsic parameters describe camera optics and extrinsic parameters connect coordinate frames. Distortion correction, calibration, and frame conventions are essential for accurate robot manipulation.

Image processing can improve signal quality and expose edges, regions, and features, but lighting and optics strongly influence success. Recognition may involve classification, detection, segmentation, tracking, or 6D pose estimation. Stereo depth follows Z=fB/d, and its uncertainty grows rapidly with distance when disparity becomes small. Structured light, time of flight, and LiDAR provide alternatives with different environmental limitations.

Deep-learning performance must be assessed using precision, recall, F1, IoU, and application-specific measures. Dataset leakage, class imbalance, inaccurate labels, and domain shift can invalidate apparently strong results. Uncertainty estimation, conservative decision-making, independent testing, and multimodal sensing are central to reliable deployment. A professional vision system is therefore an engineered perception-and-control system, not merely an image classifier.

Key Terms

Camera intrinsic matrix: Parameters describing focal length, principal point, and skew.

Extrinsic parameters: Rotation and translation relating a camera to another coordinate frame.

Pinhole model: Ideal perspective model mapping 3D points to image points.

Homogeneous coordinates: Coordinate representation that expresses projection and rigid transformations compactly.

Lens distortion: Deviation of real lens projection from the ideal pinhole model.

Calibration: Estimation of camera, distortion, or inter-frame parameters.

Hand–eye calibration: Estimation of the rigid relationship between a robot and camera.

Feature: Image structure used for matching, measurement, or recognition.

Descriptor: Numerical representation of a feature neighbourhood.

RANSAC: Robust model-fitting method that tolerates outlier correspondences.

Detection: Estimation of object classes and image locations.

Segmentation: Assignment of labels to image pixels.

Pose estimation: Estimation of an object's position and orientation.

Disparity: Difference between corresponding image coordinates in stereo images.

Structured light: Depth sensing using a projected known pattern.

Intersection over union: Ratio of overlap area to union area for predicted and true regions.

Domain shift: Change between training conditions and deployment conditions.

Covariance: Matrix describing uncertainty and correlations among estimated variables.

Epipolar geometry: Geometric relationship between views of a common 3D scene.

Uncertainty propagation: Estimation of output uncertainty from uncertain inputs.

Review Questions

Short Answer Questions

1. What physical quantities are represented by the intrinsic matrix of a camera?
2. Why is camera calibration necessary before using image coordinates for robot motion?
3. Distinguish object detection from instance segmentation.
4. State two causes of failure in stereo depth estimation.
5. Why should a robot use perception confidence during planning?

Long Answer Questions

1. Derive the pinhole projection equations and explain the role of depth.
2. Describe a complete intrinsic and hand–eye calibration procedure, including validation tests.
3. Compare stereo, structured-light, time-of-flight, and LiDAR sensing for industrial robotics.
4. Explain how dataset leakage and domain shift can produce misleading deep-learning results.
5. Discuss how image processing, feature matching, and geometric verification can be combined for visual localization.

Analytical / Application Questions

1. A stereo camera has f=800 pixels, B=0.12 m, and measured disparity d=48 pixels. Calculate the estimated depth and explain how a disparity error of 1 pixel affects the result approximately.
2. Design a perception pipeline for a robot sorting objects that vary in colour, shape, and surface reflectivity. Justify the sensor, lighting, algorithm, metrics, and failure response.
3. A detector has TP=180, FP=20, and FN=30. Calculate precision, recall, and F1, then explain whether these metrics alone are sufficient for approving a safety-critical robot application.

Multiple Choice Questions with Answers

Question 1

In the ideal pinhole camera model, which variable most directly determines the scale of a projected object?

A. The object's depth Z

B. The image file format

C. The pixel colour space

D. The number of classes in a detector

Correct Answer: A. The object's depth Z

Question 2

Which matrix maps a world-frame point into the camera frame in a rigid transformation?

A. The intrinsic matrix only

B. A rotation and translation transformation

C. The confusion matrix

D. The disparity histogram

Correct Answer: B. A rotation and translation transformation

Question 3

For rectified stereo cameras, depth is calculated as Z=fB/d. If f=600 pixels, B=0.10 m, and d=30 pixels, what is Z?

A. 0.5 m

B. 1.0 m

C. 2.0 m

D. 18 m

Correct Answer: C. 2.0 m

Question 4

Which metric is the ratio of intersection area to union area for predicted and true regions?

A. Recall

B. F1 score

C. Intersection over union

D. Calibration gain

Correct Answer: C. Intersection over union

Question 5

What is a principal danger of splitting adjacent video frames randomly between training and test sets?

A. It removes all image labels

B. It can produce overly optimistic performance

C. It prevents any neural-network training

D. It guarantees excessive lens distortion

Correct Answer: B. It can produce overly optimistic performance

Question 6

Which method is specifically designed to estimate a rigid transformation from corresponding 3D points using least squares?

A. Rigid registration with rotation constraints

B. Histogram equalization

C. Global thresholding

D. White balancing

Correct Answer: A. Rigid registration with rotation constraints

Question 7

If stereo disparity becomes smaller while all other parameters remain fixed, the estimated depth generally becomes:

A. Smaller and more precise

B. Larger and more sensitive to disparity error

C. Exactly zero

D. Independent of focal length

Correct Answer: B. Larger and more sensitive to disparity error

Question 8

Which statement best describes hand–eye calibration?

A. It estimates the relationship between a camera and a robot

B. It converts RGB images into grayscale

C. It removes all classification errors

D. It measures only camera exposure

Correct Answer: A. It estimates the relationship between a camera and a robot

Question 9

Given TP=90, FP=10, and FN=20, what is the recall?

A. 0.60

B. 0.75

C. 0.82

D. 0.90

Correct Answer: C. 0.82

Question 10

Which response is most appropriate when a robot obtains a low-confidence object pose near a human worker?

A. Increase speed without further sensing

B. Ignore the confidence value

C. Use a conservative response such as re-observation or stopping

D. Randomly select a grasp

Correct Answer: C. Use a conservative response such as re-observation or stopping