Camera Models, Calibration, and Geometric Vision

Introduction

A machine-vision system does not measure the three-dimensional world directly. It measures patterns of light projected onto a two-dimensional sensor. The central task of geometric vision is therefore to establish a reliable relationship between image coordinates, camera coordinates, robot coordinates, and physical dimensions. This relationship supports inspection, metrology, localization, robot guidance, and augmented-reality applications.

A camera model describes image formation. Calibration estimates the parameters of that model. Geometric vision then uses the calibrated model to infer structure, motion, planes, and object pose. In mechatronics, these operations are especially important because the camera must often cooperate with actuators, fixtures, coordinate-measuring systems, and robot controllers.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Explain the pinhole camera model and homogeneous coordinates.
2. Distinguish intrinsic, extrinsic, and distortion parameters.
3. Apply coordinate transformations using rotation matrices and translations.
4. Describe calibration procedures and interpret reprojection error.
5. Use homography for planar mapping and perspective correction.
6. Estimate object pose from image measurements.
7. Explain hand–eye calibration for robot–camera integration.
8. Identify error sources, observability limitations, and good engineering practice.

1. Coordinate Systems and the Pinhole Model

A camera system commonly uses four coordinate frames:

• World frame W: a fixed reference frame attached to a workcell, table, or calibration target.
• Camera frame C: its origin is the optical centre; the z-axis points approximately along the optical axis.
• Image frame: coordinates measured on the sensor, usually in pixels.
• Robot or tool frame: a frame attached to a robot base, end effector, or tool.

A point in camera coordinates is written as [[M:\mathbf{X}_C=(X_C,Y_C,Z_C)^T]]. Under the ideal perspective model, its normalized image coordinates are

The division by depth explains perspective: objects farther from the camera occupy fewer pixels. The model assumes a single viewpoint, a thin lens or ideal pinhole, a rigid camera, and no lens distortion.

Pixel coordinates are obtained using the intrinsic matrix K:

Here, [[M:u,v]] are pixel coordinates, [[M:f_x,f_y]] are focal lengths measured in pixels, [[M:c_x,c_y]] is the principal point, [[M:s]] is the skew parameter, and [[M:s]] is a homogeneous scale factor in the equation. The semicolon notation denotes matrix rows. In most modern cameras, pixel axes are nearly orthogonal, so skew is set to zero.

If the physical focal length is [[M:f]] millimetres and pixel pitches are [[M:p_x,p_y]] millimetres per pixel, then [[M:f_x=f/p_x]] and [[M:f_y=f/p_y]]. Thus focal length in pixels is not necessarily equal in the two directions.

Homogeneous Coordinates

A two-dimensional point [[M:(u,v)]] is represented homogeneously as [[M:\tilde{\mathbf{p}}=(u,v,1)^T]]. Any non-zero scalar multiple represents the same point. Homogeneous coordinates allow translation, rotation, projection, and planar mapping to be expressed as matrix multiplication. The actual pixel point is recovered by dividing by the third coordinate.

2. Extrinsic Parameters and Rigid Transformations

Intrinsic parameters describe the camera internally. Extrinsic parameters describe its position and orientation relative to another frame. A rigid transformation from world coordinates to camera coordinates is

where [[M:R_{CW}]] is a 3 × 3 rotation matrix and [[M:\mathbf{t}_{CW}]] is a translation vector. The homogeneous form is

A valid rotation satisfies [[M:R^TR=I]] and [[M:\det(R)=+1]]. These conditions preserve lengths, angles, and handedness. The inverse transformation is

Transformations must be composed in the correct order. If a point is transformed from frame A to B and then B to C, then [[M:T_{CA}=T_{CB}T_{BA}]]. Matrix multiplication is not commutative.

Rotations may be represented by Euler angles, axis–angle vectors, or quaternions. Euler angles are intuitive but depend on convention and suffer from singularities. Quaternions avoid many singularities and are useful in robot software, but must be normalized. For a rotation of angle [[M:\theta]] about a unit axis [[M:\mathbf{a}]], Rodrigues' formula is

where [[M:[\mathbf{a}]_\times]] is the skew-symmetric cross-product matrix.

3. Lens Distortion

Real lenses depart from the pinhole model. The most common effects are radial and tangential distortion. Let [[M:(x,y)]] be ideal normalized coordinates and [[M:r^2=x^2+y^2]]. A common radial model is

A common tangential model is

The distorted normalized coordinates [[M:(x_d,y_d)]] are converted to pixels using [[M:K]]. Parameters [[M:k_1,k_2,k_3]] describe radial distortion and [[M:p_1,p_2]] tangential distortion. The exact model depends on the calibration library and lens type; a wide-angle or fisheye lens may require a different model.

Distortion is small near the principal point and usually larger near image boundaries. Ignoring it can create systematic errors in straight-line geometry, measurement, and pose estimation. Undistortion may be performed by numerically inverting the model or by remapping pixels through a precomputed lookup table.

4. Camera Calibration

Calibration estimates intrinsic parameters, distortion coefficients, and often camera pose relative to a known target. A planar checkerboard, circle grid, or coded target is commonly used because its geometry is known accurately.

For each observed target point, calibration minimizes reprojection error. If [[M:\mathbf{P}_i]] is a known three-dimensional target point, [[M:\hat{\mathbf{p}}_i]] is its predicted pixel position, and [[M:\mathbf{p}_i]] is the measured position, the objective is

A weighted version may be used when measurements have unequal uncertainty. The solution is generally nonlinear because projection contains division by depth and distortion contains powers of radius.

A practical calibration procedure is:

1. Select a target with accurately known geometry.
2. Acquire images at different positions, orientations, distances, and image regions.
3. Detect corners or target centres with subpixel refinement.
4. Estimate an initial camera model.
5. Refine parameters by nonlinear least squares.
6. Inspect residuals, not merely the average error.
7. Validate using independent images and physical measurements.

The root-mean-square reprojection error is

Its unit is pixels. A low value is useful but not sufficient: a biased target, poor image coverage, incorrect model, or overfitting can still produce unreliable results. Calibration quality should be judged relative to the application. A robot pick operation may tolerate several pixels, whereas precision metrology may require subpixel image localization and a carefully controlled optical setup.

Planar Calibration and Homography

For a calibration board lying in [[M:Z_W=0]], the projection equation becomes

where [[M:H]] is a 3 × 3 homography and [[M:\mathbf{r}_1,\mathbf{r}_2]] are the first two columns of the rotation matrix. Multiple views of the plane constrain the intrinsic parameters. This is the basis of widely used planar calibration methods.

Calibration Quality and Uncertainty

Important practices include using a rigid target, maintaining focus and exposure, covering the full image area, avoiding motion blur, and preserving the same camera focus and zoom after calibration. Uncertainty comes from corner localization, target manufacturing error, lens temperature, camera vibration, and parameter correlation. The principal point, focal length, and distortion coefficients can be strongly correlated when images do not provide sufficient viewpoint diversity.

5. Homography and Perspective Correction

A homography maps points on one plane to another image plane:

A homography has eight independent degrees of freedom because its nine elements are defined only up to a non-zero scale. At least four non-collinear point correspondences are required to estimate it. More points should be used in practice, with a robust estimator such as RANSAC to reject mismatches.

Homography is valid when the observed points lie on a single plane, when the camera undergoes pure rotation, or when the scene is sufficiently far away that depth variation is negligible. It is not a general three-dimensional mapping. Applying one homography to points at different depths creates geometric errors.

Perspective correction, or rectification, uses a homography to transform an oblique view of a planar object into a fronto-parallel view. This is useful for reading labels, measuring planar parts, inspecting printed patterns, and locating holes on a plate. The output image must be chosen with an appropriate scale; rectification does not create information that was not sampled by the camera.

Figure: A homography maps an oblique view of a plane to a rectified image; it does not recover general three-dimensional depth.

6. Pose Estimation

Pose estimation determines the rotation and translation of an object or target relative to the camera. Given known three-dimensional points and their image projections, the perspective-n-point problem (PnP) estimates [[M:R]] and [[M:\mathbf{t}]] in

The points should not be arranged in a degenerate configuration. Four or more well-distributed non-coplanar points generally improve robustness, although specialized algorithms exist for planar targets. The estimated pose should be refined by minimizing reprojection error and checked for positive depth: visible points should have [[M:Z_C>0]].

Pose accuracy depends on focal length, target size, point localization, viewpoint, and calibration. A small target occupying only a few pixels gives poor angular and translational precision. Large oblique viewing angles may also amplify uncertainty. Pose estimates should be filtered only when the motion model justifies filtering; smoothing cannot correct systematic calibration error.

Worked Example: Planar Scale and Measurement

Suppose a calibrated camera views a planar work surface. A homography maps a measured feature to a rectified image in which the scale is 0.20 mm per pixel. Two feature centres are separated by 37 pixels horizontally and 24 pixels vertically. The physical separation is

This calculation is valid because both points lie on the rectified plane and the scale is uniform in the selected image. It would not be valid for points at different depths or before correcting significant perspective distortion.

7. Camera–Robot Coordinate Integration

A vision-guided robot requires a transformation between camera and robot frames. If the camera is fixed in the workcell, the chain may be written

where [[M:B]] is the robot base frame. If the camera is mounted on the robot tool, the camera pose changes with robot motion. This is the eye-in-hand configuration.

The notation must be declared explicitly because different laboratories use different conventions. A transformation [[M:T_{AB}]] should mean “coordinates in B expressed in A” or, alternatively, “pose of B relative to A”; both conventions exist. Ambiguity is a major source of integration failure.

Hand–Eye Calibration

Hand–eye calibration estimates the fixed transformation between a camera and a robot hand. A common formulation is

Here, [[M:A]] represents a robot-hand motion between two robot poses, [[M:B]] represents the corresponding target motion measured by the camera, and [[M:X]] is the constant hand–eye transformation. The exact arrangement depends on frame definitions and eye-in-hand versus eye-to-hand configuration.

A practical procedure is:

1. Rigidly mount the camera and verify that the mount cannot flex.
2. Observe a fixed calibration target from many robot poses.
3. Record accurate robot poses and synchronized images.
4. Estimate target pose in each image using calibrated intrinsics.
5. Solve the hand–eye equations.
6. Validate by moving the robot to image-derived locations.

Motion diversity is essential. Motions should include rotations about more than one axis and translations of useful magnitude. Nearly parallel rotations or very small motions make the problem poorly conditioned. The calibration target must remain stationary, and timestamps must correspond to the same physical state.

8. Worked Integration Example

Consider an eye-in-hand camera observing a target fixed in the robot base. A target feature is detected in the image, and PnP estimates its pose in the camera frame. The hand–eye calibration provides [[M:T_{HC}]], while the robot controller reports [[M:T_{BH}]]. The feature pose in the base frame is obtained by composition:

where [[M:T_{CF}]] is the feature pose relative to the camera. The desired gripper pose is then generated by applying a known tool offset [[M:T_{HG}]] and an approach orientation. The operation must account for the fact that a feature pose and a gripper pose are not interchangeable: the tool centre point, grasp offset, collision clearance, and approach direction must be specified.

A validation move should first be performed at low speed with a safe clearance. The robot should approach a known fiducial or calibration point, and the residual position and orientation errors should be measured. If errors vary systematically with robot pose, likely causes include hand–eye error, robot kinematic calibration, camera flexure, or incorrect frame composition.

9. Error Sources, Limitations, and Best Practice

Optical and Image Errors

Blur, defocus, sensor noise, saturation, rolling shutter, chromatic aberration, and reflections affect feature localization. Use appropriate lighting, short exposure when motion is present, rigid mounting, and optics suited to the working distance. Avoid relying on image edges whose appearance changes with illumination.

Geometric Errors

A planar model cannot represent a non-planar object. A single camera cannot generally determine absolute depth from one image without known geometry, multiple views, or additional sensors. Calibration outside the actual working volume may extrapolate poorly. Lens focus or zoom changes invalidate intrinsic calibration.

Numerical and Algorithmic Errors

Least-squares methods can be sensitive to outliers. RANSAC is useful for rejecting mismatched correspondences, but its result depends on thresholds and sample quality. Normalize coordinates when estimating projective transformations, and use double precision where available. Always inspect residual vectors spatially; a low mean error can conceal localized distortion.

Validation Checklist

• Confirm frame names, directions, units, and matrix convention.
• Use independent validation images rather than only calibration images.
• Report mean, RMS, maximum, and spatially distributed errors.
• Test accuracy at the actual working distance and temperature.
• Verify repeatability separately from absolute accuracy.
• Include a safe robot validation routine and a recovery procedure.
• Recalibrate after changing focus, lens, camera mount, target, or robot tooling.

10. Applied Case: Vision-Guided Placement of a Connector

This is an illustrative engineering case. A robot must place electrical connectors into a fixture. A fixed overhead camera observes a planar tray containing fiducial marks and connector pockets.

Background. The tray is manufactured with known fiducial locations. Connectors vary slightly in position because of upstream handling. Placement accuracy must be better than 1.0 mm.

Problem. Direct pixel coordinates cannot be sent to the robot because the camera is oblique and the robot base is not aligned with the image axes. Lens distortion is also visible near the image boundary.

Analysis. The team first calibrates the camera with a checkerboard covering the complete field of view. Reprojection residuals are checked across the image. The tray plane is then characterized using four or more fiducials. Because all relevant connector pockets lie on the same plane, a homography is appropriate for tray coordinates. A separate robot-to-camera calibration establishes the camera-to-base relationship.

Approach. Images are undistorted before fiducial detection. A robust homography maps image points to tray coordinates. The detected connector centre is transformed to the tray frame, and the calibrated tray-to-base transformation produces a robot target. The robot approaches vertically, using a tool offset measured during commissioning. A clearance check is performed before insertion.

Outcome. Validation is conducted on positions not used to calculate the mapping. The team reports mean and maximum placement errors, not only successful insertions. When errors increase at one side of the tray, the cause is traced to a slightly flexible camera bracket rather than to the homography algorithm. The bracket is stiffened and the calibration repeated.

Learning points. A mathematically correct mapping cannot compensate for mechanical flexure. Calibration, validation, coordinate conventions, and mechanical design must be treated as one integrated system. The planar assumption is powerful but must be explicitly justified.

11. Contemporary Developments and Responsible Use

Modern libraries provide automatic calibration, differentiable projection layers, fiducial detection, and neural networks for keypoint or pose prediction. Learning-based methods can be valuable when targets are textured, partially occluded, or difficult to model analytically. Nevertheless, neural predictions still require geometric validation when used for safety-critical motion or measurement. A network may detect an object reliably while producing biased coordinates under changes in lighting, camera viewpoint, or domain.

Uncertainty-aware vision is increasingly important. Systems should communicate confidence, reject geometrically inconsistent results, and enter a safe state when calibration validity is uncertain. In industrial settings, camera data may also identify workers or production activity, creating privacy and cybersecurity responsibilities. Access control, secure communication, data minimization, and documented change management are appropriate governance practices.

Chapter Summary

Camera geometry converts three-dimensional structure into image measurements. The pinhole model uses intrinsic parameters to map camera coordinates to pixels, while extrinsic parameters locate the camera relative to a world or robot frame. Lens distortion must be modelled or corrected when accurate measurement is required.

Calibration estimates these parameters from known targets by minimizing reprojection error. Homography provides a projective mapping for planar scenes and supports perspective correction, but it is not a general three-dimensional transformation. PnP estimates object pose from known three-dimensional points and image correspondences. Robot integration requires explicit composition of rigid transformations and careful hand–eye calibration. Accuracy depends not only on algorithms but also on target quality, optical conditions, mechanical rigidity, synchronization, frame conventions, and independent validation.

Key Terms

Intrinsic parameters: Camera parameters such as focal lengths, principal point, and skew.

Extrinsic parameters: Camera rotation and translation relative to a reference frame.

Pinhole model: Ideal perspective projection through a single optical centre.

Homogeneous coordinates: A projective representation in which points differing by scale are equivalent.

Principal point: Pixel location where the optical axis intersects the image plane.

Radial distortion: Lens distortion that varies with distance from the principal point.

Tangential distortion: Distortion caused primarily by lens–sensor misalignment.

Reprojection error: Difference between measured and model-predicted image points.

Rotation matrix: A 3 × 3 orthonormal matrix representing orientation.

Rigid transformation: Rotation and translation that preserve distances and angles.

Homography: A projective mapping between planes.

Rectification: Transformation that removes perspective effects for a selected geometry.

PnP: Perspective-n-point pose estimation from known 3D–2D correspondences.

Pose: Position and orientation of an object or coordinate frame.

Eye-in-hand: Configuration in which the camera is mounted on the robot tool.

Eye-to-hand: Configuration in which the camera is fixed externally to the robot.

Hand–eye calibration: Estimation of the fixed camera-to-hand relationship.

RANSAC: Robust estimation method that rejects outlier correspondences.

Reprojection residual: A signed or vector difference between observed and predicted image locations.

Observability: Degree to which measurements and motions constrain unknown parameters.

Review Questions

Short Answer Questions

1. What is the difference between intrinsic and extrinsic camera parameters?
2. Why is division by depth central to the pinhole camera model?
3. State two common types of lens distortion.
4. Under what geometric condition is a homography appropriate?
5. Why must robot and camera coordinate-frame conventions be documented?

Long Answer Questions

1. Explain the complete workflow for calibrating a camera with a planar checkerboard, including data collection and validation.
2. Derive the planar homography relationship from the perspective projection equation when [[M:Z_W=0]].
3. Compare Euler angles, rotation matrices, and quaternions for representing orientation.
4. Explain the eye-in-hand hand–eye calibration problem and the meaning of [[M:AX=XB]].
5. Discuss how optical, mechanical, numerical, and synchronization errors affect a vision-guided robot.

Analytical / Application Questions

1. A planar part is viewed obliquely, but its four corner coordinates are known in the part plane. Design a procedure to rectify the image and measure a hole diameter, stating assumptions and validation tests.
2. A robot places parts accurately near the centre of an image but with increasing error toward the edges. Analyse at least three possible causes and propose experiments that distinguish them.
3. Design an eye-in-hand calibration experiment. Specify the required measurements, motion diversity, frame transformations, safety checks, and independent performance tests.

Multiple Choice Questions with Answers

Question 1

Which matrix primarily contains focal lengths and the principal point?

A. The intrinsic matrix K

B. The world-to-camera translation vector

C. The hand–eye matrix only

D. The robot Jacobian

Correct Answer: A. The intrinsic matrix K

Question 2

A homography is most directly applicable when image points correspond to what geometry?

A. Points at arbitrary unrelated depths

B. Points on a single plane

C. Only points on a spherical surface

D. Only points generated by a moving lens

Correct Answer: B. Points on a single plane

Question 3

What is the unit of RMS reprojection error when image coordinates are measured in pixels?

A. Metres

B. Radians

C. Pixels

D. Newtons

Correct Answer: C. Pixels

Question 4

Which condition must a valid rotation matrix satisfy?

A. RᵀR = 0

B. RᵀR = I and det(R) = +1

C. All entries must be positive

D. Its determinant must be zero

Correct Answer: B. RᵀR = I and det(R) = +1

Question 5

Why is lens distortion commonly corrected before precise geometric measurement?

A. It increases sensor bit depth

B. It removes systematic departures from the ideal projection model

C. It changes a monocular camera into a stereo camera

D. It eliminates the need for calibration

Correct Answer: B. It removes systematic departures from the ideal projection model

Question 6

At least how many non-collinear point correspondences are theoretically required to estimate a general homography?

A. Two

B. Three

C. Four

D. Eight hundred

Correct Answer: C. Four

Question 7

In the equation AX = XB for hand–eye calibration, X represents what?

A. A fixed hand–eye transformation

B. Image brightness

C. Lens focal length only

D. A random outlier

Correct Answer: A. A fixed hand–eye transformation

Question 8

A feature separation is 37 pixels horizontally and 24 pixels vertically in an image with scale 0.20 mm/pixel. What is the approximate physical separation?

A. 4.88 mm

B. 7.40 mm

C. 8.83 mm

D. 12.20 mm

Correct Answer: C. 8.83 mm

Question 9

Which practice best improves calibration observability?

A. Using identical camera poses for every image

B. Using varied orientations, distances, and image locations

C. Using only one corner of the target

D. Changing focus between every exposure

Correct Answer: B. Using varied orientations, distances, and image locations

Question 10

Why is a single planar homography generally unsuitable for points at different depths?

A. A homography models one plane, not arbitrary three-dimensional depth variation

B. Homographies cannot map pixels

C. Homographies require a camera with no lens

D. Depth has no effect on perspective projection

Correct Answer: A. A homography models one plane, not arbitrary three-dimensional depth variation