5. Camera Geometry and Computational Imaging

Introduction and Context

A camera does not record the three-dimensional world directly. It produces a two-dimensional measurement whose appearance depends on viewpoint, optics, sensor geometry, illumination, and image processing. Computer vision therefore requires a mathematical account of how points, lines, surfaces, and motion in the world are transformed into image measurements. This account is known as camera geometry.

Camera geometry provides the foundation for calibration, object measurement, visual localization, three-dimensional reconstruction, augmented reality, robotics, autonomous navigation, and computational photography. The central problem is deceptively simple: given a scene point and a camera, where will that point appear in the image? The inverse problem is more difficult: given one or more image measurements, what can be inferred about the three-dimensional scene?

This chapter develops the principal models used to answer these questions. It begins with the pinhole and lens models, then introduces homogeneous coordinates and projective transformations. It proceeds to calibration, homographies, epipolar geometry, stereo correspondence, depth estimation, and image rectification. The discussion emphasizes both mathematical structure and engineering judgement: a model is useful only when its assumptions are understood and its errors are controlled.

Learning Objectives

After studying this chapter, a student should be able to:

1. Explain the pinhole and lens-based models of image formation.

2. Use homogeneous coordinates and the camera matrix to represent projection.

3. Distinguish intrinsic, extrinsic, and distortion parameters.

4. Describe calibration procedures and interpret calibration errors.

5. Derive and apply homographies in planar geometry and image warping.

6. Explain epipolar geometry and the essential and fundamental matrices.

7. Formulate stereo correspondence and triangulation for depth estimation.

8. Explain why rectification simplifies stereo matching.

9. Evaluate practical sources of uncertainty, failure, and bias in geometric vision systems.

10. Relate classical camera geometry to modern computational imaging and AI systems.

From a Three-Dimensional Point to an Image Measurement

The ideal pinhole camera

Consider a camera coordinate system with its origin at the optical centre and the optical axis aligned with the positive Z-axis. Under the pinhole model, a three-dimensional point P = (X, Y, Z) is projected onto an image plane located at focal length f. Similar triangles give

x = fX/Z, y = fY/Z.

The division by Z is the source of perspective. Objects farther from the camera appear smaller, and parallel lines in the world may converge in the image. The model assumes that all light passes through a single point and that the image plane is perfectly flat. Although no physical camera satisfies these assumptions exactly, the model is an excellent approximation for many systems and is the basis of most multi-view geometry.

The same geometry may be expressed using normalized image coordinates, where the focal length is temporarily taken as one. If a point is represented by a direction vector (X/Z, Y/Z, 1), its image position is obtained by perspective division. This distinction between a direction in space and a measured pixel location is important in calibration and reconstruction.

Intrinsic camera parameters

Pixel coordinates are not generally measured in the same units as the camera's physical focal length. The intrinsic matrix maps normalized coordinates to pixels:

K = [[fx, s, cx], [0, fy, cy], [0, 0, 1]].

Here fx and fy are focal lengths expressed in pixel units, cx and cy locate the principal point, and s is the skew parameter. In modern digital cameras, skew is usually close to zero, but it belongs to the general model. The camera matrix K describes internal imaging geometry and does not depend on how the camera is positioned in the scene.

Extrinsic parameters

The camera's position and orientation relative to a world coordinate system are represented by a rotation matrix R and translation vector t. A world point is transformed into camera coordinates according to

P_c = RP_w + t.

The complete perspective projection is therefore

p ~ K[R | t]P_w,

where p is a homogeneous image point and ~ denotes equality up to a non-zero scale. The 3 × 4 matrix P = K[R | t] is called the projective camera matrix. It maps a world point to an image point, but the mapping loses depth because homogeneous image coordinates are defined only up to scale.

Homogeneous Coordinates and Projective Thinking

Ordinary Euclidean coordinates are inconvenient when representing points at infinity, perspective transformations, and intersections. Homogeneous coordinates append an additional coordinate: a two-dimensional point (x, y) is represented as (x, y, 1), while any non-zero scalar multiple represents the same point. Thus (x, y, 1) and (λx, λy, λ) identify the same image location.

A projective transformation, or homography, is represented by a 3 × 3 nonsingular matrix H:

p' ~ Hp.

After multiplication, the result must be divided by its third coordinate to recover ordinary pixel coordinates. Lines can also be represented homogeneously. If l = (a, b, c), the line equation is ax + by + c = 0. The line joining two points is their cross product, and the intersection of two lines is also a cross product. These simple algebraic operations are among the main reasons projective geometry is central to computer vision.

Projective representations also clarify what cannot be recovered from an image. A single image generally cannot determine the absolute scale of a scene. A reconstruction twice as large and twice as far from the camera may produce the same image. This scale ambiguity is not a numerical defect; it is an inherent property of perspective projection.

Optical Distortion and the Real Camera

The pinhole model describes the ideal projection, whereas lenses introduce distortion. Radial distortion causes image points to move along rays from the principal point. A common model uses normalized coordinates (x, y), radius r² = x² + y², and coefficients k1, k2, and sometimes k3:

x_distorted = x(1 + k1r² + k2r⁴ + k3r⁶),

y_distorted = y(1 + k1r² + k2r⁴ + k3r⁶).

Tangential distortion arises when the lens and sensor are not perfectly aligned. It is commonly represented using coefficients p1 and p2. Distortion is often small near the image centre but substantial near the boundaries, especially for wide-angle lenses. Ignoring it can produce systematic errors in calibration, pose estimation, line detection, and stereo depth.

A practical camera model must also account for finite sensor resolution, rolling shutter, exposure, and potentially focus-dependent changes in intrinsics. For high-precision applications, calibration is not a one-time ritual; it is a measurement process whose validity depends on the operating conditions.

Calibration as an Estimation Problem

Camera calibration estimates intrinsic parameters, distortion coefficients, and, for each calibration view, the camera pose relative to a known target. A planar checkerboard is frequently used because its corner locations can be detected accurately and its geometry is known. Multiple views are needed: one view cannot adequately separate focal length, pose, and distortion.

The standard workflow is as follows:

• acquire images of a calibration target at varied orientations and positions;
• detect target features and associate them with known world coordinates;
• estimate an initial camera model from projective constraints;
• refine parameters by minimizing reprojection error;
• inspect residuals and validate the model on images not used for fitting.

For an observed image point p_i and predicted point p_i_hat, a common objective is

E = Σ ||p_i - p_i_hat||².

Nonlinear optimization, often with robust loss functions, refines the parameters. A low average error is not sufficient by itself. Residuals should be examined spatially: a curved or edge-concentrated residual pattern may indicate an inadequate distortion model, while errors that vary with time may indicate focus movement or mechanical instability.

Calibration has organizational as well as technical consequences. A production system should record the camera, lens, resolution, focus setting, calibration date, target specification, and validation results. Reusing parameters after changing resolution or lens configuration is a common and avoidable mistake.

Homographies: The Geometry of Planar Scenes

A homography relates corresponding points in two images when the points lie on a common plane, or when the camera undergoes pure rotation. If a plane is described by n^T X + d = 0, the mapping between two views has the form

H = K_2(R - tn^T/d)K_1^{-1}.

In practical terms, a homography can align a document photographed at an angle, transfer a texture onto a planar surface, or register satellite images of approximately planar terrain. At least four point correspondences are required to estimate a general homography, although more points are normally used with a robust estimator such as RANSAC.

RANSAC repeatedly selects minimal samples, estimates a model, and counts data points whose transfer error is below a threshold. It is valuable when feature matches include outliers. However, its result depends on the threshold, noise scale, sample quality, and scene configuration. A homography is not a general substitute for three-dimensional motion. Applying it to a scene with substantial depth variation can align one region while producing incorrect alignment elsewhere.

Epipolar Geometry and Two-View Reconstruction

When two cameras observe the same scene, a point in the first image does not correspond to an arbitrary location in the second image. Its possible match lies on an epipolar line. The line is the image of the plane formed by the two camera centres and the observed three-dimensional point.

The essential matrix E describes this relationship in normalized camera coordinates:

x_2^T E x_1 = 0.

If the relative rotation and translation are R and t, then

E = [t]_x R,

where [t]_x is the skew-symmetric matrix representing the cross product with t. In pixel coordinates, the fundamental matrix F incorporates both cameras' intrinsic matrices:

F = K_2^{-T} E K_1^{-1},

and corresponding points satisfy x_2^T F x_1 = 0.

The essential matrix is appropriate when calibrated camera geometry is known; the fundamental matrix can be estimated directly from image correspondences without prior calibration. Both matrices are rank-two and encode the epipolar pencils of lines. Their practical estimation is sensitive to outliers and degenerate configurations, such as points concentrated on a single plane or nearly pure forward motion.

Epipolar geometry reduces correspondence from a two-dimensional search to a one-dimensional search along a line. This reduction is central to stereo matching, visual odometry, and structure-from-motion.

Stereo Correspondence and Depth

A stereo rig contains two cameras with a known baseline B. After rectification, corresponding points share the same image row. If their horizontal positions are u_L and u_R, the disparity is

d = u_L - u_R.

For a rectified pair with focal length f in pixels, depth is

Z = fB/d.

This equation expresses an important engineering trade-off. Increasing the baseline improves depth sensitivity, while increasing focal length also increases disparity for a given depth. Conversely, distant objects generate small disparities and therefore large relative depth errors. If disparity uncertainty is σ_d, approximate error propagation gives

σ_Z ≈ (fB/d²)σ_d.

Thus depth uncertainty grows rapidly with distance.

Stereo correspondence is difficult because image patches may be textureless, repetitive, reflective, occluded, or differently illuminated. Local methods compare windows using measures such as absolute intensity difference, squared difference, or normalized correlation. Global and semi-global methods impose smoothness or path-based penalties while preserving likely discontinuities. Modern neural stereo systems learn matching costs and regularization, but they still depend on geometric calibration and may fail under domain shift.

A reliable stereo pipeline usually includes confidence estimation, left-right consistency checks, uniqueness tests, and treatment of occlusions. A visually dense depth map is not necessarily a trustworthy depth map.

Rectification and Triangulation

Image rectification applies homographies to two images so that corresponding epipolar lines become horizontal and aligned. The result permits efficient row-wise matching and simplifies hardware implementation. Rectification does not create information; it reorganizes existing measurements according to the estimated camera geometry.

Once correspondences are known, triangulation estimates the three-dimensional point whose projections best agree with the observations. In a linear formulation, each image measurement contributes equations of the form

x × (PX) = 0.

The resulting homogeneous linear system is solved using singular value decomposition, followed by conversion to Euclidean coordinates. Because image measurements are noisy, optimal triangulation minimizes geometric or reprojection error rather than relying solely on an algebraic residual. Points should also be checked for positive depth in both cameras, a condition known as cheirality.

Triangulation quality depends strongly on the angle between viewing rays. A narrow triangulation angle produces unstable depth, whereas an excessively wide angle may increase matching difficulty and occlusion. System design therefore balances baseline, field of view, expected range, and correspondence reliability.

Computational Imaging and Modern Vision Systems

Computational imaging treats image formation and inference as a coupled system. Calibration parameters, exposure control, coded apertures, multi-camera arrangements, event sensors, and learned reconstruction algorithms can be designed together. For example, a camera may be configured to maximize depth precision in a known working volume rather than simply to produce visually pleasing photographs.

In robotics, camera geometry supports localization and mapping. In augmented reality, it allows virtual objects to remain registered with physical surfaces. In industrial inspection, calibrated cameras convert pixel measurements into dimensional tolerances. In medical and scientific imaging, geometric models support registration between sensors and coordinate systems. In autonomous vehicles, multiple cameras and depth sensors are fused to estimate free space and object position.

AI does not eliminate geometric reasoning. Learned feature extractors improve correspondence, and neural networks can estimate depth from one or more images, but their predictions remain affected by calibration, viewpoint, scale, and data distribution. Hybrid systems combine geometric constraints with learned representations, often improving interpretability and robustness.

Illustrative Case Study: Stereo Inspection of a Conveyor Assembly

Background

A manufacturing company wishes to inspect whether small connectors on an electronic assembly are seated at the correct height. A single overhead camera cannot reliably distinguish height differences because several connector surfaces have similar appearance.

Situation and problem

Two cameras are mounted above the conveyor. The environment contains repeated patterns, specular plastic surfaces, and moderate vibration. The system must estimate connector height and reject assemblies outside tolerance without slowing production.

Analysis

The engineers first identify that raw disparity will be meaningful only after accurate calibration and synchronization. They select a baseline that produces measurable disparity across the working range, while ensuring both cameras can see the relevant surfaces. A planar calibration target is imaged at multiple poses. Reprojection residuals reveal radial distortion near the image edges, so distortion correction is included.

The cameras are rectified using their estimated relative pose. A learned feature matcher proposes correspondences, but matches are filtered with left-right consistency and an epipolar constraint. Triangulated points are transformed into the conveyor coordinate system. A reference plane is fitted to known assembly supports, and connector height is measured relative to that plane.

Approach and outcome

The system uses confidence-weighted depth estimates rather than accepting every pixel. Specular regions and occlusions are marked as uncertain and are handled through multiple observations or rule-based inspection. Periodic revalidation is scheduled because vibration can alter the relative pose. The approach produces a dimensional decision instead of merely a depth image: each assembly is classified as acceptable, too high, or too low, with an audit record containing images, calibration version, and confidence.

Learning points

The case illustrates that the decisive factors are not only the stereo algorithm. Baseline selection, calibration quality, coordinate-system management, uncertainty handling, and operational maintenance determine whether a geometric system is suitable for production. A sophisticated matcher cannot compensate for an unstable camera mount or an invalid calibration model.

Sources of Error, Limitations, and Professional Responsibilities

Common errors include confusing world-to-camera and camera-to-world transformations, using degrees where radians are expected, applying a calibration at a different image resolution, neglecting lens distortion, and interpreting disparity with the wrong sign. Poorly conditioned camera arrangements, insufficient texture, moving objects, rolling shutter, and synchronization errors also degrade reconstruction.

Geometric models are approximations. A rigid stereo model assumes fixed cameras; a pinhole model ignores lens and sensor effects; a static-scene assumption fails when objects move. The correct response is not to conceal these assumptions but to quantify their consequences through validation data, uncertainty estimates, and failure detection.

Privacy and governance are relevant when calibrated cameras operate in workplaces or public environments. Images and derived three-dimensional data may identify people, infer behaviour, or reveal sensitive industrial processes. Systems should apply data minimization, access control, retention limits, and documented legitimate purposes. In safety-critical applications, performance must be evaluated across lighting, surfaces, temperatures, and operating conditions rather than only on curated test scenes.

Current Developments and Best Practices

Recent systems increasingly combine classical geometry with deep networks for feature matching, depth completion, pose estimation, and uncertainty prediction. Neural radiance fields and related view-synthesis methods model appearance across viewpoints, while event cameras provide high temporal resolution for rapid motion. Multi-modal systems combine RGB, stereo, time-of-flight, lidar, or inertial measurements.

Nevertheless, several practices remain fundamental:

• define coordinate frames explicitly and document every transformation;
• calibrate under the actual resolution, focus, and mounting conditions;
• use diverse calibration views and inspect spatial residuals;
• estimate models robustly and reject outliers;
• validate geometry on independent scenes and operating conditions;
• report uncertainty and confidence, not only point estimates;
• monitor calibration drift and maintain versioned parameters;
• test degenerate cases, occlusions, motion, reflections, and low texture.

Chapter Summary

Camera geometry explains how three-dimensional structure becomes a two-dimensional image and how multiple images can be used to infer structure. The pinhole model introduces perspective projection, while the intrinsic matrix represents focal length, principal point, and skew. Extrinsic parameters describe camera pose, and the complete projection is represented by a camera matrix. Homogeneous coordinates make projective transformations and incidence relationships algebraically convenient.

Real lenses introduce radial and tangential distortion, making calibration essential for accurate measurement. Homographies describe planar mappings, whereas epipolar geometry constrains correspondences between general views through the essential and fundamental matrices. In a calibrated and rectified stereo system, disparity provides depth according to Z = fB/d, although uncertainty increases rapidly for small disparities. Triangulation, rectification, robust estimation, and confidence checking form the practical core of multi-view reconstruction. Modern AI enhances matching and inference, but dependable systems still require sound geometry, calibration governance, and explicit treatment of uncertainty.

Key Terms

Pinhole camera: An ideal camera model in which all rays pass through one optical centre.

Perspective projection: Mapping that divides image coordinates by depth, causing scale to vary with distance.

Intrinsic parameters: Internal camera properties such as focal lengths, principal point, and skew.

Extrinsic parameters: Camera rotation and translation relative to a world coordinate system.

Camera matrix: The projective matrix K[R | t] mapping world points to image points.

Homogeneous coordinates: Coordinates defined up to a non-zero scale factor.

Projective transformation: A transformation preserving incidence relationships between points and lines.

Homography: A nonsingular 3 × 3 projective mapping between image planes.

Radial distortion: Lens distortion whose displacement depends primarily on distance from the principal point.

Tangential distortion: Distortion caused by misalignment between lens and image sensor.

Calibration: Estimation of camera parameters from known geometric observations.

Reprojection error: Difference between an observed image point and its model-predicted projection.

Epipolar line: The locus of possible matches in one image for a point in another image.

Essential matrix: Matrix encoding epipolar geometry in normalized camera coordinates.

Fundamental matrix: Matrix encoding epipolar geometry in pixel coordinates.

Disparity: Difference in corresponding image positions, usually between rectified stereo images.

Rectification: Warping images so corresponding epipolar lines become aligned, usually horizontally.

Triangulation: Estimation of a three-dimensional point from multiple image projections.

Cheirality: The condition that a reconstructed point lies in front of the observing cameras.

RANSAC: A robust estimation method that fits models despite outlying observations.

Review Questions

Short Answer Questions

1. Why does perspective projection divide image coordinates by depth?

2. What roles do intrinsic and extrinsic camera parameters play?

3. Why are homogeneous coordinates useful in projective geometry?

4. Distinguish the essential matrix from the fundamental matrix.

5. What is image rectification, and why is it useful for stereo matching?

Long Answer Questions

1. Derive the pinhole projection equations and explain the assumptions behind them.

2. Describe a complete camera calibration workflow, including parameter estimation and validation.

3. Explain how a homography is used to register a planar object photographed from different viewpoints.

4. Discuss epipolar geometry, correspondence constraints, and the roles of E and F.

5. Explain stereo depth estimation, including disparity, baseline, uncertainty, and triangulation.

Analytical / Application Questions

1. A rectified stereo system has f = 800 pixels, baseline B = 0.12 m, and measured disparity d = 48 pixels. Estimate the depth and explain how a disparity error of 2 pixels would affect the estimate.

2. A calibration produces low average reprojection error but large residuals near all four image corners. What model or data problems might this indicate, and how would you investigate them?

3. A planar homography aligns the centre of an image but produces increasing misalignment near objects at different depths. Explain why this occurs and propose a more appropriate geometric strategy.

22. Multiple Choice Questions with Answers

Question 1

Which expression represents perspective projection in the ideal pinhole camera model?

A. x = fXZ and y = fYZ

B. x = fX/Z and y = fY/Z

C. x = X + fZ and y = Y + fZ

D. x = Z/fX and y = Z/fY

Correct Answer: B. x = fX/Z and y = fY/Z

Question 2

Which parameter is part of the intrinsic camera matrix?

A. World translation

B. Camera rotation

C. Principal point

D. Scene scale

Correct Answer: C. Principal point

Question 3

A homography is most directly appropriate for relating images of:

A. A general volume with large depth variation

B. A common plane

C. Independent moving objects

D. Unsynchronized cameras only

Correct Answer: B. A common plane

Question 4

What constraint do corresponding points satisfy in epipolar geometry?

A. x2^T F x1 = 0

B. x2^T K x1 = 1

C. x2 = F + x1

D. x2^T R x1 = Z

Correct Answer: A. x2^T F x1 = 0

Question 5

For a rectified stereo pair, depth is given by Z = fB/d. If disparity decreases while f and B remain constant, depth:

A. Decreases

B. Becomes zero

C. Increases

D. Remains unchanged

Correct Answer: C. Increases

Question 6

Why is rectification useful in stereo vision?

A. It removes all lens distortion permanently

B. It makes corresponding points lie on the same image rows

C. It eliminates the need for calibration

D. It recovers absolute scene scale from one image

Correct Answer: B. It makes corresponding points lie on the same image rows

Question 7

Which situation is especially likely to make stereo depth unstable?

A. A large, accurately known disparity

B. A narrow image field with high texture

C. A very small disparity for a distant object

D. A point lying in front of both cameras

Correct Answer: C. A very small disparity for a distant object

Question 8

What is the main purpose of RANSAC in homography or fundamental-matrix estimation?

A. Increasing image resolution

B. Rejecting outlier correspondences

C. Measuring lens focal length directly

D. Converting pixels into colour values

Correct Answer: B. Rejecting outlier correspondences

Question 9

A reconstructed point violates cheirality if it:

A. Has homogeneous coordinates

B. Lies behind one of the observing cameras

C. Has nonzero disparity

D. Is located on a calibrated plane

Correct Answer: B. Lies behind one of the observing cameras

Question 10

Which statement best describes calibration validation?

A. Only the average reprojection error matters

B. Calibration need not be repeated after changing resolution

C. Residual patterns and independent operating scenes should be examined

D. A calibration target is unnecessary when using stereo

Correct Answer: C. Residual patterns and independent operating scenes should be examined