Stereo, Depth and 3D Vision

Introduction and Context

Computer vision systems often need more than the appearance of objects in a two-dimensional image. They must estimate how far objects are from a camera, recover the three-dimensional structure of a scene, determine camera motion, and construct a representation that supports navigation, measurement, interaction, or simulation. These tasks constitute 3D vision.

Depth can be measured directly with sensors such as LiDAR, time-of-flight cameras, and structured-light systems, or inferred from images. This chapter focuses on image-based inference, especially stereo vision, monocular depth estimation, point clouds, structure from motion (SfM), visual odometry, simultaneous localization and mapping (SLAM), 3D reconstruction, and neural scene representations.

The central difficulty is that image formation discards information. A three-dimensional point is projected onto a two-dimensional image, so multiple 3D scenes may explain the same image. Depth estimation therefore requires additional constraints, such as observations from multiple viewpoints, assumptions about scene geometry, learned priors, or active sensing.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Explain camera models, stereo geometry, disparity, and triangulation.

2. Calibrate a stereo camera and distinguish intrinsic from extrinsic parameters.

3. Analyze correspondence, occlusion, ambiguity, and error in stereo depth.

4. Compare stereo, monocular, active, and learned depth estimation.

5. Represent 3D data using point clouds, meshes, voxels, and implicit functions.

6. Explain the main stages of structure from motion, visual odometry, and SLAM.

7. Describe the principles of multi-view reconstruction and neural scene representations.

8. Evaluate practical, safety, ethical, and engineering constraints in 3D vision systems.

1. Foundations of Image-Based 3D Vision

1.1 Coordinate systems and the pinhole camera

A camera coordinate system commonly uses X and Y for horizontal and vertical position and Z for distance along the optical axis. Under the ideal pinhole model, a 3D point (X, Y, Z) projects to image coordinates (u, v) according to

u = f_x X/Z + c_x,

v = f_y Y/Z + c_y.

Here, f_x and f_y are focal lengths measured in pixels, and (c_x, c_y) is the principal point. The equations assume a perspective camera and a point in front of the camera (Z > 0). They do not account for lens distortion.

In homogeneous coordinates, projection is often written as

s p = K [R | t] P,

where P is a 3D point in world coordinates, p is its image point, s is a nonzero scale factor, K contains intrinsic parameters, and R and t describe camera orientation and translation relative to the world. The matrix K is typically

K = [[f_x, 0, c_x], [0, f_y, c_y], [0, 0, 1]].

Radial and tangential lens distortion must generally be calibrated, particularly for wide-angle lenses. If distortion is not corrected, corresponding points do not satisfy the ideal epipolar geometry accurately.

1.2 Why depth is difficult

The projection equations contain the ratio X/Z and Y/Z rather than X, Y, and Z independently. A single image therefore cannot, in general, determine absolute depth without additional information. Depth is also affected by:

• textureless regions, where correspondence is uncertain;
• repeated patterns, which create multiple plausible matches;
• occlusion, where a point appears in one view but not another;
• specular or transparent surfaces, which violate brightness assumptions;
• motion, which can be confused with camera movement;
• scale ambiguity in monocular reconstruction;
• calibration and synchronization errors.

These limitations explain why 3D vision is an inference problem rather than a simple geometric measurement.

2. Stereo Vision

2.1 Stereo geometry and disparity

A stereo rig contains two cameras separated by a baseline B. After calibration and rectification, corresponding points lie on the same image row. Let x_L and x_R be the horizontal coordinates of a corresponding point in the left and right images. Disparity is

d = x_L - x_R.

For a standard parallel stereo configuration, depth is

Z = fB/d,

where f is the focal length in pixels, B is the baseline in the same physical length unit desired for Z, and d is disparity in pixels. The formula assumes rectified cameras, a common optical orientation, and positive disparity for points in front of the cameras.

The relationship is inverse: nearby objects have large disparity, whereas distant objects have small disparity. If d is zero, the point is effectively at infinity under this model. Depth cannot be estimated reliably when disparity is too small relative to image noise.

Once Z is known, the 3D coordinates relative to the left camera can be recovered as

X = (u - c_x)Z/f_x,

Y = (v - c_y)Z/f_y.

In a practical calibrated stereo system, the exact formulas are derived from the full camera matrices, but the simplified equations remain useful for understanding design trade-offs.

2.2 Worked example: depth and uncertainty

Suppose a rectified stereo camera has focal length f = 800 pixels and baseline B = 0.12 m. A feature has disparity d = 48 pixels. Its estimated depth is

Z = (800 × 0.12)/48 = 96/48 = 2.0 m.

Assume disparity uncertainty is approximately ±1 pixel. Differentiating Z = fB/d gives the approximate magnitude of depth sensitivity,

|ΔZ| ≈ fB|Δd|/d².

Thus,

|ΔZ| ≈ (800 × 0.12 × 1)/(48²) = 96/2304 ≈ 0.0417 m.

The estimated depth is therefore approximately 2.00 m with an uncertainty of 4.2 cm under the stated noise assumption. This illustrates why stereo becomes less precise at long range: the same disparity error causes a larger depth error as d becomes small.

2.3 Epipolar geometry

Before rectification, a point in one image does not correspond to an arbitrary point in the other. Its match must lie on an epipolar line. The fundamental matrix F relates corresponding homogeneous image points through

p_R^T F p_L = 0.

If camera calibration is known, the essential matrix E represents the relative rotation and translation in normalized coordinates. Epipolar constraints reduce correspondence search from a two-dimensional region to a one-dimensional line.

Stereo rectification applies image transformations so that epipolar lines become horizontal. This improves computational efficiency and makes disparity maps easier to interpret, although rectification itself must be computed accurately.

2.4 Calibration

Stereo calibration estimates:

• intrinsic parameters: focal lengths, principal point, and distortion;
• extrinsic parameters: relative rotation and translation between cameras;
• baseline: the magnitude of the translation between camera centers.

A common procedure observes a known calibration target, such as a checkerboard, from multiple poses. Calibration should cover much of the image field and include varied orientations and distances. Reprojection error is a useful diagnostic, but a low average error does not guarantee accurate depth everywhere. Poor synchronization, flexible camera mounts, thermal changes, and imperfect target detection can produce systematic errors.

2.5 Correspondence and disparity estimation

A stereo matching algorithm seeks the disparity that best explains local or global image similarity. Major approaches include:

• block matching, which compares rectangular intensity or color windows;
• semi-global matching, which aggregates matching costs along multiple directions;
• graph-cut or dynamic-programming methods, which impose smoothness constraints;
• deep stereo networks, which learn features and matching costs from data.

A typical matching cost may compare absolute intensity differences, census transforms, or learned feature distances. Smoothness assumptions are useful within surfaces but should not blur across object boundaries. Modern systems often combine a learned cost volume with regularization and refinement.

The output is a disparity map, often accompanied by confidence estimates. Invalid pixels occur at occlusion boundaries, reflective surfaces, low-texture areas, and regions outside the overlapping camera fields of view. Left-right consistency checking is a common quality test: a disparity estimated from the left image should agree with the corresponding estimate computed from the right image.

3. Monocular and Active Depth

3.1 Monocular depth

A single image can provide relative depth cues such as perspective convergence, occlusion, texture gradients, shading, familiar object size, and atmospheric effects. However, absolute scale is generally unobservable from one unconstrained image. A learned monocular depth model can infer plausible depth by using statistical regularities in its training data, but its prediction may be metrically inaccurate or fail under domain shift.

Monocular depth estimation is valuable when a second camera is unavailable, but it should not be interpreted as direct measurement without validation. Scale can sometimes be recovered using known object dimensions, camera height, ground-plane assumptions, inertial measurements, or additional views.

3.2 Active depth sensors

Structured-light systems project a known pattern and infer depth from its deformation. Time-of-flight sensors estimate distance from the travel time or phase of emitted light. LiDAR measures the return time of laser pulses and can provide accurate, sparse or dense range data depending on the sensor.

Active sensors can outperform passive stereo in textureless scenes, but they have limitations. Structured light may fail outdoors in strong sunlight; time-of-flight sensors may suffer from multipath interference and reflective surfaces; LiDAR can be expensive and sparse. Sensor fusion combines complementary strengths, for example by using LiDAR for metric scale and cameras for texture and semantic interpretation.

4. 3D Representations

4.1 Point clouds

A point cloud is a set of 3D points, optionally carrying color, intensity, surface normals, confidence, or semantic labels. Stereo depth can be back-projected into a point cloud using the camera model. Point clouds preserve measured samples without imposing a surface topology, making them useful for mapping, inspection, and registration.

Their disadvantages include irregular sampling, noise, missing regions, and potentially high memory requirements. Downsampling, outlier removal, normal estimation, and coordinate-frame management are essential operations.

4.2 Meshes and volumetric representations

A polygon mesh represents surfaces using vertices, edges, and faces. Meshes are compact and suitable for rendering and simulation, but reconstructing reliable connectivity from noisy points is difficult, especially at boundaries and in incomplete scans.

Voxel grids divide space into regular cells. Occupancy grids record whether cells are occupied, while signed distance fields store the distance to the nearest surface, with sign indicating inside or outside under a chosen convention. Voxels simplify spatial reasoning but consume memory that grows rapidly with resolution.

4.3 Implicit and neural representations

An implicit surface represents geometry as a function whose level set defines a surface. Neural radiance fields (NeRFs) extend this idea by learning a function that maps a 3D position and viewing direction to density and emitted color. Rendering integrates contributions along camera rays. Training usually requires multiple posed images and optimizes the representation so rendered images agree with observations.

Neural representations can produce compelling novel views and encode complex appearance, but they may require substantial computation, accurate camera poses, and many images. They can also reproduce view-dependent artifacts rather than a physically correct surface. Recent systems use explicit primitives such as 3D Gaussian splats to obtain faster rendering while retaining a learned scene representation.

5. Structure from Motion and Visual Odometry

5.1 Structure from motion

Structure from motion estimates both camera poses and 3D scene structure from multiple images. A typical pipeline is:

1. Detect features or dense visual descriptors.

2. Match features across images.

3. Reject outliers using geometric verification, often with RANSAC.

4. Estimate relative camera motion from an essential or fundamental matrix.

5. Triangulate matched points.

6. Add views incrementally or solve a global bundle-adjustment problem.

7. Refine camera poses and 3D points jointly.

Bundle adjustment minimizes reprojection error. A simplified objective is

minimize Σ_i Σ_j ρ(||p_ij - π(T_i P_j)||²),

where P_j is a 3D point, T_i is camera pose, π is the projection function, p_ij is an observed image point, and ρ is optionally a robust loss. The sum is over observed point-camera pairs. Robust losses reduce the effect of incorrect matches.

Monocular SfM has gauge freedoms: the reconstruction is typically determined only up to a global similarity transformation, meaning rotation, translation, and scale may be arbitrary. Known camera motion, stereo, GPS, inertial data, or known object dimensions can provide metric scale.

5.2 Visual odometry

Visual odometry estimates the motion of a camera incrementally from consecutive frames. Feature-based methods track keypoints and solve geometric motion problems. Direct methods instead optimize photometric alignment over image intensities, often using many pixels rather than sparse features.

Visual odometry is subject to drift. Small errors in each motion estimate accumulate, causing the estimated trajectory to diverge from the true path. Motion blur, rapid rotation, low texture, illumination change, rolling shutter, and independently moving objects are important failure sources.

Stereo visual odometry can recover metric scale from the known baseline. Monocular visual odometry cannot do so without an additional scale source.

6. SLAM: Simultaneous Localization and Mapping

SLAM estimates a robot or camera trajectory while constructing a map. It differs from simple odometry by using revisited locations to reduce accumulated drift.

A SLAM system generally contains:

• front-end tracking: feature extraction, matching, or direct alignment;
• back-end estimation: optimization over poses and map elements;
• loop-closure detection: recognition of previously visited places;
• map management: keyframes, landmarks, marginalization, and culling;
• optional sensor fusion with inertial, wheel, GPS, or depth data.

The back end may use a pose graph, a landmark-based bundle-adjustment graph, or a filtering framework. A loop closure adds a constraint between two previously separate portions of the trajectory. If correctly detected, global optimization distributes the correction over the map. Incorrect loop closures are dangerous because they can distort the entire map; geometric verification and conservative acceptance criteria are therefore essential.

SLAM maps may be sparse, semi-dense, dense, semantic, or object-centric. A navigation system may require only a geometric occupancy map, whereas augmented reality may require a detailed, stable surface model.

7. 3D Reconstruction and Registration

Multi-view stereo reconstructs dense geometry after camera poses have been estimated. It computes correspondences across several images, triangulates points, filters inconsistent estimates, and fuses observations. More views can improve coverage and confidence, but increase computational cost and the potential for inconsistent appearance.

Point-cloud registration aligns two or more scans. Iterative closest point (ICP) alternates between establishing correspondences and estimating the rigid transformation that minimizes point-to-point or point-to-plane error. ICP works best when the initial alignment is reasonably close and the surfaces contain enough geometric structure. Feature-based global registration is useful for larger initial misalignments.

A rigid transformation has the form

p' = Rp + t,

where R is a 3 × 3 rotation matrix satisfying R^T R = I and det(R) = 1, and t is a translation vector. If scale is also unknown, a similarity transformation is required instead.

Quality assessment should include geometric accuracy, completeness, uncertainty, and downstream task performance. A visually attractive reconstruction may still have incorrect scale, warped surfaces, or missing safety-critical regions.

8. Detailed Applied Case: Illustrative Indoor Robot Mapping Scenario

Background

Consider an illustrative warehouse robot equipped with a stereo camera and an inertial measurement unit. The robot must create a map, localize itself, and avoid shelving, pallets, and people.

Problem

The warehouse contains long textureless walls, repetitive shelves, changing illumination, and moving workers. A purely monocular system would suffer from scale ambiguity, while a dense 3D model could exceed the robot's computational and memory budgets.

Analysis

The stereo baseline provides metric depth, but correspondence is unreliable on blank walls and reflective packaging. IMU measurements help during rapid motion and short periods of visual degradation. Repeated shelf patterns create false matches and potential loop-closure errors. Dynamic people should not be incorporated as permanent landmarks.

Approach and solution

The system first calibrates both cameras and synchronizes image capture. Images are rectified, and a stereo matcher produces disparity with confidence estimates. High-confidence points are converted into a local point cloud. A visual-inertial front end tracks features while rejecting points belonging to detected moving objects. A factor-graph back end jointly optimizes camera poses, inertial states, and selected landmarks.

Keyframes are inserted when viewpoint change or tracking quality requires them. Place-recognition candidates are verified using feature geometry before loop closure is accepted. The map stores a sparse landmark layer for localization and a voxel-based occupancy layer for collision avoidance. Periodic scans are registered to refine local surfaces, but uncertain points are not treated as obstacles without corroborating evidence.

Outcome

The design provides metric localization, limits computation, and reduces map corruption from dynamic objects. It is more robust than either stereo alone or monocular odometry alone, although it remains vulnerable to prolonged darkness, severe vibration, and scenes with insufficient stable texture.

Learning points

1. Sensor fusion is most useful when failure modes are complementary.

2. Confidence and uncertainty should influence mapping decisions.

3. A map should be designed for its task rather than maximum visual detail.

4. Dynamic-scene handling and loop-closure verification are safety-critical.

5. Calibration, timing, and coordinate-frame conventions are engineering requirements, not optional refinements.

9. Evaluation, Limitations, and Best Practices

Depth systems are commonly evaluated using absolute error, squared error, absolute relative error, threshold accuracy, completeness, and task-specific measures. The metric must match the application. For example, an autonomous braking system should emphasize worst-case errors near obstacles, not only average image-wide accuracy.

Best practices include:

• calibrate and validate cameras in the actual operating configuration;
• maintain precise timestamps and account for rolling shutter;
• report coordinate frames, units, and scale conventions explicitly;
• retain confidence, covariance, or validity masks with depth data;
• test on lighting, surfaces, distances, and motion conditions that reflect deployment;
• separate static and dynamic scene elements;
• use robust estimation and geometric consistency checks;
• monitor drift, loop-closure failures, and map quality;
• protect recorded imagery and 3D scans as potentially sensitive data;
• validate learned models against domain shift and rare but consequential failures.

10. Professional, Ethical, and Governance Considerations

Three-dimensional maps may reveal private homes, workplaces, personal routines, or biometric information. Organizations should minimize data collection, restrict access, encrypt stored maps, define retention periods, and obtain appropriate consent. A depth map is not automatically anonymous: geometry can identify rooms, bodies, or individuals.

Safety-critical systems require traceability, testing, fail-safe behavior, and human oversight. A system should distinguish measured geometry from learned prediction and should communicate uncertainty rather than presenting a false impression of precision. Bias can arise when training data underrepresent lighting conditions, skin tones, body shapes, environments, or camera types. Procurement and deployment decisions should therefore include subgroup and environment-specific validation.

11. Contemporary Developments

Current research is integrating foundation models, self-supervised learning, transformer-based matching, neural implicit scenes, Gaussian splatting, event cameras, and multimodal sensor fusion. Self-supervised depth methods can learn from photometric consistency across views, but moving objects, exposure changes, and occlusions violate the assumptions. Foundation models can transfer across environments, yet their metric accuracy and uncertainty calibration require careful assessment.

The practical trend is toward hybrid systems: geometric constraints provide interpretability and metric consistency, while learned features and representations improve robustness in difficult visual conditions. Future systems are likely to combine cameras, inertial sensors, active depth, semantic understanding, and efficient 3D representations in real time.

Chapter Summary

Stereo vision estimates depth from correspondence between calibrated cameras. In a rectified parallel stereo system, depth is proportional to focal length and baseline and inversely proportional to disparity. Calibration, epipolar geometry, correspondence quality, occlusion handling, and uncertainty determine practical accuracy.

Monocular methods infer depth from visual and learned priors but generally lack absolute scale. Active sensors provide direct range information but introduce their own environmental and hardware limitations. Point clouds, meshes, voxels, signed distance fields, and neural representations offer different trade-offs among accuracy, memory, rendering quality, and editability.

Structure from motion jointly estimates camera motion and 3D structure, while visual odometry estimates motion incrementally and accumulates drift. SLAM adds mapping and loop closure to reduce drift. Multi-view reconstruction and registration transform observations into usable geometric models. Reliable systems depend on calibration, robust estimation, sensor fusion, uncertainty management, dynamic-scene handling, and application-specific evaluation. Privacy, safety, governance, and domain shift are central considerations in professional deployment.

Key Terms

Pinhole camera model: Perspective model relating 3D points to image coordinates.

Intrinsic parameters: Camera properties such as focal length and principal point.

Extrinsic parameters: Camera pose relative to a reference coordinate system.

Baseline: Distance between the optical centers of a stereo camera pair.

Disparity: Image-coordinate difference between corresponding stereo points.

Epipolar geometry: Geometric relationship constraining corresponding points across views.

Fundamental matrix: Matrix encoding epipolar constraints in pixel coordinates.

Triangulation: Estimation of a 3D point from multiple viewing rays.

Rectification: Transformation that makes stereo epipolar lines aligned, usually horizontally.

Point cloud: Collection of sampled 3D points with optional attributes.

Voxel: Volumetric cell in a discretized 3D grid.

Signed distance field: Function storing signed distance from points in space to a surface.

Structure from motion: Joint recovery of camera poses and scene structure from images.

Visual odometry: Incremental estimation of camera motion from visual observations.

SLAM: Simultaneous localization and mapping.

Bundle adjustment: Joint nonlinear optimization of camera parameters and 3D points.

Loop closure: Recognition that a system has returned to a previously mapped place.

ICP: Iterative closest point algorithm for point-cloud alignment.

Neural radiance field: Learned function representing scene density and view-dependent color.

Scale ambiguity: Inability of unconstrained monocular reconstruction to determine absolute size.

Review Questions

Short Answer Questions

1. Why does a single perspective image generally not determine absolute depth?

2. Define stereo disparity and explain its relationship to depth.

3. What is the purpose of stereo rectification?

4. Distinguish visual odometry from SLAM.

5. State two limitations of monocular depth estimation.

Long Answer Questions

1. Explain the complete pipeline for calibrated stereo depth estimation, including calibration, rectification, correspondence, triangulation, and validation.

2. Compare point clouds, meshes, voxel grids, and neural scene representations for 3D reconstruction.

3. Describe structure from motion and explain the role of bundle adjustment and scale ambiguity.

4. Explain the architecture of a practical SLAM system and the importance of loop closure.

5. Discuss the main sources of error in 3D vision and propose engineering practices for reducing them.

Analytical / Application Questions

1. A stereo system has focal length 700 pixels, baseline 0.10 m, and measured disparity 35 pixels. Estimate the depth and explain how a one-pixel disparity error would affect the estimate.

2. Design a 3D perception system for a mobile robot operating both indoors and outdoors. Justify your sensor choices, representation, estimation method, and validation strategy.

3. A learned monocular depth model performs well on daytime urban images but poorly in a dim warehouse. Analyze the likely causes and propose a systematic evaluation and mitigation plan.

Multiple Choice Questions with Answers

Question 1

In a rectified parallel stereo system, which relationship gives depth?

A. Z = d/(fB)

B. Z = fB/d

C. Z = f+d+B

D. Z = f/(B+d)

Correct Answer: B. Z = fB/d

Question 2

What does the stereo baseline represent?

A. The vertical image resolution

B. The distance between the two camera optical centers

C. The focal length of the right camera

D. The average disparity of the scene

Correct Answer: B. The distance between the two camera optical centers

Question 3

Why is stereo depth generally less precise at long range?

A. Long-range objects always have higher texture

B. Their disparity is smaller, so disparity noise causes larger depth error

C. The baseline becomes physically shorter

D. Rectification removes their image features

Correct Answer: B. Their disparity is smaller, so disparity noise causes larger depth error

Question 4

What is the main purpose of epipolar geometry?

A. To increase camera exposure

B. To constrain a correspondence search to an epipolar line

C. To convert a point cloud into a mesh

D. To remove all lens distortion without calibration

Correct Answer: B. To constrain a correspondence search to an epipolar line

Question 5

Which statement about unconstrained monocular structure from motion is correct?

A. It always recovers metric scale directly

B. It cannot estimate camera rotation

C. It is generally determined only up to a global similarity transformation

D. It requires a LiDAR sensor for every frame

Correct Answer: C. It is generally determined only up to a global similarity transformation

Question 6

What problem does bundle adjustment solve?

A. Joint refinement of camera parameters and 3D points using reprojection error

B. Image compression using block transforms

C. Automatic replacement of all moving objects

D. Conversion of RGB images into grayscale

Correct Answer: A. Joint refinement of camera parameters and 3D points using reprojection error

Question 7

What is loop closure used for in SLAM?

A. Increasing the stereo baseline

B. Reducing accumulated drift after recognizing a previously visited place

C. Removing the camera’s intrinsic parameters

D. Estimating object color under changing illumination

Correct Answer: B. Reducing accumulated drift after recognizing a previously visited place

Question 8

Which representation consists of sampled 3D locations that may include color and normals?

A. Point cloud

B. Fundamental matrix

C. Epipolar line

D. Camera intrinsic matrix

Correct Answer: A. Point cloud

Question 9

A stereo camera has f = 800 pixels, B = 0.12 m, and disparity d = 48 pixels. What is the estimated depth?

A. 0.20 m

B. 1.00 m

C. 2.00 m

D. 3.84 m

Correct Answer: C. 2.00 m

Question 10

Why should confidence or uncertainty be retained with a depth map?

A. It guarantees every estimate is correct

B. It allows downstream systems to distinguish reliable from unreliable geometry

C. It eliminates the need for calibration

D. It converts monocular depth into metric depth automatically

Correct Answer: B. It allows downstream systems to distinguish reliable from unreliable geometry