13. Video, 3D and Computational Imaging

Introduction and Context

Image processing traditionally treats an image as a two-dimensional array of intensity or colour values. Many important applications, however, require more than a single image. A video contains measurements distributed across time; a stereo system observes a scene from multiple viewpoints; a depth camera estimates distance; and a medical scanner reconstructs internal structure from projections or slices. Computational imaging extends this idea further by designing the sensing system, acquisition process, and reconstruction algorithm as one integrated system.

These developments change the central question from “How should an image be enhanced?” to “What information is encoded in the measurements, how can it be recovered, and what decisions can be made from it?” Video processing must distinguish genuine motion from illumination changes and sensor noise. Three-dimensional imaging must address correspondence, occlusion, calibration, and incomplete measurements. Computational imaging must solve inverse problems while managing uncertainty, computational cost, and physical limitations.

This chapter examines temporal filtering, optical flow, object tracking, and action understanding; stereo, depth, and volumetric imaging; and advanced systems such as high-dynamic-range, hyperspectral, medical, and computational cameras. The emphasis is on models, practical workflows, trade-offs, and professional judgement.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Explain how video differs from a sequence of independently processed images.

2. Design and evaluate temporal filters for denoising and motion-aware processing.

3. Distinguish apparent motion, optical flow, object tracking, and action recognition.

4. Describe stereo geometry, disparity, depth estimation, and volumetric representation.

5. Compare active and passive depth-sensing systems.

6. Explain the inverse-problem nature of HDR, hyperspectral, medical, and computational imaging.

7. Select suitable methods according to accuracy, latency, cost, privacy, and application constraints.

8. Identify failure modes, ethical concerns, and emerging research directions.

From Images to Measurements over Time and Space

A video may be represented as a function I(x,y,t), where x and y denote spatial coordinates and t denotes time. This representation creates both opportunities and difficulties. Consecutive frames usually contain redundant information, so temporal processing can reduce noise and support prediction. At the same time, motion, occlusion, scene cuts, rolling-shutter distortion, and exposure changes violate the assumption that corresponding pixels remain unchanged.

A three-dimensional scene may be represented by a depth map Z(x,y), a point cloud containing spatial coordinates and attributes, a voxel volume, or a surface model such as a mesh. These representations are not interchangeable. A depth map is efficient for a single viewing direction, whereas a point cloud preserves irregular samples and a mesh expresses connected surfaces. A voxel grid supports volumetric reasoning but may require large memory, especially at high resolution.

Computational imaging systems generally follow the model

y = H(x) + n,

where x is the unknown scene or object, H is a physical measurement operator, y is the observed data, and n represents noise or model error. Reconstruction seeks an estimate of x from y. Since many different scenes may produce similar measurements, reconstruction is often ill-posed and requires calibration, regularization, prior knowledge, or learned models.

Temporal Filtering: Using Time without Blurring Motion

The simplest temporal filter averages corresponding pixels across several frames:

Î(x,y,t) = Σ(k=-m to m) wk I(x,y,t+k),

where the weights wk sum to one. If the scene is stationary and the noise is independent, averaging improves the signal-to-noise ratio. This method fails near motion boundaries because pixels from different objects are combined, producing ghosting and trails.

A temporal median filter is more resistant to isolated disturbances and can remove transient objects when the background remains stable. It is useful for fixed-camera surveillance and impulse-like noise, but it may suppress small or slow-moving objects. Exponential smoothing gives greater weight to recent frames and is computationally inexpensive, but it remains vulnerable to motion unless alignment is performed.

Motion-compensated temporal filtering first estimates displacement and warps neighbouring frames toward a reference frame. The aligned samples can then be fused. Its quality depends on motion estimation, interpolation, occlusion handling, and the reliability of the assumed brightness or colour constancy. Incorrect alignment can produce sharper but structurally incorrect results, a particularly serious problem in scientific and medical video.

Temporal filtering is also used for background modelling. A background estimate may be updated recursively, while foreground detection compares each new frame with that estimate. Adaptive models must respond to gradual illumination changes without absorbing stationary objects that should remain classified as foreground. Practical systems often combine colour, texture, depth, and temporal consistency.

A useful design principle is to make filtering motion-aware. Strong smoothing is appropriate in regions with low motion confidence; near boundaries, the system should reduce temporal support or use edge-aware weights. Processing pipelines should also account for latency. A causal filter uses only current and past frames and can operate in real time, whereas a centred filter needs future frames and introduces delay.

Optical Flow and Motion Models

Optical flow is the apparent velocity field of image brightness. It is not automatically the true three-dimensional velocity of objects. Under the brightness-constancy assumption,

I(x,y,t) = I(x+uΔt, y+vΔt, t+Δt).

Linearisation yields the optical-flow constraint equation:

Ixu + Iyv + It = 0,

where u and v are horizontal and vertical image velocities. This equation contains two unknowns but only one constraint at each pixel, producing the aperture problem. Additional assumptions are therefore required.

Global parametric models, such as translation, affine motion, or projective motion, are effective for camera motion over approximately planar scenes. Local methods estimate flow using neighbourhood smoothness. The classical Horn–Schunck formulation combines brightness consistency with a smoothness penalty, while Lucas–Kanade estimates a locally constant flow by solving a least-squares problem in a window. Modern approaches often use pyramids to handle large displacement and deep networks to learn correspondence from data.

Flow estimation is affected by texturelessness, repetitive patterns, illumination change, motion boundaries, reflections, transparency, and occlusion. Evaluation should therefore include endpoint error, angular error, boundary accuracy, and task-level performance rather than relying on one average metric. In a traffic system, for example, accurate motion boundaries may matter more than small errors in visually uniform road regions.

Camera motion can be estimated from flow or feature correspondences and separated from independent object motion. This distinction supports video stabilisation, visual odometry, panorama construction, and moving-object detection. A rigid scene viewed by a moving camera may exhibit extensive image motion even though no object is independently moving.

Tracking and Action Understanding

Tracking associates an object across frames. A typical tracking-by-detection pipeline detects objects independently, predicts their next states, associates detections with predicted tracks, and updates each track. The state may include position, scale, velocity, and appearance. Kalman filtering is suitable when motion and noise can be approximated by linear-Gaussian models. Particle filters represent nonlinear or multimodal uncertainty, while discriminative and deep trackers learn appearance representations for data association.

Tracking quality is influenced by missed detections, false detections, occlusion, identity switches, and changes in object appearance. A robust system should maintain uncertainty rather than treating every estimate as exact. Re-identification features can help recover an object after temporary occlusion, but they may also create privacy and fairness concerns.

Action understanding operates at a higher semantic level. It may involve action classification, temporal localisation, gesture recognition, event detection, or prediction of future activity. Methods range from hand-designed trajectories and spatiotemporal interest points to three-dimensional convolutional networks, recurrent models, temporal transformers, and skeleton-based graph networks. A reliable pipeline commonly combines frame appearance, motion, object interactions, body pose, and scene context.

The temporal scale of an action is important. A short gesture may be recognised from a few frames, whereas activities such as assembly or rehabilitation require long-range temporal reasoning. Dataset bias is a major limitation: a model trained in staged indoor settings may fail in crowded, culturally diverse, or poorly illuminated environments. Evaluation should distinguish recognition accuracy from event timing, calibration, robustness, and performance across demographic and environmental groups.

Stereo, Depth, and Three-Dimensional Geometry

In calibrated rectified stereo, corresponding points lie on the same image row. If f is focal length, B is the baseline between cameras, and d is disparity, depth is approximately

Z = fB/d.

Depth becomes highly sensitive to disparity error when d is small, meaning that distant objects are difficult to measure accurately. Increasing the baseline improves depth sensitivity but also increases occlusion and correspondence difficulty. Stereo matching typically involves cost computation, aggregation, disparity selection, and refinement. Costs may compare intensity, colour, gradients, or learned feature descriptors.

Correspondence is difficult in textureless areas, repetitive patterns, thin structures, reflective surfaces, and regions visible in only one camera. Occlusion reasoning and left-right consistency checks can identify unreliable estimates. Calibration errors, lens distortion, unsynchronised capture, and rolling shutter can produce systematic depth errors.

Depth cameras may be passive or active. Passive stereo uses ambient visual texture. Structured-light systems project a known pattern and infer deformation, whereas time-of-flight systems measure the travel time or phase of emitted light. LiDAR measures distances using laser pulses or frequency-modulated signals. Active systems can function in textureless scenes but may suffer from sunlight interference, multipath reflections, limited range, or eye-safety constraints.

Three-dimensional data can be processed in camera coordinates, transformed into a world coordinate system, and fused across views. Registration methods such as iterative closest point align point clouds by minimising geometric discrepancy, usually after an initial alignment. Surface reconstruction may produce meshes, signed-distance volumes, or neural representations. The choice depends on whether the application values compactness, topology, rendering quality, or measurement accuracy.

Volumetric and Medical Imaging

A volume is a scalar or vector field sampled over three spatial dimensions. Voxel operations include interpolation, filtering, segmentation, registration, and rendering. Direct volume rendering maps voxel values to colour and opacity, allowing internal structures to be inspected without explicitly extracting a surface. Surface rendering first estimates boundaries and then displays a polygonal model.

Medical imaging illustrates the inverse-problem character of three-dimensional reconstruction. Computed tomography reconstructs attenuation from many X-ray projections; magnetic resonance imaging reconstructs spatial structure from measurements in frequency-related domains; ultrasound forms images from reflected acoustic energy; and positron emission tomography estimates radiotracer distribution. Each modality has distinct resolution, noise, contrast, acquisition time, and safety considerations.

Reconstruction algorithms may be analytic, iterative, or learned. Iterative methods can incorporate physical models and regularisation but may be computationally expensive. Deep reconstruction can accelerate inference and improve visual quality, yet a visually plausible image may contain hallucinated or suppressed structures. Clinical deployment therefore requires validation against diagnostic outcomes, uncertainty estimation, auditability, and human oversight.

High-Dynamic-Range and Hyperspectral Imaging

A conventional sensor has limited dynamic range. Multiple exposures can be combined into a high-dynamic-range radiance estimate, followed by tone mapping for display. The pipeline must account for exposure response, sensor saturation, noise, camera motion, and moving objects. Ghosting occurs when different exposures contain different scene content. Exposure fusion can produce attractive images without explicitly estimating radiance, but aesthetic quality and photometric accuracy are different objectives.

Hyperspectral imaging records many narrow spectral bands rather than only three broad colour channels. A pixel is therefore represented by a spectral signature. Applications include material identification, agriculture, mineral analysis, food inspection, environmental monitoring, and remote sensing. Important processing steps include radiometric calibration, atmospheric correction, dimensionality reduction, spectral unmixing, classification, and anomaly detection.

Hyperspectral data are high-dimensional and often highly correlated. Principal component analysis, independent component analysis, sparse models, and learned spectral-spatial networks can reduce dimensionality or improve classification. However, spectral signatures vary with illumination, surface geometry, moisture, and sensor characteristics. Models trained in one location may not transfer reliably to another.

Computational Photography

Computational photography deliberately combines optics, sensors, controlled illumination, and algorithms. Examples include panoramic imaging, light-field cameras, coded apertures, focus stacking, burst super-resolution, deblurring, low-light imaging, and event-based cameras. The system may capture measurements that appear incomplete or distorted but are easier to reconstruct computationally.

A coded aperture, for example, modifies the optical measurement so that information is distributed across the sensor. Reconstruction then inverts the coded measurement using calibration and regularisation. Light-field systems record directional information, enabling refocusing and viewpoint changes, usually at the cost of spatial resolution or sensor complexity. Event cameras report asynchronous brightness changes rather than conventional frames, offering high temporal resolution and low latency but requiring specialised reconstruction and algorithms.

The central engineering trade-off is not simply image quality. It includes exposure time, bandwidth, power, latency, interpretability, storage, privacy, and robustness. A computational camera designed for scientific measurement must preserve quantitative fidelity, whereas a consumer camera may optimise perceptual quality.

Integrated Workflow for Advanced Imaging Systems

A dependable system is developed through a sequence of decisions:

1. Define the task and the required physical quantity, such as radiance, depth, velocity, material class, or anatomical structure.

2. Select the sensing geometry, spectral range, temporal sampling, and exposure strategy.

3. Calibrate intrinsic parameters, extrinsic relationships, radiometric response, timing, and sensor-specific distortions.

4. Construct a reconstruction pipeline with explicit uncertainty and confidence measures.

5. Validate on representative scenes, including difficult cases and distribution shifts.

6. Integrate downstream decisions, human review, logging, and failure handling.

This workflow prevents a common mistake: selecting an algorithm before identifying what the sensor actually measures. A high-performing model cannot reliably recover information that was never captured or that was destroyed by saturation, occlusion, inadequate sampling, or poor calibration.

Illustrative Case Study: Depth-Aware Safety Monitoring in a Warehouse

Background

A logistics company wishes to reduce collisions between autonomous forklifts and human workers. The warehouse contains narrow aisles, reflective packaging, variable illumination, and frequent partial occlusions.

Situation and Problem

A monocular video detector identifies people but produces unreliable distance estimates. A conventional frame-difference system generates false alarms when lighting changes, while optical flow alone cannot distinguish camera vibration from human motion.

Analysis

The safety requirement is not merely object classification. The system must estimate three-dimensional proximity, track identities over time, and issue alerts with low latency. False negatives are dangerous, but excessive false positives cause operators to ignore warnings. The environment also requires careful treatment of worker privacy.

Approach

The company installs calibrated stereo cameras at aisle entrances and supplements them with inertial measurements for camera-motion compensation. A stereo pipeline produces disparity and confidence maps. A detector identifies people and forklifts, while a multi-object tracker combines Kalman prediction, appearance features, and three-dimensional positions. Temporal filtering is applied only after motion compensation and is reduced near object boundaries. A risk module estimates time-to-collision rather than using distance alone. Events are logged using anonymised track identifiers, and human supervisors review uncertain cases.

Outcome

The integrated design is more reliable than any single component because depth, motion, and semantic identity provide complementary evidence. Remaining failures occur when a person is fully occluded, reflective surfaces corrupt stereo matching, or cameras lose synchronisation. The system therefore reports confidence and uses conservative behaviour when sensing quality falls below a threshold.

Learning Points

The case demonstrates that advanced imaging is a systems problem. Calibration, sensor placement, uncertainty, temporal reasoning, and operational policy are as important as the neural detector. It also shows why evaluation should focus on safety events, latency, and false-alarm burden rather than only frame-level accuracy.

Challenges, Risks, and Professional Considerations

Advanced imaging systems commonly fail through domain shift, calibration drift, missing data, sensor interference, and unrecognised uncertainty. Temporal models may amplify errors across frames; tracking systems may lock onto the wrong identity; and reconstruction algorithms may produce visually convincing but physically false details. Reproducibility requires recording sensor settings, calibration versions, preprocessing, model versions, and evaluation protocols.

Privacy is particularly important for video, face analysis, body-pose estimation, and medical imaging. Data collection should follow purpose limitation, access control, retention limits, and applicable data-protection law. Biometric or behavioural inference should not be deployed merely because it is technically possible. Medical systems require informed governance, clinical validation, secure handling of patient data, and clear allocation of responsibility between the algorithm and professional users.

Fairness must be evaluated across lighting conditions, skin tones, body types, languages of gesture, age groups, and environments where relevant. Security threats include adversarial patterns, sensor spoofing, replay attacks, poisoned training data, and unauthorised reconstruction of sensitive scenes. Safety-critical applications should include fail-safe states, redundancy, monitoring, and a mechanism for human override.

Current Developments and Best Practices

Current research is moving toward multimodal systems that combine RGB, depth, thermal, audio, event streams, and language. Neural rendering and radiance-field methods support view synthesis and scene reconstruction, although their geometry and uncertainty must be carefully assessed. Foundation models are increasingly used for video representation, segmentation, tracking, and open-vocabulary scene understanding. Edge accelerators and event-driven hardware reduce latency and energy consumption.

Best practice involves matching the representation to the task, calibrating every sensor, preserving uncertainty, testing under realistic operating conditions, and separating perceptual enhancement from quantitative measurement. Engineers should retain raw or minimally processed data when feasible, document assumptions, monitor drift after deployment, and design explicit failure responses. A sophisticated model is not a substitute for an appropriate sensor, a valid physical model, or responsible governance.

Chapter Summary

Video and three-dimensional imaging extend image processing into temporal, geometric, spectral, and physical dimensions. Temporal filters improve noise suppression but must account for motion and latency. Optical flow estimates apparent image motion, while tracking maintains object identity and action understanding interprets longer temporal patterns. Stereo and active depth systems recover geometry through correspondence or controlled illumination, but accuracy depends strongly on calibration, disparity, occlusion, and sensing conditions. Volumetric and medical imaging are inverse problems requiring physical modelling, regularisation, and validation. HDR, hyperspectral, and computational photography systems jointly design acquisition and reconstruction to recover information beyond ordinary cameras. Across all applications, uncertainty, domain shift, privacy, fairness, safety, and operational context are central to professional decision-making.

Key Terms

Optical flow: Apparent motion of image brightness between successive frames.

Temporal filtering: Combining measurements across time to reduce noise or estimate changing signals.

Motion compensation: Alignment of frames according to estimated displacement before temporal fusion.

Aperture problem: Ambiguity in estimating motion from a locally observed one-dimensional image gradient.

Object tracking: Estimating an object’s state and identity across successive frames.

Action recognition: Classification or interpretation of human or object activity over time.

Disparity: Relative horizontal displacement of corresponding points in a rectified stereo pair.

Depth map: Image whose pixels encode distance from the camera or sensor.

Point cloud: Set of sampled three-dimensional points, often with colour or intensity attributes.

Voxel: A volumetric sample occupying a location in a three-dimensional grid.

Structured light: Active depth method using a projected pattern to infer surface geometry.

Time of flight: Distance measurement based on the travel time or phase of emitted energy.

Volumetric rendering: Display of a three-dimensional scalar field using colour and opacity mappings.

High-dynamic-range imaging: Recovery or combination of scene information across a range exceeding ordinary exposure capability.

Hyperspectral imaging: Acquisition of many narrow spectral bands for each spatial location.

Spectral unmixing: Estimation of constituent material spectra and their proportions in a mixed pixel.

Inverse problem: Estimation of an underlying scene or object from indirect measurements.

Regularisation: Additional constraint or prior used to stabilise an ill-posed reconstruction.

Radiance: Directional light power emitted or reflected by a scene, often estimated in HDR systems.

Computational photography: Joint design of optics, sensing, and algorithms to obtain or reconstruct images.

Review Questions

Short Answer Questions

1. Why does simple temporal averaging produce ghosting near moving objects?

2. State the optical-flow constraint equation and explain its variables.

3. Why does stereo depth become less reliable for distant objects?

4. Distinguish a depth map from a point cloud.

5. What is the purpose of regularisation in computational imaging?

Long Answer Questions

1. Explain temporal filtering methods and discuss how motion compensation changes their performance.

2. Compare Lucas–Kanade, Horn–Schunck, parametric motion models, and modern learned optical-flow methods.

3. Describe the complete stereo-imaging pipeline, including calibration, matching, disparity refinement, and error sources.

4. Explain how medical imaging reconstruction differs across CT, MRI, ultrasound, and PET.

5. Discuss the design, applications, and limitations of hyperspectral imaging and HDR imaging.

Analytical / Application Questions

1. A surveillance camera is fixed in a corridor where fluorescent lights periodically flicker and people walk through the scene. Design a temporal background-subtraction strategy and justify the choice of filtering, update policy, and failure handling.

2. A mobile robot uses stereo cameras in a warehouse with repetitive shelves and reflective packaging. Analyse the likely depth errors and propose sensor, calibration, and algorithmic improvements.

3. A hospital proposes an AI system that reconstructs low-dose CT images. Develop an evaluation and governance plan that addresses image quality, hallucinated structures, uncertainty, clinical validation, privacy, and human oversight.

22. Multiple Choice Questions with Answers

Question 1

Why does unaligned temporal averaging often create ghosting?

A. It increases the camera’s spatial resolution

B. It combines pixels belonging to different object positions

C. It removes all high-frequency noise before sampling

D. It converts RGB values into depth values

Correct Answer: B. It combines pixels belonging to different object positions

Question 2

The optical-flow constraint equation is underdetermined at an individual pixel because it contains:

A. Two velocity components but one local equation

B. One velocity component but two independent equations

C. No dependence on image gradients

D. Only depth and focal-length variables

Correct Answer: A. Two velocity components but one local equation

Question 3

In rectified stereo, increasing disparity generally indicates that the object is:

A. Farther from the cameras

B. Closer to the cameras

C. Moving faster vertically

D. Less textured

Correct Answer: B. Closer to the cameras

Question 4

Which condition most directly makes stereo correspondence difficult?

A. Distinct texture visible in both images

B. Accurate camera synchronisation

C. A textureless or repetitive surface

D. A large, correctly calibrated baseline

Correct Answer: C. A textureless or repetitive surface

Question 5

What is the main purpose of motion compensation before temporal fusion?

A. To align corresponding scene points across frames

B. To reduce the camera focal length

C. To convert a point cloud into a histogram

D. To eliminate the need for calibration

Correct Answer: A. To align corresponding scene points across frames

Question 6

Which statement best distinguishes optical flow from object tracking?

A. Optical flow estimates apparent pixel motion, while tracking maintains object state and identity

B. Optical flow applies only to depth images, while tracking applies only to still images

C. Tracking estimates brightness gradients, while optical flow estimates object categories

D. They are identical operations with different names

Correct Answer: A. Optical flow estimates apparent pixel motion, while tracking maintains object state and identity

Question 7

A time-of-flight depth sensor estimates distance primarily from:

A. The colour histogram of the scene

B. The travel time or phase of emitted energy

C. The number of detected image corners

D. The compression ratio of the video

Correct Answer: B. The travel time or phase of emitted energy

Question 8

Why are computational imaging systems commonly described as inverse problems?

A. They infer an underlying scene from indirect or incomplete measurements

B. They always require two cameras viewing the same object

C. They operate only on inverted image colours

D. They avoid the use of physical sensor models

Correct Answer: A. They infer an underlying scene from indirect or incomplete measurements

Question 9

What is a distinctive advantage of hyperspectral imaging?

A. It records only one broad intensity channel

B. It provides many narrow spectral measurements useful for material identification

C. It eliminates illumination effects in every environment

D. It requires no dimensionality reduction or calibration

Correct Answer: B. It provides many narrow spectral measurements useful for material identification

Question 10

Why should a medical image reconstruction model be evaluated beyond visual quality?

A. Visual plausibility guarantees diagnostic correctness

B. A model may produce plausible but physically false structures

C. Medical images contain no measurement uncertainty

D. Clinical users cannot interpret reconstructed images

Correct Answer: B. A model may produce plausible but physically false structures