9. Geometric Processing, Registration and Image Fusion

Introduction and Context

Geometric processing concerns the spatial arrangement of image information. Whereas enhancement methods modify intensity, contrast, or noise characteristics, geometric methods alter the relationship between image coordinates and the objects represented in an image. They are therefore essential whenever images must be aligned, compared, combined, tracked, or interpreted across time, viewpoint, sensor, or modality.

A satellite image may need correction for terrain and sensor geometry; a medical image acquired by magnetic resonance imaging may need alignment with a computed tomography scan; and frames from a moving camera may need registration before a panoramic mosaic is produced. In each case, the central problem is to determine how coordinates in one image correspond to coordinates in another.

This chapter develops the mathematical and practical foundations of geometric transformations, interpolation, image registration, motion estimation, homography, mosaicing, and image fusion. Particular attention is given to the assumptions behind each method, because a geometrically elegant model can still produce an incorrect result when the scene, sensor, or imaging process violates those assumptions.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Explain geometric transformations and distinguish rigid, affine, projective, and nonrigid models.

2. Describe forward and inverse warping and select appropriate interpolation methods.

3. Formulate feature-based and intensity-based image-registration problems.

4. Estimate motion and homography using robust methods such as RANSAC.

5. Design a basic image-mosaicing workflow.

6. Compare image-fusion strategies for multimodal and multiresolution data.

7. Identify registration errors, model limitations, and professional risks.

8. Evaluate contemporary optimization and learning-based approaches critically.

9.1 Coordinate Mappings and Geometric Models

Let an image be represented by a function I(x, y), where (x, y) denotes a spatial coordinate and I denotes intensity or a vector of spectral values. A geometric transformation maps a point in one coordinate system to another:

x' = T(x).

In homogeneous coordinates, a two-dimensional point is written as p = [x, y, 1]^T. This representation permits several important transformations to be expressed using matrices.

Translation, rotation and scaling

A translation by (t_x, t_y) is represented by

T = [[1, 0, t_x], [0, 1, t_y], [0, 0, 1]].

A rotation through angle θ about the origin is

R = [[cos θ, -sin θ, 0], [sin θ, cos θ, 0], [0, 0, 1]].

Scaling by factors s_x and s_y is represented by

S = [[s_x, 0, 0], [0, s_y, 0], [0, 0, 1]].

A rigid transformation combines rotation and translation and preserves distances and angles. It is appropriate for planar objects or camera motion in which relative shape is not changed. Similarity transformations add uniform scale and preserve angles, while affine transformations permit unequal scaling and shear. An affine mapping has the form

x' = a_1x + a_2y + a_3,

y' = a_4x + a_5y + a_6.

It preserves parallel lines but not necessarily lengths or angles.

Projective transformation and homography

A projective transformation, or homography, maps points according to

[x', y', w']^T = H[x, y, 1]^T,

followed by x_h = x'/w' and y_h = y'/w'. The matrix H has eight independent parameters because multiplication of H by a nonzero scalar produces the same mapping. At least four non-collinear point correspondences are required to estimate it in the absence of noise.

Homography is exact when two views observe the same physical plane, or when the camera undergoes pure rotation around its optical centre. It is often a useful approximation for scenes with limited depth variation. It is not generally valid for a three-dimensional scene observed under arbitrary camera translation, because different depths produce different image motions.

Nonrigid transformations

Deformable registration is required when anatomy changes shape, when an object bends, or when motion varies spatially. Transformation models include thin-plate splines, B-splines, elastic fields, optical-flow displacement fields, and diffeomorphic mappings. These models offer greater flexibility but introduce more parameters and a greater risk of biologically or physically implausible deformation. Regularization is therefore essential.

9.2 Resampling: From Continuous Mappings to Pixels

A transformation usually maps source pixels to noninteger target coordinates. Resampling estimates the image value at those coordinates. Two implementation strategies are common.

In forward warping, each source pixel is projected into the target image. This can produce holes, overlaps, and uneven sampling. In inverse warping, each target pixel is mapped backward through T^{-1} to the source image, where its value is interpolated. Inverse warping generally provides more complete target coverage and is widely preferred.

Nearest-neighbour interpolation assigns the value of the closest source pixel. It is fast and preserves categorical labels, but it produces block-like boundaries. Bilinear interpolation uses the four neighbouring pixels and gives a continuous piecewise-linear estimate. Bicubic interpolation uses a larger neighbourhood and usually produces smoother visual results, although it can introduce ringing near sharp edges. Sinc-based or windowed-sinc interpolation can preserve frequency content more accurately but is computationally more demanding.

The choice must reflect the data. Nearest-neighbour interpolation is appropriate for segmentation masks and land-cover labels; bilinear or bicubic interpolation is suitable for continuous intensity imagery. Interpolating a class label with bilinear interpolation creates meaningless intermediate classes.

Resampling also changes resolution and noise statistics. Repeated warping can blur edges, while downsampling without suitable low-pass filtering causes aliasing. A sound workflow therefore applies antialiasing before reduction and avoids unnecessary repeated transformations by composing transformations into a single warp.

9.3 Image Registration as an Estimation Problem

Image registration seeks a transformation T that aligns a moving image I_M with a fixed image I_F. A general formulation is

T* = arg min_T D[I_F, I_M ∘ T] + λR(T),

where D is a data or similarity term, R is a regularization term, and λ controls the balance between image agreement and transformation smoothness or plausibility.

Registration contains four decisions: the transformation model, the similarity measure, the optimization method, and the sampling or multiresolution strategy. A failure in any one of these decisions can lead to a visually plausible but incorrect alignment.

Intensity-based registration

Intensity-based methods compare image values directly. For images acquired by the same sensor under comparable conditions, sum of squared differences is useful:

SSD(T) = Σ[I_F(x) - I_M(T(x))]^2.

The sum of absolute differences is less sensitive to isolated outliers. Cross-correlation and normalized cross-correlation are effective when brightness may vary linearly. Mutual information is especially important for multimodal registration. It measures statistical dependence between intensities and does not require corresponding tissues to have identical brightness values.

Optimization may use gradient descent, Gauss–Newton methods, coordinate search, or evolutionary strategies. Since similarity surfaces can contain local optima, coarse-to-fine image pyramids are commonly used. Large-scale structure is aligned at low resolution before finer details are optimized.

Feature-based registration

Feature-based methods first detect salient structures such as corners, blobs, edges, or invariant regions. Detectors such as Harris, Shi–Tomasi, and scale-space methods identify candidate points. Descriptors encode local appearance; examples include SIFT-like gradient descriptors, binary descriptors, and learned descriptors. Matching produces tentative correspondences between the two images.

Outliers are unavoidable. Repeated textures, illumination changes, perceptual ambiguity, and incorrect descriptors can generate false matches. Robust estimation, particularly RANSAC, repeatedly selects the minimum number of correspondences needed by a model, estimates the model, and counts inliers within a geometric error threshold. The model with the strongest consistent support is retained, often followed by nonlinear refinement using all inliers.

Feature-based registration is efficient when images contain distinctive structures and the displacement is large. It is less reliable in textureless regions, repetitive patterns, or images with substantial modality differences.

Hybrid registration

A practical system often combines both approaches. Feature matches provide an initial transformation, after which intensity optimization refines the alignment. This division exploits the large capture range of features and the subpixel precision of intensity methods.

9.4 Motion Estimation and Optical Flow

Registration between successive video frames is frequently described as motion estimation. The optical-flow assumption states that the intensity of a moving point remains approximately constant over a short interval:

I(x, y, t) = I(x + uΔt, y + vΔt, t + Δt).

Linearization yields the optical-flow constraint equation

I_xu + I_yv + I_t = 0,

where (u, v) is the local velocity. This equation contains two unknowns but provides only one constraint at each pixel, creating the aperture problem. Additional assumptions are therefore required. The Lucas–Kanade method assumes locally constant motion and solves a least-squares problem over a neighbourhood. The Horn–Schunck method imposes global smoothness. Modern methods may use pyramids, robust penalties, recurrent networks, or transformer-based architectures.

Optical flow is useful for motion segmentation, video stabilization, autonomous navigation, and activity analysis. It can fail at occlusions, motion boundaries, large displacements, brightness changes, and textureless surfaces. Confidence estimation and explicit occlusion handling are important in safety-critical applications.

9.5 Mosaicing and Panoramic Reconstruction

Image mosaicing combines overlapping images into a larger composite. A typical workflow is:

1. Acquire images with sufficient overlap.

2. Detect and describe features.

3. Match features and reject outliers.

4. Estimate pairwise or global transformations.

5. Choose a reference coordinate system.

6. Warp images into the mosaic plane.

7. Compensate exposure and blend seams.

8. Crop or geometrically regularize the final result.

Pairwise registration can accumulate drift. A panorama with many images is better treated as a graph whose nodes are images and whose edges are reliable pairwise transformations. Global adjustment, often called bundle adjustment in camera systems, minimizes reprojection error across the graph.

Seam selection and blending are not merely cosmetic. Exposure differences, parallax, moving objects, and lens distortion can create visible discontinuities. Feathering is simple but may produce ghosting. Multiband blending combines low-frequency illumination smoothly while preserving high-frequency detail. When parallax is significant, a single homography is insufficient; local warps, depth-aware methods, or three-dimensional reconstruction may be necessary.

9.6 Image Fusion: Combining Complementary Evidence

Image fusion produces an image or information representation that combines multiple observations. The sources may differ in spatial resolution, spectral range, focus, exposure, viewpoint, or sensing physics.

At the pixel level, fusion rules may include averaging, weighted averaging, maximum selection, or local-activity measures. These methods are simple but sensitive to misregistration. Transform-domain methods first decompose images using wavelets, pyramids, curvelets, or shearlets, combine corresponding coefficients, and reconstruct the fused image. Low-frequency coefficients generally represent illumination and broad structure, while high-frequency coefficients represent edges and fine detail.

In multiresolution fusion, a common strategy is to select or weight high-frequency coefficients according to local salience and combine low-frequency coefficients using a controlled average. The weights may be fixed, learned, or derived from local contrast, gradient magnitude, entropy, or signal-to-noise estimates.

For multimodal data, fusion may occur at three levels. Pixel-level fusion combines registered measurements directly. Feature-level fusion extracts structures or descriptors before combining them. Decision-level fusion combines outputs from separate classifiers or detectors. Pixel-level fusion preserves the most raw information but demands accurate registration and compatible sampling; decision-level fusion is more tolerant of misalignment but may discard detail.

Fusion quality should not be judged only by visual sharpness. Useful criteria include mutual information, structural similarity, edge preservation, spatial-frequency measures, task-specific detection accuracy, and human interpretability. A fused image can look attractive while suppressing diagnostically important information or introducing features that exist in neither source.

9.7 Applied Case: Registration and Fusion of CT and MRI for Surgical Planning

Background

The following is an illustrative case involving a hospital planning system. Computed tomography provides relatively strong information about bone and spatial density, whereas magnetic resonance imaging provides superior contrast for many soft tissues. A surgeon requires a common spatial view before planning an intervention.

Situation and problem

The CT and MRI studies were acquired at different times, with different patient positioning and different voxel dimensions. Direct overlay produced displaced anatomical boundaries. A simple intensity-difference registration was inappropriate because corresponding tissues did not have the same intensity relationship.

Analysis

The registration problem involved both rigid misalignment and possible local anatomical change. Mutual information was selected for the initial multimodal alignment. Because the scans had different resolutions, both were resampled into a common physical coordinate system, with careful preservation of metadata and orientation. A rigid transformation was estimated first. If residual deformation was clinically justified, a constrained deformable model was considered, but unrestricted warping was rejected because it could alter anatomical relationships.

Approach

The workflow began with masking irrelevant background regions and correcting gross orientation. A multiresolution mutual-information registration estimated six rigid parameters. Landmark checks were then performed at anatomical points not used directly in optimization. A deformable refinement, when required, used a smoothness-regularized displacement field and was accepted only when registration error remained within the clinical tolerance defined by the institution.

Fusion was performed primarily at the visualization level rather than by producing a single undifferentiated image. CT information was displayed with a bone-sensitive transfer function, while MRI information was overlaid using controlled transparency and colour mapping. This approach preserved source identity and reduced the risk that a synthetic blended intensity would be interpreted as a direct measurement.

Outcome

The aligned display allowed the surgical team to inspect soft-tissue structures in relation to bone landmarks. Quality assurance included landmark error measurements, inspection for implausible deformation, and comparison with the original studies. The system was not approved on visual appearance alone.

Learning points

The case illustrates that multimodal registration depends more on an appropriate similarity measure and physical coordinate handling than on simple pixel correspondence. It also demonstrates that fusion is a communication and decision-support problem: preserving provenance may be more valuable than maximizing visual smoothness.

9.8 Challenges, Failure Modes and Professional Practice

Registration is underdetermined when the image contains little texture or when many transformations produce similar similarity scores. Repetitive patterns can create geometrically consistent but incorrect matches. Occlusions and independently moving objects violate a single global motion model. Illumination changes, sensor noise, lens distortion, rolling-shutter effects, and inaccurate metadata further complicate the task.

Common mistakes include using an affine model for severe perspective effects, estimating a homography from collinear points, applying interpolation designed for continuous intensities to labels, ignoring image orientation, and judging success only by a visually pleasing overlay. Another error is reporting a registration result without uncertainty or landmark validation.

A professional workflow should retain the original images, transformation parameters, coordinate conventions, resampling settings, and quality metrics. In medical, defence, industrial inspection, and geospatial applications, traceability is essential. Registration should not create unsupported evidence: a fused boundary may reflect interpolation, an optimization preference, or a sensor artefact rather than a physical structure.

Privacy and governance are also relevant. Medical images contain protected information, while geospatial imagery may expose sensitive locations. Automated systems should use access controls, audit trails, documented validation data, and human oversight where errors could affect safety or rights.

9.9 Contemporary Developments

Deep learning has changed feature extraction, optical flow, registration, and fusion. Learned descriptors can improve matching under viewpoint and illumination changes. Neural registration networks can predict transformations rapidly after training, and generative models can reconstruct missing or low-resolution information. However, speed and visual quality do not establish geometric correctness. Networks may learn dataset-specific biases, fail under domain shift, or hallucinate structures during fusion or super-resolution.

Current research increasingly combines physical models with learning. Examples include differentiable warping, self-supervised photometric objectives, uncertainty-aware registration, neural implicit representations, and multimodal foundation models. The strongest systems typically retain explicit geometric constraints, robust validation, and task-specific evaluation rather than replacing measurement with appearance generation.

Best-Practice Workflow

A reliable project begins by defining the task: alignment, measurement, visualization, tracking, or decision support. The coordinate systems and physical units should then be verified. The simplest transformation model consistent with the scene should be selected, followed by appropriate preprocessing and feature or intensity selection. Registration should be performed at multiple scales, with robust estimation used when outliers are possible. Results should be assessed using independent landmarks, residual plots, overlap measures, and application-specific performance. Finally, the transformed data and provenance should be stored so that the result can be reproduced and audited.

Chapter Summary

Geometric processing changes the spatial organization of image data through transformations ranging from rigid motion to nonrigid deformation. Resampling converts continuous coordinate mappings into discrete pixels and must be chosen according to the data type. Registration estimates a transformation between images using intensity similarity, distinctive features, or a hybrid strategy. Homography is powerful for planar scenes and rotational panoramas but is not a universal model for three-dimensional motion. Optical flow estimates local image motion under assumptions about brightness and spatial regularity. Mosaicing extends registration into global alignment, blending, and drift correction. Image fusion combines complementary observations at pixel, feature, or decision level, but its value depends on accurate registration, preservation of source meaning, and task-specific evaluation. Robust practice requires validation, uncertainty awareness, provenance, and caution against visually plausible but physically incorrect results.

Key Terms

Geometric transformation: A mapping that changes the spatial coordinates of image points.

Rigid transformation: A rotation and translation that preserve distances and angles.

Affine transformation: A linear transformation with translation that preserves parallelism.

Homography: A projective mapping between two planes represented by a 3 × 3 matrix.

Warping: Applying a geometric transformation to relocate image samples.

Resampling: Estimating image values at transformed, generally noninteger, coordinates.

Interpolation: Estimation of values between known samples.

Image registration: Alignment of two or more images into a common coordinate system.

Fixed image: The reference image whose coordinate system is retained.

Moving image: The image transformed during registration.

Mutual information: A statistical measure of dependence useful for multimodal registration.

Feature descriptor: A numerical representation of local image appearance.

RANSAC: A robust method that estimates a model from data containing outliers.

Optical flow: A field describing apparent local motion between image frames.

Parallax: Apparent displacement caused by viewpoint change and scene depth.

Mosaicing: Combining overlapping images into a larger composite.

Image fusion: Integration of information from multiple images or sensors.

Multiresolution decomposition: Representation of an image at several spatial scales.

Regularization: A constraint or penalty that favors stable or plausible solutions.

Reprojection error: The discrepancy between observed points and points predicted by a geometric model.

Review Questions

Short Answer Questions

1. Why is inverse warping generally preferred to forward warping?

2. What physical situations justify the use of a homography?

3. Why is mutual information useful for multimodal image registration?

4. State the aperture problem in optical-flow estimation.

5. Why should nearest-neighbour interpolation be used for segmentation labels?

Long Answer Questions

1. Explain rigid, similarity, affine, projective, and nonrigid transformations, including their assumptions and applications.

2. Describe a complete feature-based registration pipeline and explain how RANSAC improves its reliability.

3. Derive the optical-flow constraint equation and discuss the assumptions and limitations of Lucas–Kanade and Horn–Schunck methods.

4. Explain the stages of panoramic image mosaicing, including drift correction, seam handling, and blending.

5. Compare pixel-level, feature-level, and decision-level image fusion for a multimodal medical application.

Analytical / Application Questions

1. Two aerial images contain a flat agricultural field but were captured from different viewpoints. Determine whether an affine transformation or homography is more appropriate, and justify your decision.

2. A panorama contains severe ghosting around pedestrians despite accurate feature matches. Diagnose the likely causes and propose a revised mosaicing strategy.

3. A fused CT–MRI display appears sharp but shifts a tumour boundary by several millimetres. Explain how the error could arise and design a validation procedure before clinical use.

22. Multiple Choice Questions with Answers

Question 1

Which transformation is most appropriate for a planar scene viewed from two different camera positions?

A. Pure translation only

B. Homography

C. Random deformation

D. One-dimensional scaling

Correct Answer: B. Homography

Question 2

Why is inverse warping commonly preferred in image transformation?

A. It eliminates the need for interpolation

B. It guarantees exact preservation of all frequencies

C. It maps every target pixel back to a source location and reduces holes

D. It can be used only with nearest-neighbour interpolation

Correct Answer: C. It maps every target pixel back to a source location and reduces holes

Question 3

Which similarity measure is especially suitable when corresponding structures have different intensity relationships across modalities?

A. Mutual information

B. Raw pixel equality

C. Binary threshold overlap only

D. Unscaled coordinate distance

Correct Answer: A. Mutual information

Question 4

What is the principal purpose of RANSAC in feature-based registration?

A. To increase image bit depth

B. To reject outlier correspondences while estimating a geometric model

C. To replace image interpolation

D. To remove all image noise before feature detection

Correct Answer: B. To reject outlier correspondences while estimating a geometric model

Question 5

The optical-flow constraint equation is underdetermined at an individual pixel because it contains:

A. No spatial derivatives

B. Two motion components but only one equation

C. Only a temporal derivative

D. A complete three-dimensional velocity vector

Correct Answer: B. Two motion components but only one equation

Question 6

Which interpolation method is most appropriate for a categorical land-cover mask?

A. Bicubic interpolation

B. Windowed-sinc interpolation

C. Nearest-neighbour interpolation

D. Bilinear interpolation

Correct Answer: C. Nearest-neighbour interpolation

Question 7

A single homography performs poorly when stitching images of a deep three-dimensional scene because:

A. Homographies cannot represent any rotation

B. Different scene depths undergo different image motions

C. Feature descriptors cannot be computed on colour images

D. Interpolation is impossible after projective mapping

Correct Answer: B. Different scene depths undergo different image motions

Question 8

Which operation is most directly intended to reduce drift in a large panorama?

A. Global adjustment over the image-connection graph

B. Replacing all features with random points

C. Applying nearest-neighbour interpolation repeatedly

D. Discarding all pairwise transformations

Correct Answer: A. Global adjustment over the image-connection graph

Question 9

Why is regularization important in deformable registration?

A. It forces every image to have identical intensity values

B. It encourages smooth or physically plausible deformations

C. It guarantees that no transformation is estimated

D. It converts a nonrigid model into a translation

Correct Answer: B. It encourages smooth or physically plausible deformations

Question 10

A visually sharp fused image contains structures absent from both input images. What is the most important concern?

A. The image necessarily has perfect registration

B. The fusion process may have introduced unsupported or hallucinated information

C. Sharpness proves that the sensor calibration is correct

D. The image must be evaluated only by its file size

Correct Answer: B. The fusion process may have introduced unsupported or hallucinated information