2. Digital Images, Image Formation, and Mathematical Foundations

Introduction and Context

Computer vision begins with a physical scene and ends with a computational interpretation. Between these two points lies a chain of transformations: light is reflected by objects, transmitted through an optical system, measured by a sensor, converted into electrical signals, sampled and quantized, and finally represented as numerical data. A vision algorithm does not operate directly on objects, surfaces, or illumination. It operates on arrays of numbers that are only an approximation of the visual world.

Understanding this chain is essential for postgraduate study because many failures attributed to “poor algorithms” actually arise from image formation, inadequate sampling, sensor noise, illumination changes, or inappropriate mathematical assumptions. A classifier may fail because a camera introduced motion blur; a segmentation method may be unstable because shadows altered pixel intensities; and a convolutional model may learn sensor-specific artifacts rather than object structure. Sound computer vision practice therefore requires both algorithmic knowledge and physical understanding.

This chapter develops the foundations needed for later topics. It explains how images are formed and represented, how sampling and quantization affect information, how colour and illumination are modelled, and how convolution, linear algebra, probability, optimization, and frequency analysis support modern vision systems.

Learning Objectives

After studying this chapter, students should be able to:

1. Distinguish continuous image formation from digital image representation.

2. Explain pixels, spatial sampling, quantization, resolution, aliasing, and dynamic range.

3. Describe the roles of optics, camera sensors, illumination, and noise.

4. Compare common colour spaces and select an appropriate representation for an application.

5. Formulate image operations using vectors, matrices, convolution, and probability.

6. Interpret images in both spatial and frequency domains.

7. Explain the role of optimization in parameter estimation and image restoration.

8. Identify practical, ethical, and technical risks in image acquisition and processing.

From a Physical Scene to a Digital Image

A real scene contains surfaces with geometry, reflectance properties, and illumination. The intensity arriving at a camera depends on the interaction of these factors. A simplified image-formation model is

\[

I(x,y)=R(x,y)L(x,y),

\]

where \(R(x,y)\) denotes surface reflectance and \(L(x,y)\) denotes illumination. This multiplicative model is not complete: cast shadows, inter-reflections, transparency, atmospheric effects, lens characteristics, and sensor response may also be important. Nevertheless, it explains why the same object can have different pixel values under different lighting conditions.

The optical system maps points in the three-dimensional world to locations on a two-dimensional image plane. In a pinhole camera model,

\[

x=f\frac{X}{Z}, \qquad y=f\frac{Y}{Z},

\]

where \((X,Y,Z)\) is a point in camera coordinates, \((x,y)\) is its image position, and \(f\) is focal length. The division by depth \(Z\) produces perspective: distant objects occupy fewer image pixels than nearby objects. A complete camera model also includes principal-point offsets, pixel dimensions, and lens distortion. Radial distortion bends straight lines outward or inward, while tangential distortion results from imperfect lens alignment.

The sensor does not measure an ideal point value. Each pixel integrates light over a finite area and over an exposure interval. Consequently, the recorded value is influenced by neighbouring scene points and by motion during exposure. In simplified form, the sensor output can be regarded as a blurred and sampled version of the optical image:

\[

I_d[m,n] = Q\{(h * I_c)(m\Delta_x,n\Delta_y)+\eta[m,n]\},

\]

where \(I_c\) is the continuous image, \(h\) is the point-spread function, \(*\) denotes convolution, \(\eta\) represents noise, \(\Delta_x,\Delta_y\) are sampling intervals, and \(Q\) is quantization.

Pixels, Sampling, and Quantization

A pixel is not necessarily a physical point. It is a discrete image element associated with a spatial region and one or more measured values. A grayscale image may be represented as a matrix \(I\in\mathbb{R}^{M\times N}\), while a colour image is commonly represented as \(I\in\mathbb{R}^{M\times N\times 3}\).

Sampling converts a continuous spatial signal into measurements at discrete locations. If the sampling interval is too large relative to image detail, aliasing occurs. Fine patterns may appear as false lower-frequency patterns, edges may become jagged, and rotating structures may appear to move in the wrong direction. The Nyquist–Shannon principle states that a band-limited signal must be sampled at a rate greater than twice its highest frequency for exact reconstruction in ideal conditions.

In imaging, ideal conditions rarely hold. A practical camera uses an optical low-pass filter or sensor integration to reduce frequencies that cannot be represented at the selected sampling rate. Downsampling should normally be preceded by low-pass filtering. Simply discarding every second pixel can create moiré patterns and irreversible information loss.

Quantization maps continuous or high-precision sensor measurements to a finite set of digital levels. With \(b\) bits, the number of possible levels is \(2^b\). An 8-bit grayscale image has 256 nominal levels, whereas a 12-bit sensor has 4096 levels. Quantization introduces an error whose magnitude is bounded approximately by half a quantization step under ordinary rounding. Bit depth affects tonal precision and dynamic range, but it does not by itself guarantee better image quality; optics, exposure, noise, and sensor design also matter.

Spatial resolution concerns the ability to distinguish nearby structures. Intensity resolution concerns the number of measurable brightness levels. Dynamic range is the ratio between the largest and smallest reliably measurable signals. Saturation occurs when a signal exceeds the sensor’s upper limit, while underexposure may bury detail in noise. Neither saturated highlights nor clipped shadows can generally be recovered by later enhancement.

Illumination, Optics, and Camera Sensors

Illumination may be natural or artificial, uniform or spatially varying, diffuse or directional, and stable or time-dependent. The colour temperature of a light source changes the relative response of red, green, and blue channels. White balance attempts to compensate for this variation, but correction is imperfect when a scene contains multiple light sources or unusual surface colours.

Lens aperture controls the amount of light and influences depth of field. Shutter speed determines exposure duration and therefore affects motion blur. Sensor gain, often called ISO in photographic systems, amplifies the signal but also amplifies noise. These settings involve trade-offs. A fast shutter can freeze motion but requires more light or gain; a narrow aperture increases depth of field but reduces incoming light; high gain permits imaging in darkness but reduces signal-to-noise quality.

Image sensors commonly use complementary metal–oxide–semiconductor (CMOS) or charge-coupled device (CCD) technologies. A sensor’s response is affected by photon statistics, read noise, dark current, pixel non-uniformity, and temperature. Photon or shot noise is approximately signal-dependent and is often modelled using a Poisson distribution. Read noise is frequently approximated by a Gaussian component. A practical noise model may therefore be written as

\[

Y = I + \eta_s(I)+\eta_r,

\]

where \(\eta_s\) is signal-dependent noise and \(\eta_r\) is read noise.

Other artefacts include dead pixels, hot pixels, rolling-shutter distortion, lens flare, chromatic aberration, vignetting, and compression artefacts. Rolling shutters expose different image rows at different times, which can distort rapidly moving objects. These effects are especially important in robotics, industrial inspection, and autonomous driving.

Colour as a Computational Representation

Colour is a perceptual and physical phenomenon represented computationally through colour spaces. RGB records responses associated with red, green, and blue channels. It is convenient for display and image acquisition, but its channels are highly correlated and are not perceptually uniform.

A typical conversion from linear RGB to grayscale uses a weighted combination such as

\[

Y=0.2126R+0.7152G+0.0722B,

\]

with coefficients reflecting human sensitivity under a particular standard. Grayscale conversion discards chromatic information and is suitable only when colour is not essential.

HSV and HSL separate a colour-like hue component from saturation and brightness or lightness. They are useful for simple colour-based segmentation, although hue becomes unstable when saturation is low. CIE Lab attempts to represent perceptual differences more uniformly and is often useful for colour comparison. YCbCr separates luminance from chrominance and is widely used in image and video compression. The choice of colour space should follow the task: display, compression, perceptual comparison, segmentation, and physical measurement have different requirements.

A critical distinction is between linear and nonlinear encoded values. Many images stored as standard RGB use a gamma or transfer function. Arithmetic performed directly on gamma-encoded values can produce physically incorrect averages and blends. Operations based on light energy should generally be performed in a linear-light representation.

Mathematical Language for Images

Linear algebra provides the basic language of image processing. An image can be treated as a matrix, a vector, or a function sampled on a grid. A local image patch can be flattened into a vector, enabling operations such as projection, regression, and principal component analysis. A colour pixel can be represented as a vector \([R,G,B]^T\), and a colour transformation can be written as

\[

\mathbf{y}=A\mathbf{x}+\mathbf{b}.

\]

Matrix multiplication expresses channel transformations, geometric mappings, and dimensionality reduction. Norms quantify magnitude or error. The Euclidean norm \(\|x\|_2\) is common for energy-like measurements, while the \(\ell_1\) norm promotes sparse error patterns and can be more robust to outliers.

Convolution is a central operation in image processing. For a two-dimensional image and kernel \(K\),

\[

(I*K)(x,y)=\sum_{u}\sum_{v}I(x-u,y-v)K(u,v).

\]

A smoothing kernel reduces local variation; a derivative kernel responds to edges; and a sharpening kernel enhances high-frequency content. Convolution is translation equivariant: shifting the input shifts the output, subject to boundary handling. In practice, correlation is often implemented instead of mathematically reversed convolution, especially in machine-learning libraries. The distinction matters when a kernel is asymmetric.

Boundary treatment affects results. Zero padding may create dark borders, replication extends edge values, reflection mirrors the image, and circular padding assumes periodicity. There is no universally correct choice; the appropriate method depends on the physical or computational interpretation of the image.

Separable kernels can be expressed as the outer product of two one-dimensional kernels. Applying them in two stages reduces computation from approximately \(O(k^2)\) to \(O(2k)\) per pixel for a \(k\times k\) kernel. This idea underlies efficient Gaussian filtering and some neural-network architectures.

Probability and Statistical Reasoning

Images are measurements and therefore uncertain. Probability provides a framework for representing noise, variation, and incomplete knowledge. A random variable may describe a pixel value, an object location, or a camera parameter. The conditional probability \(p(y|x)\) describes the likelihood of an observation given a scene or model.

Bayes’ rule is

\[

p(x|y)=\frac{p(y|x)p(x)}{p(y)},

\]

where the posterior combines a likelihood with a prior. In image restoration, the observed image may be treated as evidence and a clean image as an unknown variable. A prior can express preferences for smoothness, sparsity, or natural-image structure. Priors are useful but can also introduce bias by favouring some appearances over others.

Expectation, variance, covariance, and correlation help characterize images and datasets. Covariance matrices reveal relationships among colour channels or features. However, correlation does not establish causation, and statistics estimated from one camera or environment may not generalize to another. Data collection should therefore cover relevant lighting, devices, viewpoints, and populations.

Optimization in Vision Problems

Many vision tasks can be expressed as the minimization of an objective function:

\[

\hat{\theta}=\arg\min_{\theta} \mathcal{L}(\theta;D),

\]

where \(\theta\) denotes unknown parameters and \(D\) denotes data. In camera calibration, parameters may include focal length and distortion coefficients. In denoising, they may define an estimated clean image. In machine learning, they include model weights.

A least-squares objective minimizes squared residuals and is analytically convenient, but it is sensitive to outliers. Robust losses such as Huber loss reduce the influence of unusually large errors. Regularization adds a penalty to discourage undesirable solutions:

\[

\min_x \|Ax-b\|_2^2+\lambda R(x).

\]

The parameter \(\lambda\) controls the balance between data fidelity and prior preference. Excessive regularization can erase meaningful detail, while insufficient regularization can preserve noise or produce unstable solutions.

Gradient descent updates parameters in the direction opposite to the gradient. Convex objectives have attractive global-optimum properties, but many modern vision models involve non-convex objectives. Initialization, learning rate, normalization, data quality, and validation strategy therefore matter. Optimization performance should not be confused with model validity: a well-optimized system can still learn a biased or inappropriate objective.

Spatial and Frequency Domains

The spatial domain describes intensity at image locations. The frequency domain describes how rapidly intensity changes across space. Low frequencies correspond broadly to smooth illumination and gradual structure; high frequencies correspond to edges, fine texture, noise, and abrupt transitions.

The Fourier transform decomposes an image into sinusoidal components. The convolution theorem states that convolution in the spatial domain corresponds to multiplication in the frequency domain:

\[

\mathcal{F}\{I*K\}=\mathcal{F}\{I\}\mathcal{F}\{K\}.

\]

This explains why smoothing suppresses high frequencies and why sharpening enhances them. The discrete Fourier transform is computationally practical through the fast Fourier transform (FFT). Frequency analysis supports filtering, periodic-pattern detection, image compression, deblurring, and diagnosis of sensor artefacts.

The frequency domain does not replace spatial reasoning. A filter designed only from a frequency response may create ringing near sharp edges. Similarly, a low-frequency image can preserve broad illumination while losing object boundaries. Effective systems use both representations and evaluate results in relation to the application.

Applied Case: Vision Inspection on a Manufacturing Line

Background

Consider an illustrative case involving a camera-based system for identifying scratches on painted metal panels. The system must operate on a conveyor under factory lighting and make a decision within a fraction of a second.

Situation and Problem

An initial prototype performs well during daytime testing but produces many false alarms during evening shifts. Some panels are also missed when the conveyor speed increases. Investigation shows that illumination changes the apparent colour and brightness of the paint, while longer exposures create motion blur. Compression of images for storage further obscures small scratches.

Analysis

The problem is not solely a classification issue. The image-formation chain is unstable. Illumination variation changes \(L(x,y)\) in the reflectance–illumination model. Motion during exposure convolves the scene with a directional blur kernel. The selected sampling rate may be inadequate for narrow scratches, and 8-bit processing reduces useful tonal separation in dark regions.

Approach

The engineering team installs controlled diffuse illumination and uses a fixed camera–object geometry. Exposure and gain are locked rather than left to automatic control. A calibration procedure estimates lens distortion and maps the panel to a standard image region. Images are captured at higher bit depth, converted to linear intensity for quantitative processing, and downsampled only after low-pass filtering. A colour representation separating luminance from chrominance is tested, while a grayscale reflectance-sensitive channel is retained for scratch detection.

A small Gaussian filter reduces sensor noise, followed by a derivative-based edge operation. Training and validation sets are collected across shifts, batches, temperatures, and acceptable surface variations. Performance is assessed using precision, recall, false-alarm cost, and missed-defect cost rather than accuracy alone.

Outcome

The controlled acquisition system reduces variation before machine learning is applied. The resulting model is smaller, easier to validate, and less dependent on the time of day. The case illustrates a general principle: improving image formation and measurement often provides greater benefit than immediately increasing model complexity.

Learning Points

1. Camera settings are part of the vision system, not merely hardware details.

2. Sampling and exposure must reflect the size and speed of the target phenomenon.

3. Evaluation should represent real operating conditions.

4. Physical controls, calibration, and preprocessing can reduce statistical burden.

5. A useful objective must account for the different costs of false positives and false negatives.

Challenges, Risks, and Professional Considerations

Common mistakes include resizing without anti-aliasing, mixing gamma-encoded and linear values, ignoring saturation, comparing images from uncalibrated cameras, and applying a filter without considering boundary effects. Another frequent error is treating normalization as a universal solution. Normalization can reduce variation, but it may also remove information needed for diagnosis or introduce dataset-specific assumptions.

Data leakage is a serious analytical risk. If images from the same physical object, production batch, or video sequence appear in both training and test sets, reported performance may be unrealistically high. Reproducible experiments should record camera settings, calibration files, preprocessing operations, random seeds where relevant, and dataset partitions.

Computer vision systems may process faces, medical images, workplaces, or public spaces. Image data can contain personally identifiable or sensitive information. Professional practice requires lawful collection, purpose limitation, access control, secure storage, documented retention, and appropriate consent or authorization. Models should be evaluated for performance differences across relevant demographic or environmental groups. Automated decisions affecting people should remain subject to human oversight and appeal mechanisms where appropriate.

Emerging Directions and Best Practices

Modern cameras increasingly support high dynamic range, event-based sensing, multispectral imaging, depth measurement, and computational photography. Event cameras record changes asynchronously rather than full frames, offering advantages for high-speed motion and high dynamic range. Neural image-processing systems can learn denoising, super-resolution, and colour correction, but learned reconstruction may hallucinate details not actually measured. Such systems must distinguish enhancement from faithful recovery, especially in scientific, medical, and legal contexts.

Recommended practice includes:

• characterize the sensor and optics before designing the algorithm;
• calibrate geometry, colour, and exposure when measurements matter;
• filter before downsampling;
• preserve sufficient bit depth during intermediate processing;
• use task-appropriate colour spaces and linear-light arithmetic;
• model noise and blur when restoring or interpreting images;
• validate across devices, environments, and time periods;
• report uncertainty and application-specific error costs;
• document data provenance, preprocessing, and operating limits.

Chapter Summary

A digital image is the result of a physical and computational pipeline involving illumination, optics, sensing, sampling, quantization, and representation. Pixels are discrete measurements rather than perfect point samples. Sampling determines spatial detail and can produce aliasing when performed inadequately; quantization determines intensity precision and contributes to digital error. Camera exposure, aperture, gain, sensor technology, lens distortion, and noise strongly influence image quality.

Colour spaces provide different compromises among display compatibility, perceptual uniformity, compression, and physical interpretation. Linear algebra represents images, patches, colours, and transformations, while convolution provides a foundation for smoothing, differentiation, sharpening, and learned filters. Probability describes uncertainty and variation, and optimization estimates parameters or images by balancing data fidelity with regularization. Fourier analysis reveals the relationship between spatial structure and frequency content. Together, these foundations explain why reliable computer vision depends on both mathematical modelling and careful image acquisition.

Key Terms

Pixel: A discrete image element associated with a spatial region and measured value or values.

Sampling: Conversion of a continuous spatial or temporal signal into discrete measurements.

Quantization: Mapping of continuous-valued measurements to a finite set of digital levels.

Aliasing: False or ambiguous structure produced when a signal is sampled below the required rate.

Nyquist rate: A sampling rate greater than twice the highest frequency of an ideal band-limited signal.

Dynamic range: Ratio between the largest and smallest reliably measurable signal levels.

Point-spread function: The image produced by an imaging system in response to an ideal point.

Convolution: A weighted sum of neighbouring image values using a kernel.

Reflectance: Proportion of incident illumination reflected by a surface.

White balance: Adjustment intended to compensate for the colour of an illumination source.

Shot noise: Signal-dependent noise arising from the statistical nature of photon arrival.

Read noise: Noise introduced during sensor readout and electronic conversion.

Colour space: A defined numerical representation of colour components.

Gamma encoding: Nonlinear mapping used to represent intensity values efficiently or compatibly.

Regularization: Addition of a penalty or constraint that discourages undesirable solutions.

Kernel: A finite array of weights used in a local image operation.

Fourier transform: Representation of a signal as a combination of frequency components.

Point-spread function: Spatial response characterizing blur introduced by an imaging system.

Signal-to-noise ratio: Ratio comparing useful signal strength with noise strength.

Rolling shutter: Sensor operation in which different image rows are exposed at different times.

Review Questions

Short Answer Questions

1. What is the difference between spatial resolution and intensity resolution?

2. Why should an image generally be low-pass filtered before downsampling?

3. State two effects of camera gain and explain their relationship.

4. Why can arithmetic on gamma-encoded RGB values be physically misleading?

5. What does a convolution kernel do in an image-processing operation?

Long Answer Questions

1. Explain the complete transformation from a three-dimensional scene to a digital image, including optics, sensing, sampling, and quantization.

2. Discuss the major sources of noise and distortion in camera systems and their consequences for computer vision.

3. Compare RGB, HSV, Lab, and YCbCr, and identify suitable applications for each.

4. Explain how probability, priors, loss functions, and regularization are used in image restoration.

5. Describe the relationship between spatial-domain filtering and frequency-domain filtering using the convolution theorem.

Analytical / Application Questions

1. A factory camera detects narrow defects but produces moiré patterns after image resizing. Diagnose the likely cause and propose a mathematically justified preprocessing pipeline.

2. A model performs accurately on images from one camera but poorly on another camera under the same scene illumination. Analyse possible causes involving sensor response, colour encoding, noise, and calibration.

3. Design an image-acquisition and validation strategy for a high-speed inspection system in which missed defects are five times more costly than false alarms.

22. Multiple Choice Questions with Answers

Question 1

Which change most directly reduces aliasing when an image is downsampled?

A. Increasing image saturation

B. Applying a suitable low-pass filter before downsampling

C. Replacing RGB with HSV

D. Increasing display brightness

Correct Answer: B. Applying a suitable low-pass filter before downsampling

Question 2

A 10-bit sensor can represent how many nominal intensity levels per channel?

A. 10

B. 100

C. 512

D. 1024

Correct Answer: D. 1024

Question 3

In the simplified image-formation model I(x,y)=R(x,y)L(x,y), R represents:

A. Sensor read noise

B. Surface reflectance

C. Focal length

D. Sampling interval

Correct Answer: B. Surface reflectance

Question 4

Which situation is most likely to produce motion blur?

A. Short exposure of a stationary object

B. Long exposure while the object or camera moves

C. Increasing bit depth

D. Using a colour-space conversion

Correct Answer: B. Long exposure while the object or camera moves

Question 5

Why is a colour space such as YCbCr useful in many imaging systems?

A. It eliminates all sensor noise

B. It separates a luminance-like component from chrominance components

C. It guarantees perceptual uniformity

D. It prevents lens distortion

Correct Answer: B. It separates a luminance-like component from chrominance components

Question 6

What is the main effect of applying a smoothing convolution kernel?

A. Suppressing local high-frequency variation

B. Increasing sensor dynamic range

C. Recovering saturated pixels

D. Changing camera focal length

Correct Answer: A. Suppressing local high-frequency variation

Question 7

In the optimization expression ||Ax-b||² + λR(x), increasing λ generally:

A. Removes the data-fidelity term

B. Strengthens the influence of the regularizer

C. Guarantees a global optimum for every problem

D. Increases the camera exposure time

Correct Answer: B. Strengthens the influence of the regularizer

Question 8

Which statement best describes shot noise?

A. It is independent of signal level in all conditions

B. It arises from statistical variation in photon arrival and is signal-dependent

C. It is caused only by image compression

D. It is eliminated by changing RGB to Lab

Correct Answer: B. It arises from statistical variation in photon arrival and is signal-dependent

Question 9

The convolution theorem states that spatial convolution corresponds to what operation in the frequency domain?

A. Addition of image dimensions

B. Multiplication of Fourier transforms

C. Quantization into integer levels

D. Geometric projection

Correct Answer: B. Multiplication of Fourier transforms

Question 10

In the manufacturing inspection case, why were fixed exposure and controlled illumination beneficial?

A. They increased the number of colour channels automatically

B. They reduced variation in the image-formation process

C. They guaranteed that scratches had identical shapes

D. They removed the need for validation data

Correct Answer: B. They reduced variation in the image-formation process