2. Digital Image Representation and Mathematical Foundations

Introduction and Context

Digital image processing begins with a deceptively simple question: how can a visual scene be represented as numbers that a computer can store, transmit, analyse, and transform? A camera does not directly produce an image in the mathematical sense. It measures physical energy, samples that energy at discrete spatial locations, quantizes the measurements, and encodes the results as numerical arrays. Every later operation—enhancement, restoration, segmentation, recognition, compression, or machine learning—depends on the quality and meaning of this representation.

An image-processing student therefore needs more than familiarity with pixels. It is necessary to understand coordinate systems, spatial resolution, intensity and colour models, sampling, quantization, interpolation, histograms, linear algebra, probability, and the distinction between convolution and correlation. These foundations explain both what an algorithm does and why it may fail.

A useful abstraction is to regard a continuous image as a function of spatial coordinates. For a monochrome image,

\[

f(x,y)

\]

represents brightness at location \((x,y)\). A digital image is obtained when the coordinates and intensity values are made discrete. It can then be represented as a matrix

\[

F=[f(i,j)], \quad i=0,1,\ldots,M-1,\quad j=0,1,\ldots,N-1,

\]

where each element is a pixel value. This matrix representation connects visual information with the tools of mathematics and computation.

Learning Objectives

After studying this chapter, a student should be able to:

1. Explain how continuous images become digital images through sampling and quantization.

2. Interpret pixels, coordinates, spatial resolution, intensity resolution, and dynamic range.

3. Use matrix and vector notation to describe image operations.

4. Distinguish convolution from correlation and explain their roles in filtering and template matching.

5. Apply interpolation methods and recognize their effects on image quality.

6. Interpret grayscale and colour histograms using statistical measures.

7. Compare RGB, HSV, YCbCr, multispectral, and related image models.

8. Use probability and statistics to reason about noise, contrast, and image measurements.

9. Identify common representation errors, implementation risks, and professional best practices.

From Physical Scene to Digital Array

Pixels and coordinates

A pixel is the smallest addressable sample in a digital image, although it is not necessarily a physical point. In a sensor, a pixel usually corresponds to a finite photosensitive area. Its value is an aggregate measurement over that area, influenced by optics, exposure, sensor response, and electronic noise.

Most image-processing systems use row-column indexing. The first index identifies the row and the second identifies the column. Thus, \(f(10,20)\) may denote the pixel in row 10 and column 20. Mathematical texts sometimes write coordinates as \((x,y)\), with \(x\) horizontal and \(y\) vertical, whereas programming libraries commonly use \(I[row,column]\). Confusing these conventions can rotate, transpose, or incorrectly transform an image.

The origin may be placed at the upper-left corner, lower-left corner, or image centre. A coordinate transformation expresses one convention in terms of another. For example, if the image height is \(M\), converting an upper-left origin to a lower-left origin often requires replacing a vertical coordinate \(y\) by \(M-1-y\). Coordinate conventions must be documented in scientific and industrial systems.

Spatial resolution and intensity resolution

Spatial resolution describes the ability to distinguish fine spatial detail. It depends on sampling density, optical blur, sensor geometry, and processing. An image with more pixels is not automatically sharper: if the optics blur detail before sampling, additional pixels may only represent the blur more finely.

Intensity resolution describes the number of distinguishable intensity levels. With \(b\) bits per pixel, the number of possible levels is

\[

L=2^b.

\]

An 8-bit grayscale image has 256 levels, usually represented by integers from 0 to 255. A 12-bit sensor has 4096 levels and can preserve subtler differences, particularly in scientific or medical imagery. Intensity resolution should not be confused with dynamic range. Dynamic range concerns the ratio between the largest and smallest measurable signals, while bit depth concerns how finely that range is encoded.

Sampling and quantization

Sampling makes spatial coordinates discrete. If a continuous signal is sampled at intervals \(\Delta x\) and \(\Delta y\), the resulting samples are \(f(i\Delta x,j\Delta y)\). According to the sampling theorem, a one-dimensional band-limited signal must be sampled at a rate greater than twice its highest frequency for exact theoretical reconstruction. In images, insufficient sampling causes aliasing: high-frequency patterns appear as false lower-frequency structures, such as jagged edges or moiré patterns.

Optical or digital prefiltering reduces frequencies that cannot be safely sampled. This anti-aliasing step is often preferable to allowing aliasing and attempting to remove it later, because aliasing changes the measured information and is generally irreversible.

Quantization maps continuous or high-precision sample amplitudes to a finite set of levels. If a value is rounded to the nearest level, the quantization error is approximately bounded by half the quantization interval under ideal conditions. Quantization produces banding in smooth gradients and contributes noise-like error. Increasing bit depth reduces the interval, but does not correct poor exposure, blur, or aliasing.

Mathematical Representation of Images

Matrix and vector views

A grayscale image can be treated as a matrix, a vector, or a function on a finite lattice. Matrix notation is useful for transformations and neighbourhood operations. A colour image with three channels may be represented as

\[

I\in\mathbb{R}^{M\times N\times 3}.

\]

Alternatively, each channel can be viewed as a separate matrix, such as \(R\), \(G\), and \(B\). Vectorization converts an image into a long column vector, enabling methods such as principal component analysis, least-squares estimation, and covariance analysis.

A geometric transformation can be written in homogeneous coordinates. For a point \((x,y)\), represent it as \((x,y,1)^T\). Translation, rotation, scaling, and affine transformations can then be expressed as matrix multiplication:

\[

\begin{bmatrix}x'\\y'\\1\end{bmatrix}

=

\begin{bmatrix}a&b&t_x\\c&d&t_y\\0&0&1\end{bmatrix}

\begin{bmatrix}x\\y\\1\end{bmatrix}.

\]

This formulation is important in image registration, camera calibration, panorama construction, and geometric correction.

Norms and similarity

Vector norms provide quantitative measures of image magnitude or error. For an error image \(E=I-\hat I\), the squared Euclidean norm is

\[

\|E\|_2^2=\sum_{i,j}E(i,j)^2.

\]

Mean squared error (MSE) divides this quantity by the number of pixels. Root mean squared error (RMSE) is the square root of MSE. These metrics are simple and useful, but they do not always agree with human perception. A small geometric shift can yield a large pixel-wise error while the image remains visually recognizable.

The inner product of two image patches,

\[

\langle A,B\rangle=\sum_{i,j}A(i,j)B(i,j),

\]

measures alignment in a vector space. Normalized correlation additionally compensates for patch energy and is often more reliable for matching when brightness varies.

Convolution, Correlation, and Local Operators

A local image operator combines a neighbourhood of pixels with a kernel or mask. For a kernel \(h\), two-dimensional discrete convolution is commonly defined as

\[

(g*h)(i,j)=\sum_m\sum_n g(m,n)h(i-m,j-n).

\]

The kernel is reflected in both dimensions before being shifted. Correlation is

\[

(g\star h)(i,j)=\sum_m\sum_n g(i+m,j+n)h(m,n),

\]

and does not reflect the kernel. If the kernel is symmetric, convolution and correlation produce the same result; otherwise they differ. This distinction matters in edge detection, matched filtering, and deep-learning implementations, where “convolution” layers often perform cross-correlation operationally.

The boundary policy must also be specified. Zero padding introduces artificial dark borders; replication extends boundary values; reflection mirrors the image; and valid convolution discards locations where the kernel does not fit. Different choices can materially affect results near edges.

A smoothing kernel, such as a normalized averaging mask, reduces local variation but blurs edges. A Gaussian kernel gives greater weight to nearby pixels and provides a controlled scale of smoothing. Derivative kernels estimate intensity changes. For example, a horizontal derivative responds strongly to vertical edges. The Laplacian uses second derivatives and can emphasize rapid intensity transitions, but it is sensitive to noise.

The convolution theorem explains why filtering can be performed in either spatial or frequency domains: convolution in the spatial domain corresponds to multiplication in the Fourier domain. Spatial filtering is often efficient for small kernels, while frequency-domain methods may be advantageous for very large kernels or repeated operations.

Interpolation and Geometric Resampling

Geometric transformation rarely maps output pixels exactly onto input pixels. Interpolation estimates values at non-integer coordinates. Nearest-neighbour interpolation assigns the value of the closest sample. It is fast and preserves label values, making it suitable for categorical masks, but produces blockiness.

Bilinear interpolation uses the four surrounding pixels and weights them according to distance. It produces smoother images but can reduce sharpness. Bicubic interpolation uses a larger neighbourhood and cubic weighting, usually producing smoother and more visually pleasing results at greater computational cost. No interpolation method recreates detail that was never sampled.

The choice depends on data semantics. Continuous intensity imagery may use bilinear or bicubic interpolation; segmentation labels should generally use nearest neighbour, because averaging class identifiers creates meaningless intermediate labels. Resampling should also account for the scale ratio and use anti-aliasing when reducing image size.

Histograms and Statistical Foundations

The histogram of a grayscale image counts the number of pixels at each intensity. If \(n_k\) is the number of pixels with level \(r_k\), the normalized histogram is

\[

p(r_k)=\frac{n_k}{MN}.

\]

It estimates the empirical probability distribution of intensities. From it one can calculate the mean,

\[

\mu=\sum_k r_kp(r_k),

\]

and variance,

\[

\sigma^2=\sum_k(r_k-\mu)^2p(r_k).

\]

A narrow histogram may indicate low contrast; a histogram concentrated near one end may indicate underexposure or overexposure. However, a histogram discards spatial arrangement. Two images can have identical histograms while showing completely different scenes.

Histogram stretching maps a limited intensity interval to a wider range. Histogram equalization uses the cumulative distribution function to redistribute intensities and often improve global contrast. It may also amplify noise or produce unnatural appearances. Adaptive methods operate on local regions and can reveal local detail, but excessive local enhancement may create halos and noise.

Probability models support noise analysis. Gaussian noise is often used as an approximation for electronic measurement error. Salt-and-pepper noise appears as isolated extreme pixels and is better addressed by median filtering than by simple averaging. Poisson noise is associated with counting processes and has variance related to signal level. A processing system should select a model based on acquisition physics rather than convenience alone.

Random variables, expectation, variance, covariance, and correlation are also central to image analysis. Covariance between channels indicates whether they carry redundant variation. The covariance matrix of multispectral data can be diagonalized to obtain principal components, which may reduce dimensionality or expose dominant patterns.

Grayscale, Colour, and Multispectral Models

Grayscale images represent intensity without explicit chromatic information. The conversion of RGB to grayscale should account for human visual sensitivity rather than simply averaging channels. A commonly used luminance approximation gives greater weight to green than to red or blue. The exact coefficients depend on the colour standard and the intended meaning of “brightness.”

The RGB model represents additive mixtures of red, green, and blue primaries. It is natural for displays and many cameras, but its coordinates are device-dependent and strongly coupled: changing illumination may alter all three channels.

HSV and HSI separate a colour-like component from intensity-related components. Hue describes dominant colour, saturation describes colourfulness, and value or intensity describes a brightness measure. These models are convenient for thresholding coloured objects, but hue becomes unstable when saturation is near zero and is circular rather than linear.

YCbCr separates a luminance-like component, Y, from two chrominance components. It is widely used in image and video compression because the human visual system is generally more sensitive to luminance detail than to fine chrominance detail. Chroma subsampling can therefore reduce storage while maintaining acceptable perceived quality.

Multispectral images contain measurements in more than the conventional three visible bands. Hyperspectral systems may contain many narrow, contiguous bands. Such data enable material discrimination, vegetation analysis, environmental monitoring, and remote sensing. They also introduce high dimensionality, calibration requirements, atmospheric effects, and substantial inter-band correlation. Spectral values should not be treated as ordinary RGB values without understanding sensor response and calibration.

Colour management is important in professional workflows. Images may be encoded using different transfer functions, white points, and colour spaces. Confusing linear-light data with gamma-encoded data can produce incorrect blending, filtering, and quantitative measurements. Operations such as averaging physically meaningful light should generally be performed in an appropriate linear representation.

Practical and Professional Applications

These foundations support nearly every image-processing system. In medical imaging, intensity values may represent calibrated physical quantities, so arbitrary histogram manipulation can compromise interpretation. In manufacturing, coordinate calibration and subpixel interpolation influence dimensional inspection. In remote sensing, multispectral statistics support classification and change detection. In computer vision, convolutional filters extract local structure, while colour-space selection affects segmentation and tracking.

For software and engineering teams, representation decisions affect memory, speed, reproducibility, and interoperability. A 16-bit image accidentally converted to 8-bit may lose diagnostic information. An unsigned integer subtraction may underflow rather than produce a negative difference. A library may interpret an array as BGR while an algorithm assumes RGB. These are not cosmetic programming errors; they change the data being analysed.

A sound workflow records image dimensions, bit depth, channel order, coordinate convention, colour space, calibration status, and missing-value policy. Intermediate results should be inspected numerically, not only displayed, because visualization software may rescale or clip values.

Illustrative Case Study: Automated Surface Inspection

Background

Consider an illustrative manufacturing system that inspects coated metal panels using a line-scan camera. The objective is to identify scratches and coating voids while the panel moves beneath controlled lighting.

Situation and problem

Initial trials produce inconsistent detections. Some defects are missed near the image borders, while harmless texture is classified as damage. Images from two production lines also show different brightness distributions.

Analysis

Investigation reveals three representation problems. First, the camera output is 12-bit, but one acquisition program silently converts it to 8-bit. Second, the inspection software assumes the first array index is the horizontal coordinate, transposing the physical interpretation. Third, a downsampling stage uses nearest-neighbour interpolation on intensity images without anti-aliasing, creating false patterns.

Histogram analysis shows that the two lines have different exposure levels. A flat-field calibration image further reveals gradual illumination variation. The defect signatures are not purely intensity-based: scratches are elongated and have directional contrast.

Approach and solution

The team retains 12-bit data through preprocessing and records the sensor calibration. A coordinate test pattern verifies row-column orientation. Illumination correction is applied using a reference field. Before downsampling, a low-pass filter suppresses frequencies that would alias. A small derivative-of-Gaussian filter emphasizes elongated intensity transitions, and candidate regions are evaluated using normalized local contrast rather than a global threshold. Border handling is explicitly set to reflection, and validation data include defects at all image locations.

Outcome

The revised system produces more stable measurements across production lines and reduces false detections caused by downsampling and illumination gradients. The key improvement is not one sophisticated classifier; it is the disciplined control of representation, calibration, sampling, and local statistics.

Learning points

The case demonstrates that algorithmic performance is limited by the quality and interpretation of the image array. Bit depth, coordinate conventions, interpolation, boundary policy, and normalization should be treated as engineering requirements and tested systematically.

Limitations, Risks, and Common Mistakes

Several errors recur in postgraduate projects and industrial prototypes:

• treating a pixel as an exact point and ignoring its area or optical footprint;
• confusing spatial resolution with bit depth;
• resizing without anti-aliasing;
• using bilinear interpolation on categorical labels;
• assuming convolution and correlation are interchangeable;
• comparing images with different colour spaces or gamma encodings;
• interpreting a histogram without considering spatial structure;
• applying global contrast enhancement to calibrated scientific data;
• silently clipping or converting numerical ranges;
• ignoring border effects and missing values;
• using pixel-wise error as the only measure of perceptual quality.

Noise reduction can erase genuine small features, and interpolation can create apparently smooth but unsupported values. Statistical assumptions should therefore be validated against acquisition conditions. Reproducible processing requires fixed parameters, documented formats, versioned code, and test images with known properties.

Ethical and Professional Considerations

Image representation has consequences beyond computation. In medical, biometric, surveillance, and remote-sensing applications, enhancement or false-colour display can influence human decisions. A visually attractive image may not be a quantitatively faithful one. Systems should preserve provenance, distinguish raw data from processed derivatives, and disclose transformations that affect interpretation.

Privacy, access control, retention, and auditability are essential when images contain identifiable persons or sensitive locations. Measurement systems should be tested for unequal performance caused by illumination, sensor, demographic, or environmental differences. Professional reporting should state uncertainty and avoid presenting interpolated or enhanced detail as directly observed evidence.

Emerging Directions and Best Practices

Modern learning systems use learned filters and representations, but they still depend on sampling, normalization, colour encoding, and numerical precision. Neural networks may be sensitive to rescaling conventions, channel order, compression artifacts, and domain shifts between sensors. Multispectral and hyperspectral learning increasingly combines spectral and spatial representations, while edge devices demand efficient quantization and low-memory computation.

Best practice is to:

1. define the image coordinate, channel, and numerical conventions before implementation;

2. preserve the highest useful precision until a justified conversion is required;

3. apply anti-aliasing before reduction and choose interpolation according to data type;

4. document convolution orientation, kernel normalization, and boundary handling;

5. examine both spatial images and statistical summaries;

6. calibrate sensors and distinguish physical units from display values;

7. validate algorithms using controlled patterns, real variation, and edge cases;

8. retain raw data and record every irreversible transformation.

Chapter Summary

A digital image is a sampled and quantized representation of a physical or mathematical signal. Pixels are organized in coordinate systems, and their values are governed by spatial resolution, intensity resolution, dynamic range, and sensor characteristics. Sampling without adequate bandwidth control causes aliasing; quantization introduces finite-level error. Matrix and vector models enable geometric transformations, similarity measures, and statistical analysis.

Convolution applies reflected kernels, whereas correlation does not, and boundary policies influence both results. Interpolation estimates unsampled values but cannot restore lost information. Histograms describe intensity distributions but not spatial arrangement; probability, variance, covariance, and correlation provide a foundation for modelling noise and image variability. RGB, HSV, YCbCr, grayscale, multispectral, and hyperspectral representations serve different purposes. Reliable image-processing practice requires attention to numerical precision, calibration, colour management, data semantics, reproducibility, and ethical interpretation.

Key Terms

Pixel: The smallest addressable sample in a digital image.

Sampling: Conversion of continuous spatial coordinates into discrete locations.

Quantization: Conversion of continuous or high-precision amplitudes into finite intensity levels.

Spatial resolution: Ability to distinguish detail in position or structure.

Intensity resolution: Number of distinguishable amplitude levels.

Dynamic range: Ratio or span between the smallest and largest measurable signals.

Aliasing: False structure produced when sampling is insufficient for signal frequency.

Interpolation: Estimation of values at locations between known samples.

Convolution: A sliding operation involving a reflected kernel.

Correlation: A sliding similarity operation without kernel reflection.

Kernel: A small array of weights used in a local image operation.

Histogram: A count or empirical distribution of image intensity values.

Luminance: A brightness-related component, often defined according to a colour standard.

Colour space: A coordinate system used to represent colour information.

Multispectral image: An image containing measurements from multiple spectral bands.

Covariance: A measure of joint variation between variables or channels.

MSE: Mean squared difference between corresponding image values.

Anti-aliasing: Filtering used to suppress frequencies that cannot be safely sampled.

Gamma encoding: Nonlinear encoding of intensity for display or storage purposes.

Coordinate convention: The defined relationship between array indices and physical positions.

Review Questions

Short Answer Questions

1. Distinguish spatial resolution from intensity resolution.

2. Why does undersampling produce aliasing?

3. State one mathematical difference between convolution and correlation.

4. What information does a grayscale histogram provide, and what does it omit?

5. Why is nearest-neighbour interpolation suitable for segmentation labels?

Long Answer Questions

1. Explain the complete process by which a continuous optical scene becomes a digital image, including sampling and quantization.

2. Compare RGB, HSV, and YCbCr representations and identify an application suited to each.

3. Discuss the role of linear algebra in image representation, geometric transformation, and similarity measurement.

4. Explain how probability and statistics support image noise modelling and multispectral analysis.

5. Describe the causes and consequences of boundary effects in convolution and propose suitable policies.

Analytical / Application Questions

1. An image is reduced to one-quarter of its original width and height. Explain the sampling risk and design an appropriate preprocessing sequence.

2. Two images have identical histograms but visibly different structures. Explain how this is possible and propose additional measures for comparison.

3. A defect-detection system receives 12-bit sensor data but displays it as an 8-bit image. Analyse the possible consequences and recommend a validation procedure.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes spatial resolution from intensity resolution?

A. Spatial resolution concerns the number of colour channels, while intensity resolution concerns image width.

B. Spatial resolution concerns distinguishable spatial detail, while intensity resolution concerns distinguishable value levels.

C. Spatial resolution concerns histogram shape, while intensity resolution concerns coordinate origin.

D. Spatial resolution concerns noise variance, while intensity resolution concerns interpolation method.

Correct Answer: B. Spatial resolution concerns distinguishable spatial detail, while intensity resolution concerns distinguishable value levels.

Question 2

A high-frequency stripe pattern is sampled below the required sampling rate. What is the most likely result?

A. The pattern is represented exactly with lower storage cost.

B. The image automatically gains higher intensity resolution.

C. Aliasing creates a false lower-frequency pattern.

D. Quantization error is completely eliminated.

Correct Answer: C. Aliasing creates a false lower-frequency pattern.

Question 3

For a nonsymmetric kernel, why can convolution and correlation produce different results?

A. Convolution reflects the kernel, whereas correlation does not.

B. Correlation always uses Fourier transforms, whereas convolution never does.

C. Convolution changes the image bit depth, whereas correlation changes its dimensions.

D. Correlation requires RGB data, whereas convolution requires grayscale data.

Correct Answer: A. Convolution reflects the kernel, whereas correlation does not.

Question 4

Which interpolation method is generally most appropriate for a categorical segmentation mask?

A. Bicubic interpolation

B. Bilinear interpolation

C. Nearest-neighbour interpolation

D. Gaussian interpolation followed by averaging

Correct Answer: C. Nearest-neighbour interpolation

Question 5

Two images have the same intensity histogram but different spatial arrangements. What does this demonstrate?

A. Histograms fully encode image geometry.

B. Histograms describe intensity distribution but discard spatial arrangement.

C. One image must have incorrect pixel values.

D. The images necessarily have identical edges.

Correct Answer: B. Histograms describe intensity distribution but discard spatial arrangement.

Question 6

Why can applying a simple average filter to salt-and-pepper noise be less effective than using a median filter?

A. A median filter changes every pixel to the global mean.

B. A median filter is better suited to isolated extreme values.

C. An average filter operates only on colour images.

D. A median filter increases spatial sampling frequency.

Correct Answer: B. A median filter is better suited to isolated extreme values.

Question 7

In the illustrative surface-inspection case, why was anti-aliasing applied before downsampling?

A. To increase the number of spectral bands.

B. To suppress frequencies that could create false patterns after reduction.

C. To convert categorical labels into continuous intensities.

D. To guarantee that all defects become brighter.

Correct Answer: B. To suppress frequencies that could create false patterns after reduction.

Question 8

Which representation is especially useful when luminance and chrominance are treated differently in compression?

A. YCbCr

B. Binary threshold representation

C. Homogeneous coordinates

D. A coordinate-free matrix

Correct Answer: A. YCbCr

Question 9

What is a major risk of silently converting calibrated 12-bit data to 8-bit data?

A. The image gains additional measurable intensity levels.

B. The coordinate origin necessarily moves to the centre.

C. Subtle intensity information may be lost through quantization and clipping.

D. Convolution becomes mathematically impossible.

Correct Answer: C. Subtle intensity information may be lost through quantization and clipping.

Question 10

Why should linear-light and gamma-encoded image values not be averaged indiscriminately?

A. They use different array dimensions.

B. Gamma encoding is nonlinear, so numerical averages may not represent physical light averages.

C. Gamma-encoded images contain no pixels.

D. Linear-light data cannot be stored digitally.

Correct Answer: B. Gamma encoding is nonlinear, so numerical averages may not represent physical light averages.